Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Edmond, OK vacancy
  • $45k

     ...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs...  ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before... 
    Suggested
    Full time
    Work experience placement

    MarCom Group

    Edmond, OK
    3 days ago
  • $51.63k - $124.72k

    Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Edmond, OK
    4 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Edmond, OK
    1 day ago
  •  ...fulfilling our protective legacy. Proven to be worthy of trust and confidence. Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and... 
    Suggested
    Overseas
    Trial period

    The United States Secret Service

    Oklahoma City, OK
    21 days ago
  • $17.5 - $18.5 per hour

     ...documentation, and ensuring accuracy within a highly regulated environment.Job Description:The Document Clerk will be responsible for reviewing and validating loan-related documentation, identifying missing or incomplete records, and assisting with the resolution of... 
    Suggested
    Permanent employment
    Full time
    Work at office
    Local area
    Monday to Friday

    Addison Group

    Oklahoma City, OK
    3 days ago
  •  ...pharmaceutical testing Company in the country with a well-earned reputation for premium customer service and quality Microbiology Data Reviewer I will perform routine review of microbiological testing results at times supporting verification activities in accordance with... 
    Full time
    Local area
    Monday to Friday
    Weekend work

    ARL Bio Pharma

    Oklahoma City, OK
    10 days ago
  •  ...and is a team of five built on strong relationships, growth, and a genuine commitment to serving others. Before becoming a State Farm agent, I worked as a team member and gained experience in financial planning, which continues to shape how we support our customers. I... 
    Work at office
    Flexible hours

    Blake Jordan State Farm Agent

    Edmond, OK
    2 days ago
  •  ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop... 
    For contractors
    Work at office

    Ann Smith

    Edmond, OK
    4 days ago
  • $15 - $18 per hour

     ...to help people manage the risks of everyday life, recover from the unexpected and realize their dreams.Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member.I am a third generation State Farm agent.We Look Forward To Speaking With... 
    For contractors
    Work at office
    Local area

    Thomas Webber - State Farm Agent

    Edmond, OK
    2 days ago
  • $45k - $100k

     ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads... 
    For contractors
    Work at office

    Ann Smith - State Farm Agent

    Edmond, OK
    5 days ago
  •  ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads... 
    For contractors
    Work at office

    Ann Smith State Farm Agent

    Edmond, OK
    3 days ago
  •  ...inbound marketing and sales approach. No legal experience? No problem. We train the right people. What you'll do: Prepare, review, and print court documents with precision Process incoming filings from courts and process servers Manage the full lifecycle... 
    Full time
    Work at office
    Flexible hours

    ABC Legal Services

    Oklahoma City, OK
    3 days ago
  •  ...estate deeds, lease documents, and related ownership records.  •  Review, process, and track land acquisition agreements, easements, and...  ...logic to streamline reporting.  •  Coordinate with field agents, internal team members, and outside parties to keep projects moving... 
    Full time
    Temporary work
    Work at office

    Gray Hawk Land Solutions

    Oklahoma City, OK
    12 days ago
  •  ...estate deeds, lease documents, and related ownership records. •  Review, process, and track land acquisition agreements, easements, and...  ...logic to streamline reporting. •  Coordinate with field agents, internal team members, and outside parties to keep projects moving... 
    Daily paid
    Temporary work
    Work at office

    Gray Hawk Land Solutions

    Oklahoma City, OK
    25 days ago
  • $14 - $16 per hour

    Records ClerkOklahoma City, OKOn-Site Role$14 - $16 per hourFull-Time, Temporary (6-12 months)Schedule: Monday - Friday, 8:00 a.m. - 5:00 p.m.About The RoleOur client is seeking a Records Clerk for a temporary administrative assignment. This role is essential in supporting...
    Contract work
    Temporary work
    Work at office
    Immediate start
    Monday to Friday
    Weekday work

    Eight Eleven Group

    Oklahoma City, OK
    3 days ago
  • $45k - $55.88k

    Redwood Central Inc. is seeking a File Clerk for a full-time position in Oklahoma City, OK. This role involves managing and organizing files to ensure efficient and accurate record-keeping. The File Clerk will be responsible for sorting, filing, and retrieving a variety...
    Full time
    Work at office

    Redwood Central Inc.

    Oklahoma City, OK
    1 day ago
  •  ...detail and organizational skills. Application Process Resume Review Initial discussion Video Screening Client Interview...  ...transformation solutions in Cloud, Cybersecurity, Data Analytics and AI. ISO-certified and CMMI Level 3 appraised, we proudly partner... 
    Temporary work
    Work at office

    Cogent Infotech Corporation

    Oklahoma City, OK
    21 days ago
  • Redwood Central Inc. is seeking a reliable File Clerk in Oklahoma City, OK to organize and manage company records. The role involves sorting, filing, retrieving documents, and supporting preparation and distribution for internal and external use in a busy office. The ideal...
    Work at office

    Redwood Central Inc.

    Oklahoma City, OK
    15 hours ago
  • $23 per hour

    Job Description Job Description Creative Asset Migration Specialist   Type : Contract — approximately 3 months Target  Salary : $23 per hour +, depending on experience Why This Opportunity Stands Out Put your file organization skills to work on a focused...
    Hourly pay
    Contract work
    Immediate start

    Mosaic Personnel

    Oklahoma City, OK
    26 days ago
  •  ...Reads design plans/sketches to understand land rights needed Reviews title information to create easements and other land...  ...instructions to field crews, history of contacts between Right of Way Agents and landowners, payments made, damage settlements, property values... 
    Permanent employment
    Full time
    For subcontractor
    Work at office
    Local area
    Remote work

    Coates Field Service Inc

    Oklahoma City, OK
    8 days ago
  •  ...and preparing accurate deliverables on short notice. Hybrid work option with initial training on-site. Responsibilities include maintaining databases, reviewing title information, creating easements, and coordinating with field staff and clients. #J-18808-Ljbffr Coates

    Coates

    Oklahoma City, OK
    4 days ago
  • A legal services firm in Oklahoma City is seeking a Legal Assistant to support attorneys throughout the Oklahoma foreclosure process. Responsibilities include case management, document preparation, and client communication. The ideal candidate must have at least 3 years...

    The Mortgage Law Firm

    Oklahoma City, OK
    15 hours ago
  • $30 per hour

     ...Job DescriptionJob DescriptionJob Title: Bin stocking agent Starting Pay: $30/hour RETIREES WELCOME TO APPLYHours: VERY Flexible, but must be performed within standard business hours which we consider 7am-5pm M-F Part-time: Approximately 4-10 hours / week Travel costs... 
    Part time
    Flexible hours

    Lightning Bolt

    Oklahoma City, OK
    3 days ago
  • $15 - $17 per hour

     ...Opportunity for advancement Paid time off Training & development Customer Service Representative / Receptionist State Farm Agent Team | Edmond, OK Full-Time | Monday–Friday Pay: $15–$17/hour + performance incentives Join Our Team Are you friendly,... 
    Full time
    For contractors
    Work at office
    Monday to Friday
    Night shift
    Weekend work

    Chandler Tolbert - State Farm Agent

    Edmond, OK
    9 days ago
  •  ...assignability/abandonment clauses, lease terms, and restrictions.Reviews title information and checks accuracy of legal descriptions...  ...instructions to field crews, history of contacts between Right of Way Agents and landowners, payments made, damage settlements, property... 
    Daily paid
    Permanent employment
    For subcontractor
    Work at office

    Coates Field Service

    Oklahoma City, OK
    3 days ago
  • $103.24k - $133.2k

     ...position advertised has been exempted from the federal civilian hiring freeze. Use your linguistics background to become an FBI special agent! The transition from language and linguistics to special agent is more than a career move—it’s an opportunity to upskill and become... 
    Work experience placement
    Work at office
    Local area

    Federal Bureau of Investigation (FBI)

    Oklahoma City, OK
    5 days ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Oklahoma City, OK
    5 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    Oklahoma City, OK
    3 days ago
  •  ...as a Customer Retention Specialist at Thomas Webber - State Farm Agent if: You’re the person people call first when life goes sideways...  ...with existing customers regarding renewals and coverage options.Review accounts and assist with updates or changes as needed.Identify opportunities... 
    Work at office

    Thomas Webber State Farm Agent

    Edmond, OK
    2 days ago
  • $45k - $65k

     ...and is a team of five built on strong relationships, growth, and a genuine commitment to serving others. Before becoming a State Farm agent, I worked as a team member and gained experience in financial planning, which continues to shape how we support our customers. I... 
    Work at office
    Flexible hours

    Blake Jordan - State Farm Agent

    Edmond, OK
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!