Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Orlando, FL vacancy
  • $45.3 - $78.6 per hour

     ...each other.CLA is seeking to hire an experienced Seasonal Tax Reviewer to join our talented tax team. The position offers flexibility...  ...or related field is preferred.CPA is required (JD or Enrolled Agent is acceptable in lieu).Our approach to compensation emphasizes... 
    Suggested
    Casual work
    Seasonal work

    CliftonLarsonAllen

    Orlando, FL
    3 days ago
  • $40 - $60 per hour

     ...Job details Senior Tax Review, Contract, Remote, based out of Orlando CPA Firm, reply to ****@*****.*** Salary $40 - $60 per hour A Bit About Us Growing CPA firm - Downtown Orlando Why join us? ~ Remote ~ Medical ~ Dental ~ Vision ~401k ~ Generous... 
    Suggested
    Hourly pay
    Contract work
    Work at office
    Local area
    Remote work

    Jobot Consulting

    Orlando, FL
    2 days ago
  • Employment Opportunities At Full Sail University If you're looking for the chance to learn, grow, and make a contribution to the community, look at employment opportunities with Full Sail University. You'll find your choice of career opportunities, great benefits, an environment...
    Suggested
    Work experience placement
    Work at office
    Flexible hours
    Night shift
    Weekend work

    Full Sail University

    Winter Park, FL
    5 days ago
  • $51.63k - $124.72k

     ...Job Description Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Orlando, FL
    17 days ago
  • Online Document SpecialistThe Online Document Specialist is responsible for the receiving, coding, reviewing, auditing, maintenance and management of incoming student documents, files, and enrollment data affecting various departments including admissions, financial aid... 
    Suggested
    Work at office

    Full Sail University

    Winter Park, FL
    3 days ago
  • A construction management firm in Orlando, FL is seeking a Document Controls Specialist to manage project documentation and ensure compliance with standards. The ideal candidate will have a degree in construction management, architecture, or engineering, along with a minimum...

    Procon Consulting

    Orlando, FL
    5 days ago
  • $19 per hour

     ...while maintaining speed and accuracy goals Research and collect item details and specifications for online listing preparation Review and verify inventory documentation submitted by other team members before system uploads Audit inventory, perform cycle counts,... 
    Local area
    Monday to Friday

    Nesco Resource

    Orlando, FL
    1 day ago
  • $55k - $70k

    Overview Document Controls Specialist - City/State Programs 3 days ago Be among the first 25 applicants This range is provided by Procon Consulting. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more. Base pay range $55,...
    Full time
    Worldwide

    Procon Consulting

    Orlando, FL
    5 days ago
  • $40 - $45 per hour

    RN QA Reviewer - OASIS & Chart Review (Per Diem, Remote) Do you love the moment a chart finally tells the whole story, accurately and completely? Do you find yourself spotting the one item that doesn't match the visit note? If so, we would love to meet you. ABOUT REVITALIZE... 
    Remote job
    Daily paid
    Relief
    Work from home
    Flexible hours

    Revitalize Home Healthcare

    Orlando, FL
    5 days ago
  • Revitalize Home Healthcare seeks an experienced RN QA Reviewer - OASIS & Chart Review for a remote, per diem role in Florida. You will review OASIS assessments and clinical documentation before submission, ensuring the stories align across records to protect patients and... 
    Remote job
    Daily paid

    Revitalize Home Healthcare

    Orlando, FL
    5 days ago
  •  ...Warehouse AgentDo you enjoy working in a fast-paced, safety-obsessed aviation environment? As a Warehouse Agent, you will be essential to increase operational efficiency, company profitability, and customer satisfaction by accurately and safely receiving/storing inbound... 
    Full time
    Part time
    Work experience placement
    Work at office
    Immediate start
    Flexible hours
    Night shift

    AGI

    Orlando, FL
    1 day ago
  • $15 per hour

     ...Prospect Wheelchair Agent FT- MCO DL Job Title Prospect Wheelchair Agent FT- MCO DL Date Thursday, October 15, 2026 Entity Prospect International Airport Services State Florida City Orlando Base Pay Rate: $ 15.00 Full/Part Time Full Time Exempt or Non-Exempt Position... 
    Full time
    Part time
    Shift work

    Unifi

    Orlando, FL
    5 days ago
  • $15 per hour

     ...General information Job Title Prospect Wheelchair Agent FT- MCO DL Date Thursday, October 15, 2026 Entity Prospect International Airport Services State Florida City Orlando Base Pay Rate: $ 15.00 Full/Part Time Full Time Exempt or Non-Exempt Position Non-Exempt Shift... 
    Daily paid
    Full time
    Temporary work
    Part time
    Seasonal work
    Immediate start
    Shift work
    Night shift
    Rotating shift
    Weekend work

    Prospect International Airport Services

    Orlando, FL
    3 days ago
  • $43k - $47k

     ...members to develop their careers in a dynamic and supportive environment.Dinamic AS Group is seeking a motivated and creative Advertising Agent to support advertising initiatives, promote brand awareness, and help execute marketing campaigns. This role is ideal for... 

    Alphabe Insight

    Orlando, FL
    4 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    Orlando, FL
    3 days ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Orlando, FL
    5 days ago
  •  ...Office Agents are responsible for helping customers during the shipping process by coordinating shipping details, completing compliance...  ...Regulations, etc. Release imported international freight upon review of CBP clearances and authorizations. Collect air freight... 
    Full time
    Part time
    Work experience placement
    Work at office
    Immediate start
    Flexible hours
    Night shift

    lliance Ground International

    Orlando, FL
    5 days ago
  •  ...Airport Cruise Guest Service Agent This is a fast-paced high energy part-time position requiring a positive, efficient and professional imagine. Agents will work at the airport, directly with the public and are responsible for providing service for arriving and departing... 
    Part time
    Immediate start
    Weekday work

    SMS International Shore Operations

    Orlando, FL
    6 days ago
  • $15 per hour

    Prospect Cabin Agent DayJob Title Prospect Cabin Agent Day Date Friday, October 30, 2026 Entity Prospect International Airport Services State Florida City Orlando Base Pay Rate: $ 15.00 Full/Part Time Full Time Exempt or Non-Exempt Position Non-Exempt Shift A.M. shift,... 
    Full time
    Part time
    Shift work

    Unifi

    Orlando, FL
    6 days ago
  • FIELD ASSOCIATE - ORLANDO AREA ABOUT THE ROLE We're seeking a Field Associate to join our team in the Orlando area. This contracted hourly position is a 1099 position that focuses on conducting property showings for rental properties with occasional opportunities...
    Hourly pay
    Local area
    Flexible hours

    Doorstead Inc

    Orlando, FL
    5 days ago
  •  ...Description Air General, a third-generation, family-owned, national cargo handling company, is looking for a full time Warehouse Agent - Air Cargo at our Orlando International Airport (MCO) location. The Warehouse Agent - Air Cargo will be primarily responsible... 
    Full time
    For contractors
    Flexible hours
    Shift work
    Night shift

    Air General Incorporated

    Orlando, FL
    17 days ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Winter Park, FL
    2 days ago
  •  ...Intern - State Farm Agent Team Member Francis Cabrera - State Farm Agent is seeking an organized and efficient specialist to fulfill...  ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary... 
    Full time
    For contractors
    Internship

    Francis Cabrera - State Farm Agent

    Orlando, FL
    4 days ago
  •  ...Our office is located in Winter Park and Hunter's Creek. Apply Now And Let Us Put You On The Path To Success State Farm agents are independent contractors who hire their own employees. State Farm agents' employees are not employees of State Farm. Agents are... 
    For contractors
    Internship
    Work at office
    Immediate start

    Tony Lopez

    Winter Park, FL
    3 days ago
  • $45k

     ...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs...  ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    MarCom Group

    Winter Park, FL
    3 days ago
  •  ...Booking Agent / Travel Orlando, Florida, United States We are currently seeking reliable and customer-focused individuals to join our team as Booking Agents in the travel industry. In this remote role, you will assist clients with planning and coordinating travel... 
    Remote work
    Flexible hours

    HJB Travels

    Orlando, FL
    4 days ago
  •  ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads... 
    Hourly pay
    For contractors
    Internship
    Flexible hours

    Cortney Tidwell State Farm Agent

    Orlando, FL
    3 days ago
  • Field Associate - Orlando Area We're seeking a Field Associate to join our team in the Orlando area. This contracted hourly position is a 1099 position that focuses on conducting property showings for rental properties with occasional opportunities to conduct property...
    Hourly pay
    Local area
    Flexible hours

    Doorstead

    Orlando, FL
    5 days ago
  •  ...The Federal Bureau of Investigation (FBI) is seeking highly qualified candidates for the role of special agent in Orlando, Florida. Successful candidates will utilize their STEM backgrounds to tackle national security challenges, collaborate with law enforcement professionals... 
    Work at office

    Federal Bureau of Investigation (FBI)

    Orlando, FL
    3 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Orlando, FL
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!