AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$45.3 - $78.6 per hour
...each other.CLA is seeking to hire an experienced Seasonal Tax Reviewer to join our talented tax team. The position offers flexibility... ...or related field is preferred.CPA is required (JD or Enrolled Agent is acceptable in lieu).Our approach to compensation emphasizes...SuggestedCasual workSeasonal work$40 - $60 per hour
...Job details Senior Tax Review, Contract, Remote, based out of Orlando CPA Firm, reply to ****@*****.*** Salary $40 - $60 per hour A Bit About Us Growing CPA firm - Downtown Orlando Why join us? ~ Remote ~ Medical ~ Dental ~ Vision ~401k ~ Generous...SuggestedHourly payContract workWork at officeLocal areaRemote work- Employment Opportunities At Full Sail University If you're looking for the chance to learn, grow, and make a contribution to the community, look at employment opportunities with Full Sail University. You'll find your choice of career opportunities, great benefits, an environment...SuggestedWork experience placementWork at officeFlexible hoursNight shiftWeekend work
$51.63k - $124.72k
...Job Description Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift- Online Document SpecialistThe Online Document Specialist is responsible for the receiving, coding, reviewing, auditing, maintenance and management of incoming student documents, files, and enrollment data affecting various departments including admissions, financial aid...SuggestedWork at office
- A construction management firm in Orlando, FL is seeking a Document Controls Specialist to manage project documentation and ensure compliance with standards. The ideal candidate will have a degree in construction management, architecture, or engineering, along with a minimum...
$19 per hour
...while maintaining speed and accuracy goals Research and collect item details and specifications for online listing preparation Review and verify inventory documentation submitted by other team members before system uploads Audit inventory, perform cycle counts,...Local areaMonday to Friday$55k - $70k
Overview Document Controls Specialist - City/State Programs 3 days ago Be among the first 25 applicants This range is provided by Procon Consulting. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more. Base pay range $55,...Full timeWorldwide$40 - $45 per hour
RN QA Reviewer - OASIS & Chart Review (Per Diem, Remote) Do you love the moment a chart finally tells the whole story, accurately and completely? Do you find yourself spotting the one item that doesn't match the visit note? If so, we would love to meet you. ABOUT REVITALIZE...Remote jobDaily paidReliefWork from homeFlexible hours- Revitalize Home Healthcare seeks an experienced RN QA Reviewer - OASIS & Chart Review for a remote, per diem role in Florida. You will review OASIS assessments and clinical documentation before submission, ensuring the stories align across records to protect patients and...Remote jobDaily paid
- ...Warehouse AgentDo you enjoy working in a fast-paced, safety-obsessed aviation environment? As a Warehouse Agent, you will be essential to increase operational efficiency, company profitability, and customer satisfaction by accurately and safely receiving/storing inbound...Full timePart timeWork experience placementWork at officeImmediate startFlexible hoursNight shift
$15 per hour
...Prospect Wheelchair Agent FT- MCO DL Job Title Prospect Wheelchair Agent FT- MCO DL Date Thursday, October 15, 2026 Entity Prospect International Airport Services State Florida City Orlando Base Pay Rate: $ 15.00 Full/Part Time Full Time Exempt or Non-Exempt Position...Full timePart timeShift work$15 per hour
...General information Job Title Prospect Wheelchair Agent FT- MCO DL Date Thursday, October 15, 2026 Entity Prospect International Airport Services State Florida City Orlando Base Pay Rate: $ 15.00 Full/Part Time Full Time Exempt or Non-Exempt Position Non-Exempt Shift...Daily paidFull timeTemporary workPart timeSeasonal workImmediate startShift workNight shiftRotating shiftWeekend work$43k - $47k
...members to develop their careers in a dynamic and supportive environment.Dinamic AS Group is seeking a motivated and creative Advertising Agent to support advertising initiatives, promote brand awareness, and help execute marketing campaigns. This role is ideal for...$48.85k
...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step...$93.75k - $133.2k
...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks... ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate start- ...Office Agents are responsible for helping customers during the shipping process by coordinating shipping details, completing compliance... ...Regulations, etc. Release imported international freight upon review of CBP clearances and authorizations. Collect air freight...Full timePart timeWork experience placementWork at officeImmediate startFlexible hoursNight shift
- ...Airport Cruise Guest Service Agent This is a fast-paced high energy part-time position requiring a positive, efficient and professional imagine. Agents will work at the airport, directly with the public and are responsible for providing service for arriving and departing...Part timeImmediate startWeekday work
$15 per hour
Prospect Cabin Agent DayJob Title Prospect Cabin Agent Day Date Friday, October 30, 2026 Entity Prospect International Airport Services State Florida City Orlando Base Pay Rate: $ 15.00 Full/Part Time Full Time Exempt or Non-Exempt Position Non-Exempt Shift A.M. shift,...Full timePart timeShift work- FIELD ASSOCIATE - ORLANDO AREA ABOUT THE ROLE We're seeking a Field Associate to join our team in the Orlando area. This contracted hourly position is a 1099 position that focuses on conducting property showings for rental properties with occasional opportunities...Hourly payLocal areaFlexible hours
- ...Description Air General, a third-generation, family-owned, national cargo handling company, is looking for a full time Warehouse Agent - Air Cargo at our Orlando International Airport (MCO) location. The Warehouse Agent - Air Cargo will be primarily responsible...Full timeFor contractorsFlexible hoursShift workNight shift
$10k
...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift- ...Intern - State Farm Agent Team Member Francis Cabrera - State Farm Agent is seeking an organized and efficient specialist to fulfill... ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary...Full timeFor contractorsInternship
- ...Our office is located in Winter Park and Hunter's Creek. Apply Now And Let Us Put You On The Path To Success State Farm agents are independent contractors who hire their own employees. State Farm agents' employees are not employees of State Farm. Agents are...For contractorsInternshipWork at officeImmediate start
$45k
...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs... ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before...Full timeWork experience placement- ...Booking Agent / Travel Orlando, Florida, United States We are currently seeking reliable and customer-focused individuals to join our team as Booking Agents in the travel industry. In this remote role, you will assist clients with planning and coordinating travel...Remote workFlexible hours
- ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads...Hourly payFor contractorsInternshipFlexible hours
- Field Associate - Orlando Area We're seeking a Field Associate to join our team in the Orlando area. This contracted hourly position is a 1099 position that focuses on conducting property showings for rental properties with occasional opportunities to conduct property...Hourly payLocal areaFlexible hours
- ...The Federal Bureau of Investigation (FBI) is seeking highly qualified candidates for the role of special agent in Orlando, Florida. Successful candidates will utilize their STEM backgrounds to tackle national security challenges, collaborate with law enforcement professionals...Work at office
$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!


