AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift$10k
...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift- Benefits: Bonus based on performance Competitive salary Paid time off You May Be a Great Fit as a Multiline Representative at Jeff Gotch - State Farm Insurance Agency if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You...SuggestedWork at office
- ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The...SuggestedRemote jobHourly payContract workFreelanceCurrently hiringLocal areaFlexible hours
- Markel Corporation is seeking a Document Management Technician to execute prescribed performance goals within the Specialty Division. You will process incoming documents for underwriting, audit, and claims, while maintaining strong customer service via email and adhering...
- Our client is seeking a skilled Document Control Specialist for a 6-month contract opportunity with Northern Natural Gas. This position is onsite, offers a competitive hourly pay rate, and candidates must have the legal right to work in the US without sponsorship. Interested...Hourly payContract work
- ...directly with the world's largest technology companies as a QAQC Agent. In this role, you'll be on-site at active construction projects... ...we use it in connection with your general access and use of our website, please review our general California Privacy Notice here....Full timeFlexible hours
- ...directly with the world's largest technology companies as a QAQC Agent. In this role, you'll be on-site at active construction projects... ...use it in connection with your general access and use of our website, please review our general California Privacy Notice here ( ....Full timeFlexible hours
$48.85k
...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step...$93.75k - $133.2k
...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks... ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate start- WE’RE CHANGING ENTERTAINMENT. COME JOIN US. We’re always looking for talent that believes in having fun. At PENN Entertainment, you’ll get to be a part of an exciting industry, where the days and nights are fast paced. You’ll work with an incredibly diverse set of coworkers...Work at officeLocal areaFlexible hoursShift workNight shiftAfternoon shift
$103.24k - $133.2k
...position advertised has been exempted from the federal civilian hiring freeze. Use your mathematics background to become an FBI special agent! The transition from data to special agent is more than a career move—it’s an opportunity to upskill and become an expert in your...Work experience placementWork at officeLocal area- ...United StatesAbout the RoleWe are seeking an Online Cruise Booking Agent to assist clients with researching, selecting, and booking... ...help travelers explore cruise options, understand itineraries, review requirements, and coordinate all booking details. You will ensure...Remote workFlexible hours
- ...Overview Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed.Provide prompt, accurate, and...Hourly payPart timeFor contractorsInternshipWork at officeFlexible hours
$45k
...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs... ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before...Full timeWork experience placement- Property Description At The Cottonwood, we’re more than just a boutique hotel — we’re a proud team of passionate people who care deeply about what we do, how we do it, and the impact we make in our community. Located in Omaha's vibrant Blackstone District, where history...Local areaFlexible hoursNight shift
$103.24k - $133.2k
The Federal Bureau of Investigation (FBI) is seeking candidates for the role of special agent. This opportunity allows individuals with a psychology background to apply their skills in investigative work that protects our nation from various threats. The position demands...Work at office$40k - $60k
...Future Opening: Office Representative - State Farm Agent Team Member Full Time • Omaha - NE Position Overview Successful State Farm Agent... ...potential. Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Full timeFor contractorsWork at office$35k - $60k
...Future Opening: Office Representative - State Farm Agent Team Member Full Time • Omaha - NE Replies within 24 hours Position... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Full timeFor contractorsWork at officeFlexible hours$40k - $60k
...Office Representative - State Farm Agent Team MemberSuccessful State Farm Agent is seeking a qualified professional to join their winning... ...potential.ResponsibilitiesUse a customer-focused, needs-based review process to educate customers about insurance options.Develop...Hourly payFor contractorsWork at office$35k - $60k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...For contractorsWork at officeFlexible hours$35k - $60k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads...For contractorsWork at officeFlexible hours- ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The...Remote jobContract workCurrently hiringLocal areaFlexible hours
- ...workflows, and maintaining compliance with internal and client-specific standards. Responsibilities Assist with receiving, reviewing, and processing project documents such as drawings, specifications, submittals, RFIs, change orders, and correspondence. Maintain...Full timeWork at officeFlexible hours
- ...confidence in the insights they rely on. For HDR, it means enterprise and project data that is ready to support analytics, automation, and AI in a way that reflects how we work.This role is about making data simpler, more useful, and more connected to the way we do business,...Local area
- ...opportunity to plan for a brighter future. I’ve had the honor of serving my community for the past 14 years, and being a State Farm agent has truly changed my life both personally and professionally. I was born in Mexico City and moved to Omaha in the mid-90s. Most of my...Work at officeFlexible hours
$40k - $60k
...willingness to learn, we should meet to discuss our career opportunity. Come work with an energetic, fun team at Bob Speck - State Farm Agent! About Our Agency Our mission is to help people manage the risks of everyday life, recover from the unexpected and realize their...For contractorsWork experience placementWork at office$17 - $22.96 per hour
...support our recruiting process, we may use artificial intelligence ("AI") or automated technologies to collect, organize, sort, and... ...tool. Hiring decisions are made by Associated Bank personnel who review applicant information and exercise independent judgment in accordance...Hourly payNight shift- **If you are a current FIB employee, please apply through the Career Worklet in the Employee Portal This position is located at our Bellevue, NE Galvin Rd. S. branch. What's Important to You We know your career is just one aspect of a meaningful, complex, ...Night shift
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!



