AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...INDEPENDENT FREIGHT AGENT R.B. Humphreys, Inc. — Rome, NY (Remote) ROLE OVERVIEW As an Independent Freight Agent with R.B. Humphreys, you will build and manage your own book of business while operating under our established authority. Backed by more than 60 years...SuggestedRemote workRelocation package
RB Humphreys
Aurora, COa month ago - A technology company in Denver seeks an AI Conversation Designer to lead the optimization of their conversational AI agents. The successful candidate will have relevant experience in AI platforms and conversation design principles to ensure seamless user experiences. Additional...Suggested
$188k
...mission to govern the world’s intelligence . We deliver the industry’s most advanced platform for Any API, Any Event, and Any AI Agent , trusted by global leaders like Michelin, Roche, and Blue Yonder. Why join us? The Mission : We are the first to bridge...SuggestedGravitee
Denver, CO6 days ago- AppFolio is hiring a Senior Machine Learning Engineer to design and ship next-gen voice and conversational AI agents within Realm-X. You will help define production voice and chat agent pipelines, collaborating across Product, Voice, and ML Platform teams to deliver reliable...Suggested
$33 per hour
...success depends on people who are committed to our purpose and have an appetite for progress. Your ‘day to day’: Perform critical review of executed batch production records while managing intermediate and finished product nonconformities and related corrective and preventive...SuggestedHourly payFull timeTemporary workWork experience placementImmediate startFlexible hoursPhilip Morris International
Aurora, COa month ago$22 - $35 per hour
...Job Description Job Description This is a remote position. Position Summary The Transcription QA Reviewer (" Proofreader/Auditor ") is responsible for the quality assurance step in a two-person production model: one transcriber produces the transcript, and a...Hourly payContract workRemote workTranslation Excellence, Inc.
Aurora, CO4 days ago- ...The Geo Group - - Responsibilities: Performs data entry and maintenance of computer files; Collects and reviews all pertinent documents associated with the legal transfer of individuals; Maintains the records for each new individual that arrives at the facility; Updates...
The GEO Group, Inc.
Aurora, CO2 days ago - ...related to the work performed. Primary Duties and Responsibilities Performs data entry and maintenance of computer files. Collects and reviews all pertinent documents associated with the legal transfer of individuals. Maintains the records for each new individual that...Work at officeWorldwideFlexible hours
The GEO Group Australia Pty Ltd.
Aurora, CO2 days ago - ...our team members. What Makes This Role Unique Rapid Growth Trajectory : We are rapidly expanding our valuation service line into new... ...oriented and technically skilled Commercial Real Estate Appraisal Reviewer to support the firm’s quality assurance and reporting...Work at officeLocal areaImmediate start
- ...Warehouse Agent - Air Cargo Air General, a third-generation, family-owned, national cargo handling company, is looking for a full... ...pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor....Full timeWork at officeFlexible hoursShift workNight shift
Air General
Aurora, CO8 hours ago - ...individuals with disabilities to perform the essential functions. These duties and responsibilities will be rated on the Annual Performance Review. Transfer mail tubs from loading docks to mail room in the morning and mailroom to dock in the afternoon (if needed). Sort...Hourly payFor contractorsWork at officeFlexible hoursDay shift
- ...procedures. * Sort, categorize, label, scan, and upload documents as needed. * Retrieve records and documents promptly when requested. * Review files for accuracy, completeness, and proper organization. * Enter and update document and record information in company systems....Full timePart timeWork at officeFlexible hours
Handal's Auto Repair
Aurora, CO2 days ago - GEO Secured Services in Aurora, Colorado seeks a detail-oriented clerical specialist to support facility records with discretion and confidentiality. The role emphasizes accurate data entry, maintaining detainee files, and processing documents related to transfers. You ...Work at office
$51.63k - $92.22k
...Border Patrol Agent Location: Various U.S. Border Patrol Duty Stations (Relocation May Be Required) Job Type: Full-Time | Federal Law Enforcement Career Job Summary: The U.S. Border Patrol (USBP) is seeking entry-level candidates to join as Border Patrol...Full timeWork experience placementRelocationMarCom Group
Aurora, CO2 days ago$17.5 per hour
...Assists students with registration processes via Navigate. Banner and Self-Service Banner. English and Math Self-Assessments. Help review prerequisite errors, time conflict errors, and Department Approval Required Errors to effectively communicate the registration...Hourly payWork at office- The Dixon Group is seeking a proactive Buyer’s Agent in the United States to convert pre-approved buyers into signed agreements. You will show homes, draft offers, and negotiate strategically, all while owning a fast-moving pipeline and hitting weekly conversion targets...
- Company DescriptionWork with Us. Change the World.At AECOM, we're delivering a better world. Whether improving your commute, keeping the lights on, providing access to clean water, or transforming skylines, our work helps people and communities thrive. We are the world'...Work experience placementLocal areaWorldwideFlexible hours
- The State of Colorado is seeking a Human Resources Assistant for a 9-month or less assignment in Denver. The role focuses on processing personnel actions, maintaining HR records, and supporting selection activities in a confidential environment. Responsibilities include...
- Wagner Equipment Co. is seeking a Finance Coordinator to review loan applications and financing orders for lender submission. The role includes UCC filings, lease/payoff documentation, and support across finance processes. Strong administrative and customer service skills...
- ServiceSource, a nonprofit agency operating within the AbilityOne program, seeks a mailroom clerk to manage mail production and distribution. The role includes sorting, researching, and delivering interoffice mail with adherence to project timelines and accuracy. Qualifications...Work at office
- ServiceSource, Inc. is seeking a mailroom clerk to handle mail production and distribution within guidelines. The role involves sorting, researching, and delivering interoffice mail, with routine tasks and occasional travel of mail tubs between docks and mailroom. Full-...Full timeWork at office
- GEO Secured Services is seeking a clerical support member to maintain accurate detainee records and assist with legal transfer documentation. The role emphasizes discretion, confidentiality, and teamwork within a correctional facility. Responsibilities include data entry...
- Coinbase is hiring a Senior Software Engineer for the AI Platform team. You will build and operate LLM and agent infrastructure used company-wide, shaping how Coinbase scales AI. You’ll lead end-to-end platform initiatives, from gateway and runtimes to knowledge bases...Remote job
$222k - $242k
...about our Total Rewards philosophy . AI is a fundamental part of how work gets... ...the Role: We're looking for a GTM AI Agent Builder to build, implement, and support... ...personal information very seriously. Please review our Fraudulent Activity Disclaimer ....Remote jobFull timeWork at officeLocal area2 days per week3 days per week$55.54k - $57.82k
...same, however a typical day in the life of an Asset Protection Agent will likely include the following responsibilities: Resolve theft... ...related to this job posting that are specific to your state. Review the Career Site FAQ’s for relevant information and guidelines....Hourly payFull timeLocal areaFlexible hours- ...orientation and learning documentation Partner with subject matter experts to gather requirements and document workflows and systems Review and update documentation to reflect system, process, or policy changes Apply documentation standards, templates, and style...
LauraMac, LLC
Denver, CO4 days ago - ...looking for growth, support, and a seriously fun environment, we’d love to meet you! ROLE DESCRIPTION: Ryan Vanderveen - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Work at office
Ryan Vanderveen - State Farm Agent
Aurora, CO6 days ago - ...difference, and be part of a supportive community, we’d love to have you join our team! ROLE DESCRIPTION: Justin Stearns - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Work at officeLocal areaFlexible hours
Justin Stearns - State Farm Agent
Aurora, CO10 days ago - ...is a local CPA firm specializing in Special District accounting and auditing. The goal of this position is to have a detailed review of all financial statements and reports before going to the CPA for final review and signature. The candidate should have excellent...Work at officeLocal areaFlexible hours
Simmons & Wheeler, P.C.
Englewood, CO6 days ago $20 per hour
...needed for various projects Organize, print and compile bid documents Provide travel administrative support and coordination Reviews, reconciles and processes staff credit card statements Monitors visitor access and issues passes when required Researches,...Hourly payWork at officeTAKRAF
Denver, CO2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
