Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Denver, CO vacancy
  • $20 - $30 per hour

     ...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent...  ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every...  ...at yet and judge it step by step. • Review — Take an existing annotation, written... 
    Suggested
    Hourly pay
    Full time
    Contract work
    Immediate start
    Remote work

    Bespoke Labs

    Thornton, CO
    4 days ago
  • $50 per hour

     ...Earn Up To $50/hr as an OpenClaw Agent Builder Shape the future of autonomous agents...  ...completely free ~ Over $200/month in AI access, included at no extra cost. Overview...  ...projects offer incentive payments. Please review the payment terms for each project.... 
    Suggested
    Weekly pay
    Extra income
    Contract work
    Freelance
    Internship
    Remote work
    Worldwide

    Outlier

    Denver, CO
    3 days ago
  • $175k - $190k

     ...what doesn't), and help evolve how we build AI-powered software. We are excited about...  ...Nice to have: Exposure to LLM APIs or agent frameworks Experience building AI-enabled...  ...that changed how the team approaches code review Mentored a Senior Engineer through... 
    Suggested
    Full time
    Work at office
    Work visa
    Shift work
    3 days per week

    Focused

    Denver, CO
    2 days ago
  • $55k - $70k

     ...embedded logos, pictures, graphics, and maps QA/QC: perform detailed review for errors, consistency, and professionalism Production: prepare...  ...to be used by other technical staff Innovation: lead research on AI tools and products for document production Applicants must have... 
    Suggested
    Temporary work
    Live in
    Flexible hours

    SE Group

    Denver, CO
    22 hours ago
  • $53.4 - $90.7 per hour

     ...Seasonal Tax Reviewer CLA is a top 10 national professional services firm where our purpose is to create opportunities every day, for...  ...field is preferred. ~ CPA is required (JD or Enrolled Agent is acceptable in lieu). Our approach to compensation emphasizes... 
    Suggested
    Remote job
    Casual work
    Seasonal work

    CliftonLarsonAllen

    Denver, CO
    1 day ago
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Remote job
    Hourly pay
    Contract work
    Freelance
    Currently hiring
    Local area
    Flexible hours

    Signup Expert

    Denver, CO
    5 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    U.S. Customs and Border Protection

    Aurora, CO
    22 hours ago
  •  ...is a local CPA firm specializing in Special District accounting and auditing. The goal of this position is to have a detailed review of all financial statements and reports before going to the CPA for final review and signature. The candidate should have excellent... 
    Work at office
    Local area
    Flexible hours

    Simmons & Wheeler, P.C.

    Englewood, CO
    29 days ago
  • $213k - $236k

     ...Learn more about ourTotal Rewards philosophy.AI is a fundamental part of how work gets...  ...As a Senior Staff Software Engineer on the Agent & Partner Experiences team, you'll be a key...  ...information very seriously. Please review our Fraudulent Activity Disclaimer.Personal... 
    Full time
    Work at office
    Local area
    Remote work
    2 days per week
    3 days per week

    Gusto

    Denver, CO
    1 day ago
  • Kiewit Infrastructure Engineers is seeking an Entry Level Document Controls Specialist to assist with management of engineering documents in accordance with government and company regulations on multiple design projects. You will handle distribution workflows, perform quality...
    Remote job

    Kiewit Corporation

    Littleton, CO
    4 days ago
  • Sky Health Care Management is seeking a File Clerk for an onsite or remote-capable position in Colorado. You will organize, file, digitize, and securely store documents while supporting clerical tasks. The role requires a high school diploma, basic computer skills, and ...
    Remote job
    Hourly pay

    Sky Health Care Management

    Aurora, CO
    4 days ago
  • $32k - $40.88k

    Cedar South Solutions, a mid-sized IT services company in Denver, CO, seeks a File Clerk for a FULL_TIME position. The role offers a salary range of $32,000 - $40,875 and is available until November 23, 2026. As a File Clerk, you will be responsible for maintaining our...
    Full time
    Work at office

    Cedar South Solutions

    Denver, CO
    5 days ago
  • $44k - $56.43k

     ...it is successfully filled. The File Clerk will scan and file documents, assist with data entry tasks, and submit applications for review. This role requires experience in file management and administrative tasks, along with proficiency in office software and attention... 
    Permanent employment
    Full time
    Work at office

    Magnolia Cassia

    Denver, CO
    1 day ago
  • $20.11 - $32.99 per hour

     ...distribute files when requested by authorized personnel. Update records and ensure documents are stored in the correct locations. Review files for missing information, duplicate records, or filing errors. Maintain document logs and tracking systems. Protect confidential... 
    Remote job
    Hourly pay
    Work at office

    Sky Health Care Management

    Aurora, CO
    5 days ago
  • File Maintenance Clerk A Day in the Life: As a File Maintenance Clerk, you'll play a key role in ensuring pricing accuracy and signage compliance throughout the store. You'll be responsible for implementing corporate price changes, maintaining pricing records, and ensuring...
    Weekly pay
    Minimum wage
    Local area
    Flexible hours
    Shift work
    Day shift
    Afternoon shift

    Vons

    Littleton, CO
    2 days ago
  • File Maintenance ClerkA Day in the Life: As a File Maintenance Clerk, you'll play a key role in ensuring pricing accuracy and signage compliance throughout the store. You'll be responsible for implementing corporate price changes, maintaining pricing records, and ensuring...
    Shift work
    Day shift
    Afternoon shift

    Balducci's Food Lover's Market

    Littleton, CO
    4 days ago
  • $17.5 per hour

     ...Assists students with registration processes via Navigate. Banner and Self-Service Banner. English and Math Self-Assessments. Help review prerequisite errors, time conflict errors, and Department Approval Required Errors to effectively communicate the registration... 
    Hourly pay
    Work at office

    Colorado Community College

    Aurora, CO
    2 days ago
  • TekWissen provides a unique portfolio of innovative capabilities that seamlessly combines clients insights, strategy, design, software engineering, and systems integration. Job Description Job Summary Under minimal supervision, maintains Device Master Record and Device ...

    TekWissen LLC

    Westminster, CO
    3 days ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Denver, CO
    4 days ago
  •  ...Schedule Role Summary This Entry-Level Remote Booking Agent role focuses on assisting clients with hotel...  ...performance We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses... 
    Remote work
    Flexible hours

    Work From Home Travel Careers

    Denver, CO
    22 hours ago
  •  ...Job Type: Remote | Flexible ScheduleRole SummaryThis Entry-Level Remote Booking Agent role focuses on assisting clients with hotel reservations and travel-related inquiries. No prior travel industry experience is required — we provide training, tools, and support to help... 
    Remote work
    Flexible hours

    Destination Careers

    Denver, CO
    22 hours ago
  •  ...We are currently seeking organized and customer-focused individuals to join our team as a Booking Agent . This role involves assisting clients with coordinating reservations and providing professional support throughout the booking process. The ideal candidate enjoys... 
    Remote work

    Remote Careers

    Denver, CO
    1 day ago
  • Kiewit Infrastructure Engineers seeks a Document Controls Specialist to support the management of engineering documents in compliance with government and company regulations on permanent design projects. This entry-level role is based in Lone Tree, CO or Omaha, NE and offers...
    Permanent employment

    Kiewit

    Littleton, CO
    3 days ago
  • Job Description A Day in the Life: As a File Maintenance Clerk, you'll play a key role in ensuring pricing accuracy and signage compliance throughout the store. You'll be responsible for implementing corporate price changes, maintaining pricing records, and ensuring all...
    Weekly pay
    Minimum wage
    Local area
    Flexible hours
    Shift work
    Day shift
    Afternoon shift

    Albertsons

    Littleton, CO
    1 day ago
  •  ...position is 100% performance-commission based, offering total schedule independence, uncapped earning capabilities, and a direct path to scaling your own business. Ready to leave expensive, low-quality leads behind? Submit your application today to review the program!... 
    Contract work
    Remote work

    Beacon National Agency

    Denver, CO
    1 day ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    Denver, CO
    22 hours ago
  • $20.25 per hour

     ...that mission starts on the ground.  As an Aircraft Appearance Agent at Denver International Airport, you help prepare each aircraft...  ...pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor.... 
    Hourly pay
    Full time
    Part time
    Local area
    Shift work
    Night shift
    Day shift
    Afternoon shift

    Frontier Airlines

    Denver, CO
    1 day ago
  •  ...The Booking Agent will represent clients, seeking performance or appearance opportunities and negotiating on their behalf. They work closely with event organizers, promoters, and venues to secure engagements that align with the artists or talents career goals. The agent... 
    Contract work
    Remote work

    Magical Destinations Travel

    Denver, CO
    4 days ago
  •  ...Tax Site Preparer and/or Quality Reviewer The International Rescue Committee (IRC) is seeking Volunteer Tax Preparers to provide free tax preparation services for low-income clients through its National Volunteer Income Tax Assistance (VITA) program. Volunteers will... 
    Work at office
    Work from home
    Afternoon shift

    Impactpool

    Denver, CO
    1 day ago
  • $35k - $55k

     ...Insurance Staff Position - State Farm Agent Team MemberSuccessful State Farm Agent is seeking a qualified professional to join their...  ...and friendly customer service.Use a customer-focused, needs-based review process to educate customers about insurance options.Work with the... 
    For contractors
    Work at office

    Jesus Cruz - State Farm Agent

    Denver, CO
    22 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!