Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Colorado Springs, CO vacancy
  • $50 per hour

    About the company Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing,...  ...eligible US states No patient contact, no malpractice exposure: you review written clinical cases All contributors are license-verified... 
    Suggested
    Part time
    For contractors
    Remote work
    Flexible hours

    Mindrift

    Colorado Springs, CO
    2 days ago
  • $35k - $47.55k

    Atlas National Corp. is currently seeking a dedicated File Clerk to join their team in Colorado Springs, CO. This full-time position is crucial for organizing and managing a wide array of financial documents, ensuring their accuracy and compliance with the company's established...
    Suggested
    Full time
    Work at office

    Atlas National Corp

    Colorado Springs, CO
    2 days ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Suggested
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Colorado Springs, CO
    2 days ago
  • $15.16 per hour

     ...Wheelchair Agent Hot Job COLORADO SPRINGS, CO 80916 Start Date 01/14/2025 End Date 12/31/2026 Overview Salary Range $15.16 - $15.16 Hourly Description Job Title: Passenger Service Attendant – Wheelchair Agent Reports To: Supervisor/Manager/Dispatcher... 
    Suggested
    Hourly pay
    Shift work

    Huntleigh USA

    Colorado Springs, CO
    1 day ago
  • $17 per hour

     ...Job Description Job Description General information Job Title Aircraft Cabin Agent - COS Date Monday, October 5, 2026 City Colorado Springs State Colorado Base Pay Rate: $ 17.00 Full/Part Time Part... 
    Suggested
    Daily paid
    Full time
    Part time
    Monday to Friday
    Flexible hours
    Shift work
    Night shift
    Weekend work
    Day shift

    ERMC

    Colorado Springs, CO
    3 days ago
  • $16 - $25 per hour

     ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ..., and billing clarification.Use a customer-focused, needs-based review process to educate customers about insurance options.As an Agent... 
    Hourly pay
    For contractors
    Immediate start

    Reeves Maddox - State Farm Agent

    Colorado Springs, CO
    2 days ago
  • $16 - $25 per hour

     ...Future Opening: Insurance Staff Position - State Farm Agent Team Member Position Overview Are you outgoing and customer-focused...  ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options.... 
    Hourly pay
    For contractors

    Reeves Maddox

    Colorado Springs, CO
    1 day ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Colorado Springs, CO
    3 days ago
  • $47k - $70k

     ...Position Overview Successful State Farm Agent is seeking a qualified professional to join their winning team for the role of Intern...  .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work with... 
    For contractors
    Internship
    Work at office
    Immediate start

    State Farm

    Colorado Springs, CO
    7 hours ago
  •  ...Final Expense Agent Opportunity Losing a loved one is already difficult. The last thing a family should have to worry about is how they're going to pay for funeral and final expenses. As a Final Expense Agent, you have the opportunity to help families prepare for... 
    Remote work

    GIA Legacy Planning

    Colorado Springs, CO
    4 days ago
  •  ...Final Expense Agent We're excited to invite ambitious, friendly go-getters to join our dynamic team as Final Expense Agents. This isn't a cookie-cutter jobit's a modern career path built for people who want flexibility, purpose, and real earning potential. You'll have... 
    Immediate start
    Remote work
    Flexible hours

    GIA Legacy Planning

    Colorado Springs, CO
    3 days ago
  • $19.15 - $21.95 per hour

     ...Contact Center Agent Some calls are simple. Someone forgot a password. A debit card isnt working. A payment needs to get there today. Other calls are bigger. A member is stressed, confused, or trying to make a smart money move and needs someone who can listen, explain... 
    Hourly pay
    Full time
    Local area
    Monday to Friday

    Air Academy Credit Union

    Colorado Springs, CO
    2 days ago
  •  ...Airport Operations Agent - COS (PT) Unifi Aviation, LLC is seeking individuals to join our growing team who are passionate, have integrity, and have empathy for how their words and decisions affect our employees, customers, and partners. No experience is required. Paid... 
    Hourly pay
    Weekly pay
    Full time
    Seasonal work
    Immediate start
    Night shift
    Rotating shift
    Weekend work

    Unifi

    Colorado Springs, CO
    1 day ago
  • $51.63k - $124.72k

    Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Manitou Springs, CO
    2 days ago
  •  ...GROUND SERVICE AGENT Must be able to work a flexible schedule including nights, weekends, and holidays. Hours will vary based upon flight schedules. Number of weekly hours may increase or decrease due to seasonal flight changes. GENERAL PURPOSE OF THE JOB:... 
    Seasonal work
    Immediate start
    Flexible hours
    Shift work
    Night shift

    Trego/Dugan Aviation

    Colorado Springs, CO
    4 days ago
  •  ...role At Connie Health, we're revolutionizing the way older Americans navigate Medicare, but we're doing it together. As an independent agent, you'll have the freedom to build your own business with the full support of our team. We've built a culture of collaboration,... 
    Hourly pay
    Shift work

    Connie Health

    Colorado Springs, CO
    3 days ago
  • Connie Health is recruiting licensed agents to sell Medicare and Life on a commission-only basis. You’ll receive training through Connie Academy, an onboarding stipend, and ongoing coaching, with leads provided and an internal support team handling post-sale questions.... 
    Remote work
    Work from home

    Connie Health

    Colorado Springs, CO
    5 days ago
  • $31.49k - $68.87k

     ...brief phone interviews for this position. Must be able to commit at least 10-15 hours per week. Experience as an Independent Insurance Agent, consultant, customer service, health care services, sales, and insurance, Mortgage Broker, nursing or teaching background is a plus... 
    Full time
    Part time
    Home office
    10 hours per week

    CJC Benefits LLC (2)

    Colorado Springs, CO
    2 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Linkup Inc

    Colorado Springs, CO
    7 hours ago
  • About the job Work from Home Booking Agent We are looking for candidates who have strong interpersonal and communication skills for our organization as a Remote booking agent. The role of a booking agent involves providing support and guidance to others. The applicant must... 
    Remote work
    Work from home

    Four Point Getaways

    Colorado Springs, CO
    5 days ago
  •  ...opportunity to experience first‑hand what it takes to be a State Farm Agent. You will build and develop customer relationships within the...  ...meet marketing goals. Use a customer‑focused, needs‑based review process to educate customers about insurance options. Maintain... 
    Hourly pay
    For contractors
    Work at office

    Kendra Gries

    Colorado Springs, CO
    4 days ago
  • $88k - $132k

     ...Agency Manager at Mike Wuest - State Farm Agent At Mike Wuest - State Farm Agent, we're here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support. Whether we're helping someone navigate a claim or personalize... 
    For contractors
    Work at office
    Immediate start

    Mike Wuest - State Farm Agent

    Colorado Springs, CO
    2 days ago
  • $51.63k - $124.72k

     ...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S...  ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    Marcom Group

    Colorado Springs, CO
    7 days ago
  • $51.63k - $124.72k

    Job Description Border Patrol Agent (BPA)\n With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Colorado Springs, CO
    4 days ago
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Remote job
    Contract work
    Currently hiring
    Local area
    Flexible hours

    signup-expert

    Colorado Springs, CO
    4 days ago
  •  ...& development ABOUT OUR AGENCY: I’ve been a State Farm agent since 2008, and today our team includes five experienced professionals...  ...meet marketing goals. Use a customer-focused, needs-based review process to educate customers about insurance options.... 
    For contractors
    Work at office
    Flexible hours

    Chris Hard - State Farm Agent

    Colorado Springs, CO
    29 days ago
  • $55k - $75k

     ...Products Representative - State Farm Agent Team Member At Chris Hard - State Farm Agent, we are looking for a motivated Products Representative - State Farm Agent Team Member who is eager to contribute to the continued success of our agency. In this role, you'll engage... 

    Chris Hard - State Farm Agent

    Colorado Springs, CO
    2 days ago
  • $50k - $100k

     ...Reeves Maddox - State Farm Agent Opportunity Do you have aspirations to run your own business? If so, you may want to consider working...  .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work with... 
    For contractors
    Work at office
    Immediate start

    Reeves Maddox - State Farm Agent

    Colorado Springs, CO
    5 days ago
  •  ...development Bonus based on performance Competitive salary You May Be a Great Fit as an Agency Manager at Mike Wuest - State Farm Agent if:  You take ownership and follow through until the issue is fully resolved. You communicate clearly and confidently , even... 
    Work at office

    Mike Wuest - State Farm Agent

    Colorado Springs, CO
    8 days ago
  • $30k - $48k

     ...Performs accurate data entry into internal systems and databases. Reviews documents for quality accuracy completeness and compliance with...  ...to verify your identity. Use of artificial intelligence (AI) tools of any kind during Peraton interviews is strictly prohibited... 
    Full time
    Contract work
    Temporary work
    Work at office
    Shift work

    Peraton

    Colorado Springs, CO
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!