Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Bellevue, NE vacancy
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    U.S. Customs and Border Protection

    Bellevue, NE
    1 day ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Bellevue, NE
    4 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    Adzuna

    Bellevue, NE
    1 day ago
  • Benefits: Bonus based on performance Competitive salary Paid time off You May Be a Great Fit as a Multiline Representative at Jeff Gotch - State Farm Insurance Agency if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You...
    Suggested
    Work at office

    Jeff Gotch

    Bellevue, NE
    1 day ago
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Suggested
    Remote job
    Hourly pay
    Contract work
    Freelance
    Currently hiring
    Local area
    Flexible hours

    Signup Expert

    Omaha, NE
    6 days ago
  • Markel Corporation is seeking a Document Management Technician to execute prescribed performance goals within the Specialty Division. You will process incoming documents for underwriting, audit, and claims, while maintaining strong customer service via email and adhering...

    Markel Service Inc.

    Omaha, NE
    2 days ago
  • Our client is seeking a skilled Document Control Specialist for a 6-month contract opportunity with Northern Natural Gas. This position is onsite, offers a competitive hourly pay rate, and candidates must have the legal right to work in the US without sponsorship. Interested...
    Hourly pay
    Contract work

    BizTek People, Inc.

    Omaha, NE
    3 days ago
  •  ...directly with the world's largest technology companies as a QAQC Agent. In this role, you'll be on-site at active construction projects...  ...we use it in connection with your general access and use of our website, please review our general California Privacy Notice here.... 
    Full time
    Flexible hours

    Olsson

    Omaha, NE
    2 days ago
  •  ...directly with the world's largest technology companies as a QAQC Agent. In this role, you'll be on-site at active construction projects...  ...use it in connection with your general access and use of our website, please review our general California Privacy Notice here ( .... 
    Full time
    Flexible hours

    Olsson

    Omaha, NE
    4 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    Omaha, NE
    1 day ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Omaha, NE
    5 days ago
  • WE’RE CHANGING ENTERTAINMENT. COME JOIN US. We’re always looking for talent that believes in having fun. At PENN Entertainment, you’ll get to be a part of an exciting industry, where the days and nights are fast paced. You’ll work with an incredibly diverse set of coworkers...
    Work at office
    Local area
    Flexible hours
    Shift work
    Night shift
    Afternoon shift

    Ameristar Casino

    Council Bluffs, IA
    5 days ago
  • $103.24k - $133.2k

     ...position advertised has been exempted from the federal civilian hiring freeze. Use your mathematics background to become an FBI special agent! The transition from data to special agent is more than a career move—it’s an opportunity to upskill and become an expert in your... 
    Work experience placement
    Work at office
    Local area

    Federal Bureau of Investigation (FBI)

    Omaha, NE
    4 days ago
  •  ...United StatesAbout the RoleWe are seeking an Online Cruise Booking Agent to assist clients with researching, selecting, and booking...  ...help travelers explore cruise options, understand itineraries, review requirements, and coordinate all booking details. You will ensure... 
    Remote work
    Flexible hours

    HB Travels

    Omaha, NE
    5 days ago
  •  ...Overview Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed.Provide prompt, accurate, and... 
    Hourly pay
    Part time
    For contractors
    Internship
    Work at office
    Flexible hours

    Sherman Willis State Farm Agent

    Omaha, NE
    2 days ago
  • $45k

     ...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs...  ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    MarCom Group

    Omaha, NE
    1 day ago
  • Property Description At The Cottonwood, we’re more than just a boutique hotel — we’re a proud team of passionate people who care deeply about what we do, how we do it, and the impact we make in our community. Located in Omaha's vibrant Blackstone District, where history...
    Local area
    Flexible hours
    Night shift

    The Cottonwood Hotel

    Omaha, NE
    2 days ago
  • $103.24k - $133.2k

    The Federal Bureau of Investigation (FBI) is seeking candidates for the role of special agent. This opportunity allows individuals with a psychology background to apply their skills in investigative work that protects our nation from various threats. The position demands... 
    Work at office

    Federal Bureau of Investigation (FBI)

    Omaha, NE
    5 days ago
  • $40k - $60k

     ...Future Opening: Office Representative - State Farm Agent Team Member Full Time • Omaha - NE Position Overview Successful State Farm Agent...  ...potential. Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Develop... 
    Full time
    For contractors
    Work at office

    Greta Ritchie - State Farm Agent

    Omaha, NE
    3 days ago
  • $35k - $60k

     ...Future Opening: Office Representative - State Farm Agent Team Member Full Time • Omaha - NE Replies within 24 hours Position...  ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop... 
    Full time
    For contractors
    Work at office
    Flexible hours

    Specht Insurance and Financial Solutions, L.L.C.

    Omaha, NE
    2 days ago
  • $40k - $60k

     ...Office Representative - State Farm Agent Team MemberSuccessful State Farm Agent is seeking a qualified professional to join their winning...  ...potential.ResponsibilitiesUse a customer-focused, needs-based review process to educate customers about insurance options.Develop... 
    Hourly pay
    For contractors
    Work at office

    Greta Ritchie - State Farm Agent

    Omaha, NE
    1 day ago
  • $35k - $60k

     ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop... 
    For contractors
    Work at office
    Flexible hours

    Candace Specht - State Farm Agent

    Omaha, NE
    3 days ago
  • $35k - $60k

     ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State...  ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads... 
    For contractors
    Work at office
    Flexible hours

    Candace Specht - State Farm Agent

    Omaha, NE
    1 day ago
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Remote job
    Contract work
    Currently hiring
    Local area
    Flexible hours

    signup-expert

    Omaha, NE
    2 days ago
  •  ...workflows, and maintaining compliance with internal and client-specific standards. Responsibilities Assist with receiving, reviewing, and processing project documents such as drawings, specifications, submittals, RFIs, change orders, and correspondence. Maintain... 
    Full time
    Work at office
    Flexible hours

    Olsson

    Omaha, NE
    10 days ago
  •  ...confidence in the insights they rely on. For HDR, it means enterprise and project data that is ready to support analytics, automation, and AI in a way that reflects how we work.This role is about making data simpler, more useful, and more connected to the way we do business,... 
    Local area

    HDR

    Omaha, NE
    1 day ago
  •  ...opportunity to plan for a brighter future. I’ve had the honor of serving my community for the past 14 years, and being a State Farm agent has truly changed my life both personally and professionally. I was born in Mexico City and moved to Omaha in the mid-90s. Most of my... 
    Work at office
    Flexible hours

    Monica Rodriguez - State Farm Agent

    Bellevue, NE
    14 days ago
  • $40k - $60k

     ...willingness to learn, we should meet to discuss our career opportunity. Come work with an energetic, fun team at Bob Speck - State Farm Agent! About Our Agency Our mission is to help people manage the risks of everyday life, recover from the unexpected and realize their... 
    For contractors
    Work experience placement
    Work at office

    State Farm Agent

    Omaha, NE
    4 days ago
  • $17 - $22.96 per hour

     ...support our recruiting process, we may use artificial intelligence ("AI") or automated technologies to collect, organize, sort, and...  ...tool. Hiring decisions are made by Associated Bank personnel who review applicant information and exercise independent judgment in accordance... 
    Hourly pay
    Night shift

    Associated Banc-Corp

    Bellevue, NE
    2 days ago
  • **If you are a current FIB employee, please apply through the Career Worklet in the Employee Portal This position is located at our Bellevue, NE Galvin Rd. S. branch. What's Important to You We know your career is just one aspect of a meaningful, complex, ...
    Night shift

    First Interstate

    Bellevue, NE
    5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!