Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Manchester, NH vacancy
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The Opportunity... 
    Suggested
    Full time
    Contract work
    Currently hiring
    Local area
    Remote work
    Flexible hours

    Signup Expert

    Manchester, NH
    6 days ago
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Suggested
    Remote job
    Hourly pay
    Contract work
    Freelance
    Currently hiring
    Local area
    Flexible hours

    Signup Expert

    Manchester, NH
    6 days ago
  • Signup Expert is seeking Online Sweepstakes Reviewers & Content Contributors to support one of our publishing partners by producing high-quality written reviews and evaluating online sweepstakes platforms. This fully remote, contract-based role is ideal for individuals... 
    Suggested
    Remote job
    Contract work

    Signup Expert

    Manchester, NH
    6 days ago
  • $20.11 - $32.99 per hour

     ...distribute files when requested by authorized personnel. Update records and ensure documents are stored in the correct locations. Review files for missing information, duplicate records, or filing errors. Maintain document logs and tracking systems. Protect confidential... 
    Suggested
    Remote job
    Hourly pay
    Work at office

    Sky Health Care Management

    Manchester, NH
    6 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    U.S. Customs and Border Protection

    Pinardville, NH
    1 day ago
  •  ...Job Title: Passenger Service Attendant - Wheelchair Agent/cabin cleaner Reports To: Supervisor/Manager/Dispatcher (varies by operational size of location) FLSA Status: Non-Exempt, eligible for overtime as governed by federal and/or state law Position Objective... 
    Shift work

    Huntleigh USA

    Manchester, NH
    3 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    Manchester, NH
    1 day ago
  • Terracon in Manchester, NH is seeking a Project Delivery Support professional to assist with document control, project tracking, and communications that enhance client experience. You will collaborate with project managers, financial analysts, and the operations team to...
    Contract work

    Terracon

    Manchester, NH
    6 days ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Manchester, NH
    5 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Adzuna

    Manchester, NH
    1 day ago
  •  ...Group 1 Automotive, a leader in automotive retail. We are looking to add a qualified, experienced Business Development Center (BDC) Agent to our team. Responsibilities Handle inbound leads (phone, Internet) and direct to appropriate contact Contact present customers... 
    Local area

    Group 1 Automotive

    Manchester, NH
    6 days ago
  • $60k

    Taubman Financial Management (TFM) is seeking licensed Life & Annuity agents to work remotely with warm leads, vetted carriers, and coaching from founder Scott Taubman. No cold calling, generous commissions, and a 1099, commission-only structure mean your income grows... 
    Remote job

    True North Recruiters

    Manchester, NH
    5 days ago
  •  ...Our Agency:Our agency opened in 2019 and is a team of four who value discipline, teamwork, and strong performance. Before becoming an agent, I served in the military, which shaped my leadership style and commitment to accountability, collaboration, and results. We're very... 
    For contractors
    Work at office

    Jacob Bramel - State Farm Agent

    Bedford, NH
    1 day ago
  •  ...Agency: Our agency opened in 2019 and is a team of four who value discipline, teamwork, and strong performance. Before becoming an agent, I served in the military, which shaped my leadership style and commitment to accountability, collaboration, and results. We’re very... 
    Work at office
    Flexible hours

    State Farm

    Bedford, NH
    1 day ago
  • $60k - $70k

     ...Agency: Our agency opened in 2019 and is a team of four who value discipline, teamwork, and strong performance. Before becoming an agent, I served in the military, which shaped my leadership style and commitment to accountability, collaboration, and results. We're very... 
    For contractors
    Work at office
    Flexible hours

    Jacob Bramel - State Farm Agent

    Bedford, NH
    2 days ago
  • $64.93k - $110.37k

    Data Manager Job Description Because of the need for consistent, in-person collaboration and/or the requirement to perform all work onsite due to the nature of this particular role, this position is to be performed full-time on site, conducted on location at a BAE Systems...
    Full time
    Work experience placement
    Work at office
    Local area
    Worldwide
    Flexible hours

    BAE Systems

    Merrimack, NH
    3 days ago
  • $25 - $30 per hour

     ...including procedures, work instructions, forms, drawings, specifications, standards, customer requirements, and supplier documentation. Review document submissions for completeness, correct formatting, required approvals, effective dates, revision history, and compliance... 
    Hourly pay
    Work at office
    Flexible hours

    Naprotek

    Londonderry, NH
    2 days ago
  • $48k - $52k

     ...Agency: Our agency opened in 2019 and is a team of four who value discipline, teamwork, and strong performance. Before becoming an agent, I served in the military, which shaped my leadership style and commitment to accountability, collaboration, and results. We're very... 
    For contractors
    Work at office
    Flexible hours

    Jacob Bramel - State Farm Agent

    Bedford, NH
    6 days ago
  • SemiGen, Inc. in Londonderry, NH seeks a dedicated Document Control Administrator to sustain the Quality Management System and document control across facilities. You will ensure controlled documents are accurate, approved, and accessible, while safeguarding sensitive information...

    SemiGen, Inc.

    Londonderry, NH
    2 days ago
  • SemiGen is seeking a Document Control Administrator to support the Quality Management System, configuration management, quality assurance, engineering, and manufacturing. The role reports to the Quality Director and ensures controlled information and quality records are...

    Naprotek, LLC

    Londonderry, NH
    2 days ago
  • $22.31 - $29.31 per hour

    Job Duties Provides accurate information and assistance to customers, which includes responding to inquiries, problem-resolution and selling of company services. Performs administrative support functions necessary for efficient station operations. Performs other duties ...
    Shift work

    FedEx Corporation

    Londonderry, NH
    4 days ago
  •  ...Shelter it!We are seeking an exceptional entrepreneur to make a difference in Manchester, MO and become our newest Shelter Insurance Agent. Do you have a desire to meet new people, develop new contacts, and become known in your community? Are you passionate about... 
    Full time
    Contract work
    Work at office
    Relocation

    Shelter

    Manchester, NH
    2 days ago
  • $16.28 - $30.23 per hour

    Gaming Systems Operator This position is responsible for executing manual activities in the gaming systems application, reporting daily activities and incidents, upholding the security integrity of the gaming systems and its operations, and effectively communicating...
    Hourly pay
    Full time
    Work experience placement
    Worldwide
    Visa sponsorship
    Work visa
    Shift work

    Aristocrat

    Manchester, NH
    3 days ago
  •  ...your own participation schedule. At this job, you will be performing various tasks such as data entry, performing email response, reviews, polls and other online projects. This work at home opportunity is very rewarding and will help shape the marketplace and influence... 
    Extra income
    Part time
    Immediate start
    Work from home

    USASJB

    Manchester, NH
    6 days ago
  • $51.63k - $124.72k

     ...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S...  ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    Marcom Group

    Manchester, NH
    6 days ago
  • Data Entry Operator | Junior (Remote)Manchester, NH, United States Job Openings Data Entry Operator | Junior (Remote)About the Job Data Entry Operator | Junior (Remote)Important: You Will Receive An Email Within Next 2 Minutes After Applying, Check Your Inbox or Spam Folder...
    Remote job

    Only Data Entry

    Manchester, NH
    2 days ago
  •  ...AGENCY: Our agency opened in 2019 and is a team of four who value discipline, teamwork, and strong performance. Before becoming an agent, I served in the military, which shaped my leadership style and commitment to accountability, collaboration, and results. We’re... 
    Work at office
    Flexible hours

    Jacob Bramel - State Farm Agent

    Bedford, NH
    a month ago
  • $22 - $30 per hour

    Dental Assistant at NHOMS I MassOMSJoin our innovative team at NHOMS | MassOMS and play an essential role in creating outstanding experiences for our clients, partners, and communities. By embracing our core values of innovation, compassion, accessibility, and one team...
    Hourly pay
    Work at office

    NHOMS

    Manchester, NH
    2 days ago
  • $16 - $30 per hour

    Hello and thank you for your interest! We’re looking for folks nationwide who are great at data entry and typing. We offer a flexible work from home remote position that allows you to stay home with the family! The pay range is flexible from $16/ph to $30/ph DOE and level...
    Hourly pay
    Part time
    Work at office
    Immediate start
    Remote work
    Work from home
    Flexible hours

    Jobconversion, LLC

    Manchester, NH
    6 days ago
  • $18 - $21.27 per hour

    Join Camden National BankIf you're looking to build your career at a forward-thinking organization with deep community roots and a vision for growth, success, and giving back, you've come to the right place.We're your local community bank—and have been since 1875—that ...
    Contract work
    Local area

    CNB

    Manchester, NH
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!