Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Huntsville, AL vacancy
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Suggested
    Remote job
    Hourly pay
    Contract work
    Freelance
    Currently hiring
    Local area
    Flexible hours

    Signup Expert

    Huntsville, AL
    2 days ago
  • $51.63k - $124.72k

     ...Job Description Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Huntsville, AL
    10 days ago
  •  ...Perform advanced editing, text formatting, form creation, page manipulation, and redaction using Adobe Acrobat. Quality Assurance: Review incoming and outgoing documents for accuracy, proper formatting, completeness, and compliance. Version Control: Implement and... 
    Suggested
    Contract work
    Work at office

    Top Stack

    Huntsville, AL
    2 days ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Suggested
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Huntsville, AL
    4 days ago
  •  ...Opportunity PeopleTec is currently seeking a Software Engineer (AI Agents & Orchestration) to support our Huntsville AL location....  ...communicate with the simulator and provide tools for instructors to review and evaluate system performance. Duties: Develop software... 
    Suggested
    Full time
    Local area

    PeopleTec

    Huntsville, AL
    1 day ago
  • $13 - $15 per hour

     ...Flexible schedule ROLE DESCRIPTION: Shon Henry - State Farm Agent is seeking an organized and efficient contributor to fulfill...  ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary... 
    Hourly pay
    Full time
    For contractors
    Internship
    Work at office
    Flexible hours

    State Farm

    Huntsville, AL
    4 days ago
  • $10 - $15 per hour

     ...ROLE DESCRIPTION Amanda Suciu - State Farm Agent is seeking an organized and efficient contributor to fulfill administrative duties and...  ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary... 
    Hourly pay
    Full time
    Part time
    Internship
    Work at office

    Amanda Suciu - State Farm Insurance Agent

    Huntsville, AL
    3 days ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Huntsville, AL
    3 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Adzuna

    Huntsville, AL
    23 hours ago
  •  ...Responsive RecruiterAt Ken Davis - State Farm Agent, we're here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support. Whether we're helping someone navigate a claim or personalize a plan, we believe in... 
    For contractors
    Work at office
    Immediate start
    Flexible hours

    Ken Davis - State Farm Agent

    Huntsville, AL
    4 days ago
  • $15 - $18.32 per hour

     ...single patient’s request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of...  ...related telephone calls and inquiries accurately and timely. Review, research, resolve and respond to inquiries that are received via... 
    Hourly pay
    Immediate start
    Remote work

    Datavant

    Huntsville, AL
    2 days ago
  •  ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The... 
    Remote job
    Contract work
    Currently hiring
    Local area
    Flexible hours

    signup-expert

    Huntsville, AL
    2 days ago
  • $51.63k - $124.72k

    Job Description Border Patrol Agent (BPA)\n With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Huntsville, AL
    3 days ago
  • $71k - $96k

     ...documentation necessary for change. Coordinates with customers and manufacturers to determine a process for change reporting. Reviews released engineering change data and changes documenting activities to ensure adherence to configuration management procedures and... 
    Full time
    Contract work
    Temporary work
    Remote work

    GD Ordnance and Tactical Systems

    Huntsville, AL
    25 days ago
  •  ...Agent Aspirant – Future Agency Owner Madalynn Williams State Farm | Huntsville, AL We are looking for a highly motivated, competitive, and growth-minded Agent Aspirant who wants more than just a job. This opportunity is designed for someone interested in developing... 
    Work at office

    Madalynn Williams - State Farm Agent

    Huntsville, AL
    2 days ago
  • $40k - $50k

     ...and currently lead a team of three. Before becoming a State Farm agent, I worked as a physician recruiter and later as an Office...  ...life balance. We're proud to have earned more than 1,200 five‑star reviews by treating every customer like family. Our culture is laid‑back... 
    Full time
    For contractors
    Work at office
    Flexible hours

    Insuredwithshon

    Huntsville, AL
    1 day ago
  • $535 per month

    Job Title: Administrative Assistant II - Part-time Records Clerk Retirees of the Texas Teachers Retirement System (TRS) may apply. Consistent with district policy SD-7.17 Employment of Retirees selected applicants are responsible for reimbursing the district through...
    Full time
    Part time
    Casual work
    Remote work

    Windham School District

    Huntsville, AL
    1 day ago
  •  ...and Renters Insurance. Our office is located in Scottsboro, AL. Our office is open 9:00-5:00. Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member. Our agency has received awards including: Chairman's Circle, Ambassador... 
    Internship
    Work at office

    David Perkins

    Madison, AL
    2 days ago
  • $12.5 - $17.5 per hour

     ...and Renters Insurance. Our office is located in Scottsboro, AL . Our office is open 9:00-5:00. Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member. Our agency has received awards including: Chairman's Circle,... 
    Hourly pay
    Part time
    For contractors
    Internship
    Work at office
    Flexible hours

    State Farm

    Madison, AL
    3 days ago
  • $35k - $75k

     ...Bonus based on performance Competitive salary You May Be a Great Fit as an Acquisitions Specialist at Amanda Suciu - State Farm Agent if You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care about your community... 
    Work at office

    Amanda Suciu State Farm Insurance Agent

    Huntsville, AL
    3 days ago
  • $35k - $75k

     ...Multi-Line Representative - State Farm Agent Team Member Full Time in Huntsville/ Madison- AL Opportunity for advancement Paid time off Bonus based on performance You May Be a Great Fit as a Multiline Representative at Amanda Suciu - State Farm Agent... 
    Full time
    Work at office

    Amanda Suciu - State Farm Insurance Agent

    Madison, AL
    2 days ago
  • Benefits ~ Health insurance ~ Paid time off ~401(k) matching ~ Bonus based on performance ~ Competitive salary ~ Dental insurance ~ Employee discounts You May Be a Great Fit as a Multiline Representative at Diana Lockmiller State Farm if: ...
    Work at office

    Dianaismyagent

    Madison, AL
    2 days ago
  •  ...about their development. If you want to work in an environment that is fun, challenging, and rewarding, then Ken Davis - State Farm Agent may be the right fit for you! If you want to make an impact, see immediate success, we could be the place for you! If you're willing... 
    For contractors
    Work at office
    Immediate start

    Ken Davis - State Farm Agent

    Huntsville, AL
    2 days ago
  •  ...Representante Agente de Viajes Huntsville, Alabama, United States Or refer someone Job Openings Representante Agente de Viajes Sobre el puesto: Representante de Atencin al Cliente de Viajes Como Representante de Atencin al Cliente de Viajes, desempears un papel... 
    Remote work
    Flexible hours

    Archer Travel Group

    Huntsville, AL
    1 day ago
  •  ...todays market area. As a Team Member, you will certainly be doing numerous work such as online data entry, performing e-mail response, reviews, studies and also various other online jobs. This work at home work opportunity is extremely rewarding and will certainly help... 
    Full time
    Part time
    Work from home

    USASJB

    Huntsville, AL
    5 days ago
  • $51.63k - $124.72k

     ...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S...  ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    Marcom Group

    Huntsville, AL
    17 days ago
  • $55k - $65k

     ...Job Description Job Description Director of Agent Recruitment & Growth Location: Huntsville, Alabama Employment: Full-Time | Primarily In-Office Base Salary: $55,000–$65,000 annually, commensurate with experience, plus production-based recruiting incentives... 
    Full time
    Work at office
    Afternoon shift

    Better Homes and Gardens Real Estate Southern Premiere

    Huntsville, AL
    12 days ago
  •  ...advancement ROLE DESCRIPTION: Madalynn Williams - State Farm Agent is seeking an organized and efficient contributor to fulfill...  ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use... 
    Full time
    Internship
    Work at office

    Madalynn Williams - State Farm Agent

    Huntsville, AL
    27 days ago
  • Data Entry Operator | Junior (Remote)Huntsville, AL, United States Job Openings Data Entry Operator | Junior (Remote)A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting data...
    Remote job

    Only Data Entry

    Huntsville, AL
    1 day ago
  • Data Entry ClerkHuntsville, Alabama, United States Job Openings Data Entry ClerkA Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting data from paper documents into digital ...

    Only Data Entry

    Huntsville, AL
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!