AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The...SuggestedRemote jobHourly payContract workFreelanceCurrently hiringLocal areaFlexible hours
$51.63k - $124.72k
...Job Description Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift- ...Perform advanced editing, text formatting, form creation, page manipulation, and redaction using Adobe Acrobat. Quality Assurance: Review incoming and outgoing documents for accuracy, proper formatting, completeness, and compliance. Version Control: Implement and...SuggestedContract workWork at office
$93.75k - $133.2k
...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks... ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college...SuggestedPermanent employmentFull timeWork experience placementWork at officeLocal areaImmediate start- ...Opportunity PeopleTec is currently seeking a Software Engineer (AI Agents & Orchestration) to support our Huntsville AL location.... ...communicate with the simulator and provide tools for instructors to review and evaluate system performance. Duties: Develop software...SuggestedFull timeLocal area
$13 - $15 per hour
...Flexible schedule ROLE DESCRIPTION: Shon Henry - State Farm Agent is seeking an organized and efficient contributor to fulfill... ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary...Hourly payFull timeFor contractorsInternshipWork at officeFlexible hours$10 - $15 per hour
...ROLE DESCRIPTION Amanda Suciu - State Farm Agent is seeking an organized and efficient contributor to fulfill administrative duties and... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary...Hourly payFull timePart timeInternshipWork at office$10k
...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift- ...Responsive RecruiterAt Ken Davis - State Farm Agent, we're here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support. Whether we're helping someone navigate a claim or personalize a plan, we believe in...For contractorsWork at officeImmediate startFlexible hours
$15 - $18.32 per hour
...single patient’s request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of... ...related telephone calls and inquiries accurately and timely. Review, research, resolve and respond to inquiries that are received via...Hourly payImmediate startRemote work- ...hands-on work, local market insight, and direct user engagement. We are currently hiring on behalf of a leading online sweepstakes review publisher to support ongoing content creation, platform testing, and digital growth initiatives within the iGaming industry. The...Remote jobContract workCurrently hiringLocal areaFlexible hours
$51.63k - $124.72k
Job Description Border Patrol Agent (BPA)\n With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift$71k - $96k
...documentation necessary for change. Coordinates with customers and manufacturers to determine a process for change reporting. Reviews released engineering change data and changes documenting activities to ensure adherence to configuration management procedures and...Full timeContract workTemporary workRemote work- ...Agent Aspirant – Future Agency Owner Madalynn Williams State Farm | Huntsville, AL We are looking for a highly motivated, competitive, and growth-minded Agent Aspirant who wants more than just a job. This opportunity is designed for someone interested in developing...Work at office
$40k - $50k
...and currently lead a team of three. Before becoming a State Farm agent, I worked as a physician recruiter and later as an Office... ...life balance. We're proud to have earned more than 1,200 five‑star reviews by treating every customer like family. Our culture is laid‑back...Full timeFor contractorsWork at officeFlexible hours$535 per month
Job Title: Administrative Assistant II - Part-time Records Clerk Retirees of the Texas Teachers Retirement System (TRS) may apply. Consistent with district policy SD-7.17 Employment of Retirees selected applicants are responsible for reimbursing the district through...Full timePart timeCasual workRemote work- ...and Renters Insurance. Our office is located in Scottsboro, AL. Our office is open 9:00-5:00. Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member. Our agency has received awards including: Chairman's Circle, Ambassador...InternshipWork at office
$12.5 - $17.5 per hour
...and Renters Insurance. Our office is located in Scottsboro, AL . Our office is open 9:00-5:00. Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member. Our agency has received awards including: Chairman's Circle,...Hourly payPart timeFor contractorsInternshipWork at officeFlexible hours$35k - $75k
...Bonus based on performance Competitive salary You May Be a Great Fit as an Acquisitions Specialist at Amanda Suciu - State Farm Agent if You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care about your community...Work at office$35k - $75k
...Multi-Line Representative - State Farm Agent Team Member Full Time in Huntsville/ Madison- AL Opportunity for advancement Paid time off Bonus based on performance You May Be a Great Fit as a Multiline Representative at Amanda Suciu - State Farm Agent...Full timeWork at office- Benefits ~ Health insurance ~ Paid time off ~401(k) matching ~ Bonus based on performance ~ Competitive salary ~ Dental insurance ~ Employee discounts You May Be a Great Fit as a Multiline Representative at Diana Lockmiller State Farm if: ...Work at office
- ...about their development. If you want to work in an environment that is fun, challenging, and rewarding, then Ken Davis - State Farm Agent may be the right fit for you! If you want to make an impact, see immediate success, we could be the place for you! If you're willing...For contractorsWork at officeImmediate start
- ...Representante Agente de Viajes Huntsville, Alabama, United States Or refer someone Job Openings Representante Agente de Viajes Sobre el puesto: Representante de Atencin al Cliente de Viajes Como Representante de Atencin al Cliente de Viajes, desempears un papel...Remote workFlexible hours
- ...todays market area. As a Team Member, you will certainly be doing numerous work such as online data entry, performing e-mail response, reviews, studies and also various other online jobs. This work at home work opportunity is extremely rewarding and will certainly help...Full timePart timeWork from home
$51.63k - $124.72k
...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S... ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before...Full timeWork experience placement$55k - $65k
...Job Description Job Description Director of Agent Recruitment & Growth Location: Huntsville, Alabama Employment: Full-Time | Primarily In-Office Base Salary: $55,000–$65,000 annually, commensurate with experience, plus production-based recruiting incentives...Full timeWork at officeAfternoon shift- ...advancement ROLE DESCRIPTION: Madalynn Williams - State Farm Agent is seeking an organized and efficient contributor to fulfill... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use...Full timeInternshipWork at office
- Data Entry Operator | Junior (Remote)Huntsville, AL, United States Job Openings Data Entry Operator | Junior (Remote)A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting data...Remote job
- Data Entry ClerkHuntsville, Alabama, United States Job Openings Data Entry ClerkA Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting data from paper documents into digital ...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
- agent Huntsville, AL
- tsa agent Huntsville, AL
- work from home chat agent Huntsville, AL
- import export agent Huntsville, AL
- commissioning agent Huntsville, AL
- signing agent Huntsville, AL
- state farm agent Huntsville, AL
- executive protection agent Huntsville, AL
- airport agent Huntsville, AL
- cruise agent Huntsville, AL


