AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$80k - $120k
...solving skills Exposure in building ServiceNow agents, ServiceNow platform configuration, or technical design concepts AI enabled delivery, configuration, testing, and... ...practices Comfortable performing peer reviews, smoke testing, and troubleshooting technical...Suggested$34.57 - $46.11 per hour
...what makes you thrive at every stage of your career. Try new things, learn new skills and discover what you excel at—all from Day One.Reviews appraisals and reports including desktop valuations from a wide variety of bank channels. Reviewer Appraiser positions review all...SuggestedWork at office3 days per week$93.75k - $133.2k
...The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, your career is defined by how you think, analyze, and act in moments that directly impact national security. By harnessing your background to transition into federal...SuggestedWork at officeLocal areaShift work- ...Special Agent PositionJoining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and protection. Special Agents play a critical role in safeguarding...Suggested
$97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...SuggestedWork at officeLocal area- ...protective legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. This is no ordinary job, and our special agents are no ordinary individuals. Show us you have the talent and background we need, and we'll show you the rewards that come with...Overseas
$103.24k - $133.2k
...FBI Special Agent The position advertised has been exempted from the federal civilian hiring freeze. Job Title SPECIAL AGENT, GL-1... ...and/or exchange information. REQUIRED DOCUMENTATION Please review the list of required documents below and submit them with your application...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate start- ...BDC Agent The BDC Agent plays a vital role in interacting with customers, providing information, and scheduling appointments for interested individuals at Mark Sweeney Buick GMC in Cincinnati, OH. Interact with customers to gather information and address inquiries...
$49k - $55k
...Advertising Agent Catch Vibe Voice is seeking an Advertising Agent to join our team in Los Angeles, CA. In this role, you will support... ...advertising activities, timelines, and project deliverables. Review campaign materials to ensure consistency, accuracy, and...- Field Representative - Flexible Schedules For College Students - No Experience Needed Flexible schedules, you can work within the assigned window anytime the store is open! Need to study for a final? You can move your schedule around to fit your needs! Dress is casual...Part timeCasual workFlexible hoursShift work
$53k - $57k
...Advertising Agent At Beloform, we specialize in creating customized service solutions that bridge innovation and operational excellence. With a commitment to delivering high-quality results for our clients, we prioritize reliability, transparency, and tailored service...$52k - $56k
...Advertising Agent Beloform Craft is a growing production-focused company committed to precision, quality, and operational excellence... ...consistency and campaign accuracy Support the creation, review, and placement of advertising materials Monitor campaign performance...Full time- ...The Federal Bureau of Investigation (FBI) in the United States seeks a special agent to protect national security and lead investigations spanning cyber, fraud, and public safety. The role offers growth, rigorous training, and a comprehensive benefits package; candidates...Work at office
$47k - $51k
...are committed to driving success for both companies and individuals in today’s dynamic job market. Job Description As an Advertising Agent at NextShining, you will support the planning, coordination, and execution of advertising initiatives designed to increase brand...- We offer Daily pay! Access up to 50% of your earned wages after every shift. The Cincinnatian Hotel, Curio Collection by Hilton , located in Downtown Cincinnati , is hiring for a full-time Housekeeping Inspector . The person hired for this position will need ...Daily paidFull timeFlexible hoursShift workWeekday work
$17 - $18 per hour
...Property & Casualty InternSuccessful State Farm Agent is seeking a qualified professional to join their winning team for the role of... ...growth potential.ResponsibilitiesUse a customer-focused, needs-based review process to educate customers about insurance options.Work with...Hourly payFor contractorsInternshipLocal area$17 - $18 per hour
...Paid time off Position Overview Successful State Farm Agent is seeking a qualified professional to join their winning team for... ...potential. Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work...Hourly payFor contractorsInternshipLocal area- The Federal Bureau of Investigation (FBI) in the United States seeks a special agent to protect national security and lead investigations spanning cyber, fraud, and public safety. The role offers growth, rigorous training, and a comprehensive benefits package; candidates...Work at office
- ...celebrate who you are, and we respect life beyond work with flexibility that's recognized externally. Enabled by modern platforms and AI, you'll do the most meaningful work of your career and see your impact at scale. Job Summary This position is responsible for...Work at officeLocal area
- Job Description As a Geek Squad Agent, you’ll be the first point of contact for people seeking technology support, knowledge and solutions for computers and other devices at our local Best Buy store. We’ll provide the training and resources you need to assess customers...Hourly payFull timePart timeLocal areaWork from homeFlexible hoursNight shift
$45k - $50k
...collaborative and growth-focused work environment. Job Description Elevare Branding is seeking a motivated and detail-oriented Advertising Agent to join our growing team in Cincinnati, Ohio. In this role, you will assist in planning, coordinating, and executing advertising...Work at office- Pyramid Global Hospitality is seeking a Housekeeping Inspector for our Cincinnati property. The role focuses on ensuring guest room cleanliness, training staff, and distributing daily duties within the housekeeping team at The Phelps building. Ideal candidates have basic...
$40k - $60k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop leads...Hourly payFor contractorsWork at officeFlexible hours- ...attention to detail. Comfortable working in a fast‑paced environment. Prior experience in customer service, business support, or insurance preferred. Must possess or be able to obtain necessary state insurance licences. #J-18808-Ljbffr Michael Cason - State Farm AgentWork at office
$35k - $50k
...Receptionist - State Farm Agent Team Member Deatria Walker - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent...For contractorsWork at office$40k - $60k
Receptionist - State Farm Agent Team Member Full Time in Cincinnati - OH Randy Wagner - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist...Full timeWork at officeFlexible hours$30k - $40k
...career around helping people protect what matters most. Before opening my agency, I worked as a team member for another State Farm agent, which gave me the foundation to create a successful business centered on exceptional customer service and strong relationships. Our...Full timeWork at officeFlexible hours- DHL - - Responsibilities: Process import orders and coordinate with customs brokerage; Prepare and manage cargo release documentation; Communicate with air carriers to ensure timely release; Provide accurate job costing forecasts and billing; Maintain strong customer relationships...
$21.32 - $28.43 per hour
Air Export AgentYou will be responsible for the movement of our customer's air export shipments. You will coordinate and be the main point of contact for our outbound Air Export shipments from initial customer contact to final destination.Key Responsibilities:Coordinate...Work at officeOverseas$7.25 per hour
Experience Required None Minimum Education Required High School Diploma/G.E.D. Compensation $7.25 / hourly Hours Per Week 40 Number Of Positions 1 Work Schedule and Shift Requirements First (Day) Job Description Job Duties Ensures safe...Hourly payFull timeShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!

