AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...translates into income Leadership development - not just production Actually helping families protect what matters most This is for agents who: Are ready for a real breakthrough Want support instead of figuring it out alone Value flexibility and ownership of their...SuggestedRemote workFlexible hours
$60k - $90k
...sharing, training & development, competitive salary You May Be a Great Fit as a Multiline Representative at Bret Andreas - State Farm Agent if: You're the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about...SuggestedFor contractorsWork at officeBret Andreas - State Farm Agent
Carson City, NV2 days ago$65k - $85k
...successful careers. If you want to work in an environment that is fun, challenging, and rewarding, then Nicole Williams - State Farm Agent may be the right fit for you! If you're willing to work hard and you expect to get results from yourself and those around you, we...SuggestedWork at officeFlexible hoursNicole Williams DDS PA
Carson City, NV1 day ago$24 per hour
...help the company in maintaining excellent working conditions by keeping our facilities safe and problem-free. The Uniformed Unarmed Agent will protect people and property from possible criminal activity. Supervisory Responsibilities : None. Duties/Responsibilities...SuggestedFull timeLocal areaNight shiftTriumph Protection Group
Carson City, NV4 days ago$25 - $26.7 per hour
...single patient’s request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of... .... As an HCC (Hierarchical Condition Category) coder you will review medical records to identify and code diagnoses using a standardized...SuggestedRemote workFlexible hours- ...someone who is hungry for success and willing to put in the work to achieve it. ROLE DESCRIPTION: Jake Johnston - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual...Work at officeFlexible hours
Jake Johnston - State Farm Agent
Minden, NV19 days ago - Data Entry Operator | Junior (Remote) Carson City, NV, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...Remote job
- ...focused environment where you can grow and make an impact. Location Address: Carson City, NV 89703 At Nicole Williams - State Farm Agent, we’re here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support....Work at officeLocal areaFlexible hours
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- ...• Perform secretarial duties for committees of the facility as directed. • Collect and assemble/compile records for committee review, as requested, and prepare reports for staff/other committees as directed. Personnel Functions • Report known or suspected...Work at officeImmediate startShift workWeekend work
NorthStar Services
Carson City, NV3 days ago $18 per hour
...invoices; disburses overpayments on true-up invoices. Independently reviews and interprets sold documentation and completes Billing set-up.... .... The company’s expertise, combined with the most advanced AI-enabled technology available, sets the standard for solutions in...Part timeWork at officeLocal areaShift workSedgwick
Carson City, NV3 days ago$21 - $25 per hour
If you are unable to complete this application due to a disability, contact this employer to ask for an accommodation or an alternative application process. WARRANTY CLERK Full Time Clerical Carson City, NV, US Salary Range: $21.00 To $25.00 Hourly About the Role: The Warranty...Hourly payFull time- US:NV:Carson City | Food & Nutrition Services | Full Time Description US:NV:Carson City Patient Food Services Full Time Variable Shift Summary Responsible for the sanitation of kitchen/ cafeteria servery, floors, floor mats, ceiling/walls, counters, walk-in refrid, service...Full timeShift work
- Job Description We are now hiring Great People Who Care! We are looking for a fearless and accountable Clerk who is positive and passionate! Our Clerks will be responsible for executing a wide variety of tasks throughout the store, including working throughout a variety...Weekly payPart timeImmediate start
- Clerk Processor IThe Clerk Processor I is a Full-Time position and is responsible for cash handling, merchandising, sorting, cleaning, hanging, pricing, and tagging miscellaneous donated merchandise.Adheres to Kaizen principles of work station organization.Cleans sorts,...Full timeFlexible hours
- Loss Prevention ClerkProtects company assets from internal and external theft. Reviews individual warehouse inventory numbers and assists in creating / executing a plan of action to reduce shrink. Is familiar with various facets of operation that contribute to shrink and...Temporary workLocal area
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFull timeFlexible hours$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFlexible hours$18 - $20 per hour
Nevada State Bank a division of Zions Bancorporation N.A. was founded on Dec. 9 1959 by 12 businessmen full of Nevadas pioneering spirit. As a full-service bank we offer a complete range of consumer private and business banking services. Our colleagues regularly volunteer...Part timeWork experience placementFlexible hoursNevada State Bank
Carson City, NV7 days ago$18 - $22 per hour
Job Description Job Description About the Role At Rocket EMS , we manufacture high-reliability printed circuit board assemblies (PCBAs) for leading technology companies. As a Stockroom Clerk , you will help ensure that electronic components and materials are...Afternoon shiftRocket EMS
Carson City, NV14 days ago$1,200 - $2,500 per week
A logistics company is seeking an Entry-Level Freight Dispatcher to assist owner-operators in managing daily dispatch operations. This independent contractor role offers weekly earnings between $1,200 and $2,500, depending on freight volume and performance. The ideal candidate...For contractorsRemote workAmerican Logistics Authority
Carson City, NV1 day ago$1,200 - $2,500 per week
A logistics company is seeking an Entry-Level Freight Dispatcher to support owner-operators. This independent contractor role involves coordinating freight, communicating with brokers, and maintaining dispatch records. Candidates must have remote work experience, strong...For contractorsWork experience placementRemote workAmerican Logistics Authority
Carson City, NV4 days ago- Idexcel is a global staffing and IT services organization, with a mission to bring great people and great organizations together. Our diverse client base represents a wide range of industries, including technology, telecom, insurance, healthcare, manufacturing, banking ...Work at officeLocal area
$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements: Prior...For contractorsRemote workAmerican Logistics Authority
Carson City, NV18 hours ago- PART-TIME FRONT OFFICE ASSISTANT / RECEPTIONISTBighorn Plumbing is looking for a dependable, friendly, and organized Front Office Assistant/Receptionist to join our growing team. This is an entry-level position and a great opportunity to build professional office and customer...Part timeWork at officeNight shiftWeekday work
eddy
Carson City, NV1 day ago $23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Placement Service
Carson City, NV7 hours ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekly payWeekend work$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
Carson City, NVmore than 2 months ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
