AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$51k
...Border Patrol Agent U.S. Customs and Border Protection (CBP) Multiple Duty Locations Northern and Southern U.S. Borders Job Description U.S. Customs and Border Protection (CBP) is seeking Border Patrol Agents to help secure the nation's borders, prevent...SuggestedWork experience placementLocal areaRemote workRelocationMarCom Group
Valdosta, GA1 day ago- Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting data from paper documents into digital spreadsheets, updating order statuses ...SuggestedRemote work
Only Data Entry
Valdosta, GA2 days ago - ...schedule Opportunity for advancement Paid time off Training & development ROLE DESCRIPTION: Travis Pate - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our clients. We are currently seeking a dynamic individual...SuggestedWork at officeFlexible hours
- Full-time Description BJM Group is a network of local accounting firms throughout Georgia and Florida. While the local firms under BJM may differ, we are one network and one team with a unified mission of profitable growth and outstanding client service. We provide...SuggestedFull timeWork at officeLocal areaImmediate startRemote work
Bjm Employer Inc
Valdosta, GA10 hours ago - Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...SuggestedHourly payFull timeFlexible hours
- Join First Commerce Credit Union as a Teller*** Interested in joining First Commerce Credit Union? This posting allows us to connect with qualified candidates for current and future Teller opportunities. Openings may become available at various Financial Center locations...Hourly pay
First Commerce Credit Union
Valdosta, GA1 day ago - ...organizations including Tesla, American Medical Response, Mercedes Benz, and Mustang Cat. Similar Career Fields Include: Cargo and Freight Agent, Cargo Inspector, Vehicle Operator. About Our Organization: The U.S. Army is wealth of possibilities for your future –...Contract workPart timeImmediate startRelocation packageFlexible hours
U.S. Army Reserve
Valdosta, GA3 days ago - ...close Facilitates and insures accurately and timely billing of services and fees to facility clients Completes interim statement review to ensure charges are accurate Participates in month-end closing and audits as necessary Provides customer support via phone...Full timeContract workWork experience placementInterim roleWork from home
BrightSpring Health Services
Valdosta, GA2 days ago - Join First Commerce Credit Union as a Teller *** Interested in joining First Commerce Credit Union? This posting allows us to connect with qualified candidates for current and future Teller opportunities. Openings may become available at various Financial Center locations...Hourly pay
First Commerce Credit Union
Valdosta, GA5 days ago $26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
Valdosta, GAmore than 2 months ago$15 - $20 per hour
Office Administrator / Administrative Assistant Location: Valdosta, GA Schedule: Monday–Friday 8:00 AM–5:00 PM , Pay: $15.00–$20.00 per hour Position Overview We are seeking a friendly and dependable Office Assistant to help with basic day-to-day office...Hourly payWork at officeMonday to FridaySouth Georgia Staffing
Valdosta, GA26 days ago- Job Description Job Description Verify Insurance, Collect Balances, Check patients in, Appointment Reminder Calls. \nCompany Description Medical Office Privately Owned Company Description Medical Office Privately OwnedWork at office
South Georgia Medical Associates Pc
Valdosta, GA28 days ago - NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$1,200 - $2,500 per month
A logistics company is seeking reliable individuals for an Entry-Level Freight Dispatcher role. This position supports owner-operators with daily dispatch operations and offers remote work flexibility. The estimated weekly earnings range between $1,200 and $2,500, based...Remote workAmerican Logistics Authority
Valdosta, GA2 days ago- Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and ...Hourly payWeekend work
- Job Description Job Description ***Interested in joining First Commerce Credit Union? This posting allows us to connect with qualified candidates for current and future Teller opportunities. Openings may become available at various Financial Center locations throughout...Hourly pay
First Commerce Credit Union
Valdosta, GA7 days ago $23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
Valdosta, GAmore than 2 months ago$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...Hourly payFull timeWork at officeNight shiftUSPS
Valdosta, GA2 days ago- Why Bank OZKFounded on a legacy of more than 120 years in banking, Bank OZK is much more than just a company. We're nationally recognized as an industry leader in financial services. That means we combine exceptional service with innovative technologies to deliver smart...Work at officeLocal areaDay shift
- ABOUT Act as our eyes and ears in the sky, mapping coasts and ports for Navy intelligence. Gain extensive training in radar, sonar, and air traffic communications. If you feel more at home above the sea than on it, a career in aviation might be for you. Below are some...Full timePart time
U.S. Navy
Valdosta, GA1 day ago $10.31 per hour
: TITLE: LIHEAP/CSBG Office Clerk PROGRAM: LIHEAP/CSBG CLASS: Non-Exempt REPORTS TO: CSBG Director LOCATION: Central Office JOB SUMMARY: Under the direction of the Community Services Director, the LIHEAP/CSBG Office Clerk will be responsible for checking...Temporary workPart timeWork experience placementWork at office$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements: Prior...Full timeFor contractorsRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
