AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$51.63k - $124.72k
...Job Description Border Patrol Agent (BPA)\n With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shiftCustoms and Border Protection
Hot Springs, AR13 days ago- We are immediately hiring part time Field Representatives in your area! Are you a military spouse and looking for supplemental income? Make extra spending money to help with your car payment, social activities, and more! Benefits: A comprehensive initial training ...SuggestedExtra incomePart timeImmediate startFlexible hoursShift work
RetailData
Hot Springs National Park, AR1 day ago $18 - $22 per hour
...Multiline Representative at Heather Powell - State Farm Agent Benefits: Health insurance, paid time off, vision insurance, competitive salary, dental insurance, free food & snacks You May Be a Great Fit if: You're the person people call first when life goes sideways...SuggestedHourly payFull timeFor contractorsWork at office$18 - $22 per hour
...General Manager at Heather Powell - State Farm Agent Benefits: Paid time off Competitive salary Free food & snacks You May Be a Great Fit if: You're the person people call first when life goes sideways calm, steady, and ready to help. You genuinely...SuggestedHourly payFor contractorsWork at office- Telework Eligible Yes Major Duties Manage the full administrative responsibilities for customer asset inventory management across their respective Area of Consideration (AOR). Responsible for the accounts payable (AP), accounts receivable (AR) processes within...SuggestedFull timeContract workRemote work
Defense Logistics Agency
Hot Springs, AR1 day ago - Data Entry Operator | Junior (Remote) Hot Springs, AR, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...Remote work
Only Data Entry
Hot Springs National Park, AR3 days ago $16 - $18 per hour
Job Full Description CUSTOMER SERVICE & OFFICE SUPPORT Pay: $16.00-$18.00/hour Schedule: Tuesday-Saturday, 8:30 AM-5:30 PM About the Position We are looking for a friendly, dependable person to join a fun, family-owned small business...Work at officeExpress Employment Professionals Defunct
Hot Springs, AR3 days ago- ...be a part of our success story. #OC Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor.Hourly payDaily paidWork at officeLocal area
America's Car-Mart
Hot Springs National Park, AR5 days ago - ...Overview Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Work with the agent to establish and meet marketing goals. Develop leads, schedule appointments,...Hourly payPart timeFor contractorsWork at office
- Mutuel Teller Position We are currently seeking a talented individual to become a Mutuel Teller. An individual could be successful if they possess the following. Essential Duties And Responsibilities The Mutuel Teller is responsible for selling and cashing pari...Part timeSeasonal workShift workDay shift
$1,200 - $2,500 per week
Entry-Level Freight Dispatcher – Weekly Earnings $1,200–$2,500We are seeking reliable and organized individuals for an Entry-Level Freight Dispatcher opportunity. This is an independent contractor role supporting owner-operators with daily dispatch operations.Estimated...For contractorsAmerican Logistics Authority
Hot Springs National Park, AR3 days ago$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$1,200 - $2,500 per week
A logistics company is seeking reliable individuals for an Entry-Level Freight Dispatcher role in Hot Springs, Arkansas. This independent contractor position supports owner-operators with daily dispatch operations. Candidates should have strong communication and organizational...For contractorsRemote workAmerican Logistics Authority
Hot Springs National Park, AR3 days ago$22.37 per hour
...motion functions in Book Print, Quality Control, and Mail Out. Book Print- Maintain control and accountability of passport books Review application information and printed passport books for errors and quality Remove or reject applications and passport books with...Hourly payWeekly payWork at officeLocal areaAfternoon shift- Job Description Job Description America's Car-Mart is seeking a dynamic and self-motivated Office Clerk to join our friendly team. If you have previous experience in cash handling, excellent customer service skills, and a passion for creating positive customer experiences...Hourly payDaily paidWork at officeLocal area
America's Car-Mart, Inc.
Hot Springs, AR14 days ago $1,200 - $2,500 per week
A logistics company is seeking an Entry-Level Freight Dispatcher to support owner-operators with daily dispatch operations. This independent contractor role offers estimated weekly earnings between $1,200 and $2,500, depending on various factors. Responsibilities include...For contractorsWork experience placementRemote workAmerican Logistics Authority
Hot Springs, AR1 day ago$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements:...For contractorsRemote workAmerican Logistics Authority
Hot Springs National Park, AR4 days ago$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFlexible hoursPostal Jobs Resource
Hot Springs National Park, ARmore than 2 months ago- Company description Terry Soot Management Group (TSMG) is a field data collection company founded in 2017 in Europe. We collect data where automation is not possible. We count features, take pictures, make videos, record speech, and scan areas for every detail you need ...Monday to FridayFlexible hours
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
Hot Springs National Park, ARmore than 2 months ago- Overview At Fast Pace Health, Clinical Techs provide clinical and administrative support essential for effective patient care under the direct supervision of a Provider, Nurse Practitioner or Physician Assistant. They provide outstanding patient service, maintaining...Seasonal workWork at office
Fast Pace Health
Hot Springs National Park, AR1 day ago $22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...Hourly payFull timeWork at officeNight shiftUSPS
Hot Springs, AR5 days ago$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekend workPostal Jobs Resource
Hot Springs National Park, ARmore than 2 months ago- Sorting Room Worker Receives, sorts, and prepares donated items for display in the store; determines suitability of items based on established standards; assists in maintaining the orderliness and cleanliness of the warehouse area. Assumes responsibilities of Store ...
- Truck Driver & Sorting Room Worker Drives Salvation Army truck along scheduled route to pick-up donations and/or deliver merchandise to customers; operates the truck in compliance with established Salvation Army standards and state driving regulations; lifts, carries...
- ABOUT Act as our eyes and ears in the sky, mapping coasts and ports for Navy intelligence. Gain extensive training in radar, sonar, and air traffic communications. If you feel more at home above the sea than on it, a career in aviation might be for you. Below are some...Full timePart time
U.S. Navy
Hot Springs, AR2 days ago - The United States Postal Service is hiring nationwide for entry-level Clerk positions with no experience required. Paid training is provided and offers job security along with full benefits. Starting hourly pay ranges from $26.47 to $43.62, averaging about $72,400 per year...Hourly pay
$26.47 - $43.62 per hour
...answers – To make sure you are fully prepared, you will be given sample exam questions AND answers. You will be shown exactly what to review and how to review, so you can get the highest score possible ~ Salary Gradings – You will be given a list of salary gradings for...Hourly payFull timeCurrently hiringImmediate startFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
