AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour

Full-time

Bespoke Labs

Milford, DE

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Apply

Vacancy posted 2 days ago

Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Milford, DE vacancy

American Airlines Gate Agent
Now hiring an experienced American Airlines Gate Agent to perform daily responsibilities with dedication. Ensure compliance with company and safety standards. Work with your team to maintain efficiency and service quality. Perks include competitive pay, flexible schedules...
Suggested
Immediate start
Flexible hours
TradeJobsWorkforce
Houston, DE
5 days ago
Receptionist - State Farm Agent Team Member
...insurance ~ Opportunity for advancement ~ Paid time off ~ Training & development ROLE DESCRIPTION: Hunter Emory - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...
Suggested
Work at office
Flexible hours
Hunter Emory - State Farm Agent
Milford, DE
4 days ago
Receptionist - State Farm Agent Team Member
$35k - $50k
Receptionist - State Farm Agent Team Member Hunter Emory - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team...
Suggested
For contractors
Work at office
Immediate start
Hunter Emory - State Farm Agent
Milford, DE
3 days ago
Border Patrol Agent
$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...
Suggested
Full time
Work experience placement
Immediate start
Night shift
US Customs and Border Protection
Frederica, DE
1 day ago
Data Entry Operator | Junior (Remote)
Data Entry Operator | Junior (Remote) Milford, Delaware, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include ...
Suggested
Remote work
Only Data Entry
Milford, DE
3 days ago
Night Crew Clerk
POSITION TITLE: Night Crew ClerkDEPARTMENT: GroceryREPORTS TO: Grocery Manager / Store DirectorFLSA STATUS: Non-ExemptJOB SUMMARY:To maintain pricing, stocking, and rotation of merchandise in the grocery department.ESSENTIAL JOB FUNCTIONS:1) Assist in unloading the merchandise...
Shift work
Night shift
Redner's Warehouse Markets
Milford, DE
4 days ago
Night Crew Clerk
Night Crew ClerkJob Category: Operations GroupRequisition Number: NIGHT01341Part-TimeLocation: 57 Milford, Milford, DE 19963, USAJob DetailsPosition Title: Night Crew ClerkDepartment: GroceryReports To: Grocery Manager / Store DirectorFLSA Status: Non-ExemptJob Summary...
Shift work
Night shift
Redner's Markets
Milford, DE
5 days ago
Mail Handler in Houston, DE
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...
Hourly pay
Full time
Flexible hours
Postal jobs listings
Houston, DE
1 day ago
Bank Teller
Serves customers by completing account transactions. Provides account services to customers by receiving deposits and loan payments, cashing checks, issuing savings withdrawals, and recording night and mail deposits. Sells cashiers checks, travelers checks, and series...
Night shift
TradeJobsWorkForce
Houston, DE
1 day ago
Mail Handler
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...
Hourly pay
Flexible hours
Postal Jobs Placement Service
Milford, DE
1 day ago
Postal Clerk - Lincoln
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...
Hourly pay
Full time
Currently hiring
Work at office
Postal Jobs Source (a division of Labor Services)
Lincoln, DE
3 days ago
Postal Mail Handler - Milford
$26.47 - $43.62 per hour
...answers – To make sure you are fully prepared, you will be given sample exam questions AND answers. You will be shown exactly what to review and how to review, so you can get the highest score possible ~ Salary Gradings – You will be given a list of salary gradings for...
Hourly pay
Full time
Currently hiring
Immediate start
Flexible hours
Postal Jobs Source
Milford, DE
1 day ago
Postal Clerk - Milford
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Milford, DE
5 days ago
Postal Clerk - Lincoln
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Lincoln, DE
5 days ago
Remote Freight Dispatcher - Entry-Level IC with Growth
$1,200 - $2,500 per week
A logistics service provider is seeking an organized Entry-Level Freight Dispatcher to manage dispatch operations for owner-operators. The role offers estimated weekly earnings of $1,200 – $2,500 based on performance. Key responsibilities include coordinating freight, ...
Remote work
American Logistics Authority
Milford, DE
2 days ago
Remote Freight Dispatcher - Entry Level with Growth Path
$1,200 - $2,500 per month
A logistics company is seeking reliable and organized individuals for an Entry-Level Freight Dispatcher role. This is an independent contractor position supporting owner-operators with daily dispatch operations. Responsibilities include coordinating freight, communicating...
For contractors
Work experience placement
Remote work
American Logistics Authority
Milford, DE
2 days ago
Sales Associate/Office Assistant
$15 - $17 per hour
Fish Window Cleaning Sales Representative Fish Window Cleaning is Hiring! We are currently seeking a highly motivated individual to join as a sales representative. Because we are the largest and the best window cleaning company, you will have many benefits to offer ...
Hourly pay
Work at office
Flexible hours
Weekend work
Fish Window Cleaning
Milford, DE
2 days ago
Entry-Level Freight Dispatcher - $1,200-$2,500 Weekly
$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements:...
For contractors
Remote work
American Logistics Authority
Milford, DE
5 days ago
Mail Processor
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...
Hourly pay
Postal Jobs Placement Service
Milford, DE
1 day ago
Mail Processor - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...
Hourly pay
Postal Jobs Resource
Milford, DE
more than 2 months ago
USPS Mail Clerk
Bring your energy to the role of USPS Mail Clerk to sort incoming and outgoing mail, distribute letters, and prepare packages for carriers. Deliver excellent service to customers and coworkers. Deliver excellent service to customers and coworkers. Perks include competitive...
Flexible hours
TradeJobsWorkforce
Houston, DE
3 days ago
USPS Postal Workers - Mail Carrier/Assistant $22-$36/hr
$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...
Hourly pay
Full time
Work at office
Night shift
USPS
Milford, DE
5 days ago
Mail Carrier - Rural Assistant - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...
Hourly pay
Weekend work
Postal Jobs Resource
Houston, DE
more than 2 months ago
Part-Time Teller
$18 - $26.21 per hour
Overview: Specializes in our branch customer experience through digital engagement, lobby management, transactional processing and retail servicing interactions. Processes a variety of financial transactions in an accurate and efficient manner. Primary Responsibilities...
Hourly pay
Part time
Immediate start
M&T Bank
Harrington, DE
1 day ago
Mail Handler
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact ...
Hourly pay
Flexible hours
Postal Jobs Placement Service
Milton, DE
5 days ago
Mail Handler
$72.4k
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...
Flexible hours
Postal Jobs Placement Service
Harrington, DE
4 days ago
AIR TRAFFIC CONTROLLER
ABOUT Act as our eyes and ears in the sky, mapping coasts and ports for Navy intelligence. Gain extensive training in radar, sonar, and air traffic communications. If you feel more at home above the sea than on it, a career in aviation might be for you. Below are some...
Full time
Part time
U.S. Navy
Milford, DE
2 days ago
Mail Processor
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...
Hourly pay
Postal Jobs Placement Service
Milton, DE
5 days ago
88M Motor Transport Operator - Find Your Full Time or Part Time Army Career (19963)
...organizations including Tesla, American Medical Response, Mercedes Benz, and Mustang Cat. Similar Career Fields Include: Cargo and Freight Agent, Cargo Inspector, Vehicle Operator.About Our Organization:The U.S. Army is wealth of possibilities for your future - whether you are...
Full time
Contract work
Part time
Immediate start
Relocation package
Flexible hours
U.S. Army Reserves
Milford, DE
2 days ago
Mail Processing Clerk
As a PSE Mail Processing Clerk, you will play a critical role in supporting the operations of the postal service by handling and preparing mail for distribution, managing sorting machines, and maintaining mailroom equipment. This position is designed for individuals who...
US Gov Careers
Ellendale, DE
more than 2 months ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!