AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour

Full-time

Bespoke Labs

Milford, DE

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Apply

Vacancy posted 1 day ago

Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Milford, DE vacancy

Security patrol agent - overnights
...Border Patrol Agent (BPA) – in the Federal Security and Public Safety Sector (Entry Level) A MISSION WORTHY OF A CAREER! But, if you'... ...official employment site, to complete your application. Be certain to review ALL details of the job opportunity announcement and follow all...
Suggested
Full time
Work experience placement
Immediate start
Relocation
Night shift
US Customs and Border Protection
Harrington, DE
1 day ago
Receptionist - State Farm Agent Team Member
...opened my agency on January 1, 2024, and currently have a team of four full-time and one part-time team member. Before becoming an agent, I worked as a manager for Cintas, but I always had a strong interest in both insurance and running my own business. Outside of work...
Suggested
Full time
Part time
Casual work
Work at office
Local area
Flexible hours
Shawn Collins - State Farm Agent
Milford, DE
9 days ago
Mail Handler in Houston, DE
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...
Suggested
Hourly pay
Full time
Flexible hours
Postal jobs listings
Houston, DE
1 day ago
Full and Part Time Tellers
$14.42 - $24.04 per hour
DE - Milford Part time R232461 Position Overview At PNC, our people are our greatest differentiator and competitive advantage in the markets we serve. We are all united in delivering the best experience for our customers. We work together each day to foster ...
Suggested
Full time
Part time
Work experience placement
Work at office
PNC
Milford, DE
5 days ago
Night Crew Clerk
Night Crew Clerk Job Category: Operations Group Requisition Number: NIGHT01341 Part-Time Location: 57 Milford, Milford, DE 19963, USA Job Details Position Title: Night Crew Clerk Department: Grocery Reports To: Grocery Manager / Store Director ...
Suggested
Part time
Shift work
Night shift
Redner's Markets
Milford, DE
2 days ago
Part-Time Teller: Elevate Customer Banking Experience
$14.42 - $24.04 per hour
PNC is seeking a Part Time Teller for the Milford, DE area at the Milford Square Branch. The role focuses on delivering a unique customer experience, educating customers on digital options, and driving branch performance with accurate daily operations. The position emphasizes...
Hourly pay
Part time
Work at office
PNC
Milford, DE
2 days ago
Postal Clerk - Lincoln
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Lincoln, DE
4 days ago
Postal Clerk - Milford
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...
Hourly pay
Full time
Currently hiring
Work at office
Postal Jobs Source (a division of Labor Services)
Milford, DE
2 days ago
Postal Clerk - Houston
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Houston, DE
4 days ago
USPS Mail Sorter
Bring your energy to USPS Mail Sorter and help us make every day count. In this position, youll join a company that rewards reliability and dedication while supporting daily operations and contributing to success. Reliable pay, flexible scheduling, and advancement opportunities...
Flexible hours
TradeJobsWorkForce
Houston, DE
5 days ago
Remote Freight Dispatcher - Entry-Level IC with Growth
$1,200 - $2,500 per week
A logistics service provider is seeking an organized Entry-Level Freight Dispatcher to manage dispatch operations for owner-operators. The role offers estimated weekly earnings of $1,200 – $2,500 based on performance. Key responsibilities include coordinating freight, ...
Remote work
American Logistics Authority
Milford, DE
1 day ago
Remote Freight Dispatcher - Entry Level with Growth Path
$1,200 - $2,500 per month
A logistics company is seeking reliable and organized individuals for an Entry-Level Freight Dispatcher role. This is an independent contractor position supporting owner-operators with daily dispatch operations. Responsibilities include coordinating freight, communicating...
For contractors
Work experience placement
Remote work
American Logistics Authority
Milford, DE
1 day ago
EntryLevel Freight Dispatcher Weekly
$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements: Prior...
For contractors
Remote work
American Logistics Authority
Milford, DE
3 days ago
Mail Processor - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...
Hourly pay
Postal Jobs Resource
Houston, DE
more than 2 months ago
Sales Associate/Office Assistant
$15 - $17 per hour
Fish Window Cleaning Sales Representative Fish Window Cleaning is Hiring! We are currently seeking a highly motivated individual to join as a sales representative. Because we are the largest and the best window cleaning company, you will have many benefits to offer ...
Hourly pay
Work at office
Flexible hours
Weekend work
Fish Window Cleaning
Milford, DE
2 days ago
Mail Carrier - Rural Assistant in Milford
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...
Hourly pay
Weekend work
Postal service
Milford, DE
2 days ago
USPS Postal Workers - Mail Carrier/Assistant $22-$36/hr
$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...
Hourly pay
Full time
Work at office
Night shift
USPS
Milford, DE
4 days ago
Mail Carrier - Rural Assistant - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...
Hourly pay
Weekend work
Postal Jobs Resource
Houston, DE
more than 2 months ago
Teller
$18 - $26.21 per hour
Overview: Specializes in our branch customer experience through digital engagement, lobby management, transactional processing and retail servicing interactions. Processes a variety of financial transactions in an accurate and efficient manner. Primary Responsibilities...
Hourly pay
Immediate start
M&T Bank
Harrington, DE
1 day ago
AIR TRAFFIC CONTROLLER
ABOUT Act as our eyes and ears in the sky, mapping coasts and ports for Navy intelligence. Gain extensive training in radar, sonar, and air traffic communications. If you feel more at home above the sea than on it, a career in aviation might be for you. Below are some...
Full time
Part time
U.S. Navy
Milford, DE
1 day ago
Postal Mail Processor - Milton
POSTAL MAIL PROCESSOR NO EXPERIENCE REQUIRED - PAID TRAINING PROVIDED - JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience...
US Postal Service
Milton, DE
5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!