AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour

Full-time

Bespoke Labs

Smyrna, DE

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Apply

Vacancy posted 5 days ago

Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Smyrna, DE vacancy

ONA AI Agent Intern (Logistics Focus) - OVIP
$30 per hour
...autonomous database and industry’s broadest and deepest suite of AI-powered cloud applications. The following facts and figures highlight... ...Responsibilities We are seeking a highly motivated AI Agent Intern to join Oracle’s Supply Chain Applications (SCM) Solution...
Suggested
Hourly pay
Temporary work
Internship
Flexible hours
Oracle
Dover, DE
3 days ago
Registered Agent Specialist
...Registered Agent Specialist Under the general supervision of the Delaware Supervisor, the RA Specialist is responsible for sending out registered agent delinquency notices, preparing and processing state filings, registered agent assignment research and processing resignation...
Suggested
Work at office
Day shift
Ready 4 Work
Dover, DE
4 days ago
Inpatient Medical Coder - FT - Up to $5,000 Sign on Bonus
$32 - $42 per hour
...single patient’s request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of... ...facility. Experience with significant level of coding quality review feedback Experience in computerized encoding and abstracting...
Suggested
Hourly pay
Full time
Remote work
Relocation package
Flexible hours
Datavant
Dover, DE
3 days ago
Receptionist - State Farm Agent Team Member
$40k - $50k
...Receptionist - State Farm Agent Team Member Matt Basile - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team...
Suggested
For contractors
Work at office
Matt Basile - State Farm Agent
Dover, DE
5 days ago
Data Entry Operator | Junior (Remote)
About the job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying , Check Your Inbox or Spam Folder For next steps. A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures...
Suggested
Remote job
Only Data Entry
Smyrna, DE
2 days ago
Receptionist - State Farm Agent Team Member
...officially opened our agency on June 1, 2024, but my roots with State Farm run much deeper as I spent over 18 years working in another agent’s office before launching my own. Today, I lead a small but passionate team of three full-time professionals who are dedicated to...
Full time
Work at office
Local area
Flexible hours
Lindsay Dougherty - State Farm Agent
Dover, DE
12 days ago
Acquisition Specialist - State Farm Agent Team Member
...(k) matching ~ Bonus based on performance You May Be a Great Fit as an Acquisitions Specialist at Danny Schanne - State Farm Agent if: You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care about your...
Work at office
Danny Schanne - State Farm Agent
Smyrna, DE
21 days ago
Telemarketer - State Farm Agent Team Member
$13 - $18 per hour
...Recruiter Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Work with the agent to establish and meet marketing goals. Develop leads, schedule...
Hourly pay
Part time
For contractors
Work at office
Flexible hours
Matt Basile - State Farm Agent
Dover, DE
3 days ago
Part Time Teller
$14.42 - $24.04 per hour
DE - Smyrna Part time R231917 **Position Overview** At PNC, our people are our greatest differentiator and competitive advantage in the markets we serve. We are all united in delivering the best experience for our customers. We work together each day to foster an inclusive...
Full time
Part time
Work experience placement
Work at office
PNC
Smyrna, DE
5 days ago
Postal Clerk - Clayton
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...
Hourly pay
Full time
Currently hiring
Work at office
Postal Jobs Source (a division of Labor Services)
Clayton, DE
1 day ago
Postal Clerk - Kenton
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Kenton, DE
3 days ago
Postal Clerk - Smyrna
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Smyrna, DE
3 days ago
Data Entry Operator | Junior (Remote)
Data Entry Operator | Junior (Remote) Dover, Delaware, United States Job Openings Data Entry Operator | Junior (Remote) About the Job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying, Check Your Inbox...
Remote job
Only Data Entry
Dover, DE
1 day ago
Mail Handler - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...
Hourly pay
Flexible hours
Postal Jobs Resource
Kenton, DE
more than 2 months ago
Remote Freight Dispatcher - Entry Level with Growth Path
$1,200 - $2,500 per week
A logistics service provider is seeking an Entry-Level Freight Dispatcher to support owner-operators with daily dispatch operations. This role allows for remote work and offers earnings between $1,200 and $2,500 weekly, depending on freight type and performance. Responsibilities...
Remote work
American Logistics Authority
Smyrna, DE
3 days ago
Multi-Line Representative - State Farm Agent Team Member
Job Description Job Description Benefits: ~ Attend community-based events ~ Birthday celebrations ~ No cold calling - we are responding to customer inquiries ~401(k) ~ Bonus based on performance ~ Company parties ~ Competitive salary ~ Dental insurance...
Work at office
Day shift
Stefanie Wooten - State Farm Agent
Middletown, DE
3 days ago
Now Hiring Freight Dispatchers - Earn $800-$1,500+/week!
$800 - $1,500 per week
Now Hiring Freight DispatchersEarn $800–$1,500+/week!Job Title: Freight Dispatcher (Entry-Level & Experienced)Pay: $800–$1,500+ per week (depending on experience, performance, and skill level)Job Description: Truck Driver Nation is seeking motivated individuals to join...
Immediate start
Remote work
Flexible hours
American Logistics Authority
Smyrna, DE
3 days ago
EntryLevel Freight Dispatcher Weekly
$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements: Prior...
For contractors
Remote work
American Logistics Authority
Smyrna, DE
1 day ago
Mail Processor - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...
Hourly pay
Postal Jobs Resource
Cheswold, DE
more than 2 months ago
Now Hiring Freight Dispatchers Earn $800$1,500+/week!
$800 - $1,500 per week
Now Hiring Freight Dispatchers Earn $800$1,500+/week! Job Title: Freight Dispatcher (Entry-Level & Experienced) Pay: $800$1,500+ per week (depending on experience, performance, and skill level) Job Description: Truck Driver Nation is seeking motivated individuals...
Immediate start
Remote work
Flexible hours
American Logistics Authority
Smyrna, DE
1 day ago
Remote Freight Dispatcher - Entry-Level IC with Growth
$1,200 - $2,500 per week
A logistics company is seeking an Entry-Level Freight Dispatcher to assist owner-operators with daily dispatching tasks. This independent contractor role offers potential earnings of $1,200 to $2,500 per week, depending on performance and volume. Responsibilities include...
Remote job
For contractors
Work experience placement
American Logistics Authority
Smyrna, DE
2 days ago
PT Clerk - Produce
Retail Store PositionsBorn in the DMV, we've been here for over 88 years and counting. From our first store on Georgia Avenue to becoming the #1...Job Requisition: 300590Address: USA-DE-Middletown-200 Dove Run Centre Store Code: GF - Us Store Mgrs (2502278)
Giant Food
Middletown, DE
5 days ago
Stockroom Clerk
$16 - $20 per hour
Ace Hardware Stockers and Receivers are responsible for receiving shipments, unloading trucks, organizing inventory in the stockroom, stocking shelves, building product displays, performing inventory counts, and keeping the warehouse and sales floor organized. Overnight...
Hourly pay
Full time
Night shift
Ace Hardware
Middletown, DE
4 days ago
Dairy Clerk
POSITION TITLE: Dairy ClerkDEPARTMENT: GroceryREPORTS TO: Dairy ManagerFLSA STATUS: Non-ExemptJOB SUMMARY:To maintain pricing, stocking, and rotation of merchandise in the dairy department.ESSENTIAL JOB FUNCTIONS:1) Assist in unloading the merchandise.2) Transport stock...
Shift work
Redner's Warehouse Markets
Dover, DE
3 days ago
Mail Carrier - Rural Assistant - No Experience Required
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...
Hourly pay
Weekend work
Postal Jobs Resource
Smyrna, DE
more than 2 months ago
Clerk - Revenue Audit
...Win Report. Perform daily audit of paperwork for front desk agents, food & beverage cashiers, valet cashiers, and VIP Service cashiers... ...to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Ballys...
Work experience placement
Night shift
Weekend work
Ballys Corporation
Dover, DE
3 days ago
Citizens Teller - Part Time
$21 per hour
Description Starting Salary: $21 / hour and up What you'll do You’ll greet our customers in our lobbies. You’re comfortable enough with technology to demonstrate how to use our mobile app, or our ATMs, to complete simple transactions. You’ll create amazing experiences ...
Hourly pay
Part time
Local area
Flexible hours
Afternoon shift
Citizens Bank
Dover, DE
1 hour ago
Teller
$33k - $55k
Full Time TellerAt PNC, our people are our greatest differentiator and competitive advantage in the markets we serve. We are all united in delivering the best experience for our customers. We work together each day to foster an inclusive workplace culture where all of ...
Full time
Temporary work
Part time
Work experience placement
Work at office
PNC
Middletown, DE
5 days ago
PT Clerk - Pick From Store - 0388
Store Operations Support With over 2 million weekly customers and annual sales topping $5 billion, Giant is the #1 grocer in the Baltimore-Washington area. Giant Food
Giant Food
Middletown, DE
2 days ago
Clerk - Front Office
$16 per hour
...and write English. Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Ballys Corporation
Local area
Shift work
Night shift
Weekend work
Ballys Corporation
Dover, DE
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!