AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour

Full-time

Bespoke Labs

Dover, DE

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Apply

Vacancy posted 5 days ago

Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Dover, DE vacancy

Seasonal Tax Reviewer
$42 - $63 per hour
...Seasonal Tax Reviewer Multiple Locations - Rockford, IL 61107; Dover DE - Dover, DE 19904 Salary Range $42.00 - $63.00 Hourly Position Type Seasonal Job Shift Day Description The Seasonal Tax Reviewer plays an important role in supporting our clients through...
Suggested
Hourly pay
Work experience placement
Seasonal work
Work at office
Work visa
Shift work
Savant Wealth Management
Dover, DE
3 days ago
Security Patrol Agent - Overnights
$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...
Suggested
Full time
Work experience placement
Immediate start
Night shift
US Customs and Border Protection
Dover, DE
4 days ago
Multi-Line Representative - State Farm Agent Team Member
$45k - $65k
...about their development. If you want to work in an environment that is fun, challenging, and rewarding, then Katie Bennett - State Farm Agent may be the right fit for you! If you're willing to work hard and you expect to get results from yourself and those around you, we...
Suggested
For contractors
Work at office
State Farm
Camden, DE
5 days ago
Security Patrol Agent - Overnights
$51.63k - $124.72k
Job DescriptionBorder Patrol Agent (BPA)\ N With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...
Suggested
Full time
Work experience placement
Immediate start
Night shift
Customs and Border Protection
Dover, DE
2 days ago
Receptionist - State Farm Agent Team Member
$40k - $50k
...Receptionist - State Farm Agent Team Member Matt Basile - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent...
Suggested
For contractors
Work at office
Matt Basile - State Farm Agent
Dover, DE
5 days ago
Document Automation Specialist
Telework Eligible Yes Major Duties Manage the full administrative responsibilities for customer asset inventory management across their respective Area of Consideration (AOR). Responsible for the accounts payable (AP), accounts receivable (AR) processes within...
Full time
Contract work
Remote work
Defense Logistics Agency
Dover, DE
3 days ago
Receptionist - State Farm Agent Team Member
...officially opened our agency on June 1, 2024, but my roots with State Farm run much deeper as I spent over 18 years working in another agent’s office before launching my own. Today, I lead a small but passionate team of three full-time professionals who are dedicated to...
Full time
Work at office
Local area
Flexible hours
Lindsay Dougherty - State Farm Agent
Dover, DE
a month ago
Receptionist - State Farm Agent Team Member
...OUR AGENCY: I opened my agency in 2022 and currently lead a team of four. State Farm has been a big part of my life - my dad was an agent for 38 years, and I spent 15 years working in his agency before becoming an agent myself. Outside of work, I'm a proud mom of two boys...
Work at office
Flexible hours
Alexandra Vazquez - State Farm Agent
Dover, DE
3 days ago
Acquisition Specialist - State Farm Agent Team Member
...advancement Paid time off Training & development You May Be a Great Fit as an Acquisitions Specialist at Richard Stock - State Farm Agent if You're the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about your community...
Work at office
Flexible hours
Richardstocksf
Dover, DE
4 days ago
Telemarketer - State Farm Agent Team Member
$13 - $18 per hour
Overview Join to apply for the Telemarketer - State Farm Agent Team Member role at State Farm Agent . Position Overview: Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities...
Hourly pay
Part time
For contractors
Work at office
Flexible hours
State Farm Agent
Dover, DE
3 days ago
Data Entry Operator | Junior (Remote)
Data Entry Operator | Junior (Remote) Dover, Delaware, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...
Remote job
Only Data Entry
Dover, DE
6 days ago
Telemarketer - State Farm Agent Team Member
...Overview: Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities: ~Work with the agent to establish and meet marketing goals. ~Develop leads, schedule...
Hourly pay
Part time
Work at office
Flexible hours
Matt Basile - State Farm Agent
Dover, DE
7 days ago
88M Motor Transport Operator - Find Your Full Time or Part Time Army Career
...organizations including Tesla, American Medical Response, Mercedes Benz, and Mustang Cat. Similar Career Fields Include: Cargo and Freight Agent, Cargo Inspector, Vehicle Operator. About Our Organization: The U.S. Army is wealth of possibilities for your future – whether...
Full time
Contract work
Part time
Immediate start
Relocation package
Flexible hours
U.S. Army Reserves
Dover, DE
11 days ago
88M Motor Transport Operator
...organizations including Tesla, American Medical Response, Mercedes Benz, and Mustang Cat. Similar Career Fields Include: Cargo and Freight Agent, Cargo Inspector, Vehicle Operator. About Our Organization: The U.S. Army is wealth of possibilities for your future – whether...
Contract work
Part time
Immediate start
Relocation package
Flexible hours
U.S. Army Reserves
Cheswold, DE
11 days ago
Citizens Teller - Part Time
$21 per hour
Description Starting Salary: $21 / hour and up What you'll do You’ll greet our customers in our lobbies. You’re comfortable enough with technology to demonstrate how to use our mobile app, or our ATMs, to complete simple transactions. You’ll create amazing experiences ...
Hourly pay
Part time
Local area
Flexible hours
Afternoon shift
Citizens Bank
Dover, DE
1 day ago
Postal Clerk - Dover
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...
Hourly pay
Full time
Currently hiring
Work at office
Postal Jobs Source (a division of Labor Services)
Dover, DE
more than 2 months ago
Construction billing specialist
Job Description Job Description CONSTRUCTION BILLING & OFFICE ADMINISTRATOR Local construction company seeking an experienced, organized professional to join our office team. Responsibilities include AIA billing/pay applications, lien waivers, certified payroll...
Full time
Work at office
Local area
Construction Services LLC
Dover, DE
3 days ago
Part Time Lead Teller
$37.5k - $62.5k
## Lead TellerApply: DE - Dover: Full time: Posted Yesterday: R235575# **Position Overview**At PNC, our people are our greatest differentiator and competitive advantage in the markets we serve. We are all united in delivering the best experience for our customers. We work...
Full time
Temporary work
Part time
Work experience placement
Work at office
PNC Financial Services Group
Dover, DE
1 day ago
Part Time Lead Teller
$37.5k - $62.5k
DE - Dover Full time R235575 Position Overview At PNC, our people are our greatest differentiator and competitive advantage in the markets we serve. We are all united in delivering the best experience for our customers. We work together each day to foster an...
Full time
Temporary work
Part time
Work experience placement
Work at office
PNC
Dover, DE
1 day ago
Dairy Clerk
...verbal instructions. Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Redner's Markets
Part time
Shift work
Redner's Markets
Dover, DE
3 days ago
Office Clerk
...scheduled work shift. Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Redner's Markets
Work at office
Shift work
Redner's Markets
Dover, DE
5 days ago
AMENDED - Initial Assistance Representative (Clerk) - Areas 1-4 (9 month Roster)
Wage and Investment Division Job Posting The wage and investment division is seeking candidates for various positions in customer assistance, relationship, and education. Vacancies will be filled in the following specialty areas: customer assistance, relationship, and...
Full time
Temporary work
Part time
Work experience placement
Work at office
Phenom People
Dover, DE
6 days ago
Customer Service Registrar
Service Line Registrar If you care about the opportunity to grow, to make a difference, to build a future and a life, then we just might have the career for you. Care to talk? Bayhealth Medical Center is Central and Southern Delaware's healthcare leader with hospitals...
Hourly pay
Full time
Work at office
Shift work
Bayhealth
Dover, DE
5 days ago
Mail Handler
$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...
Hourly pay
Full time
Flexible hours
Postal jobs listings
Little Creek, DE
5 days ago
Clerical Office Assistant
...of freight shipping and receiving is a plus. Responsibilities Process and enter freight bills accurately and in a timely manner Review freight documentation for completeness and accuracy Perform general clerical and data-entry duties Answer and direct incoming phone...
Full time
Work at office
T E C Industrial Inc
Dover, DE
3 days ago
Postal Clerk - Little Creek
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Little Creek, DE
3 days ago
Postal Clerk - Camden
NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...
Full time
US Postal Service
Camden, DE
3 days ago
Medical office admin
$18 - $21 per hour
...Experience level: Experienced Randstad is a world leader in matching great people with great companies. Our experienced agents will listen carefully to your employment needs and then work diligently to match your skills and qualifications to the right job and...
Hourly pay
Permanent employment
Temporary work
Work experience placement
Work at office
Local area
Shift work
Randstad
Dover, DE
5 days ago
Medical Office Admin
Medical Office Administrator Randstad is partnering with a well-established healthcare and clinical practice in Dover, DE, to find a professional, friendly, and reliable Medical Office Administrator. In this role, you will serve as the primary administrative support...
Work at office
Local area
Randstad
Dover, DE
4 days ago
Medical Records Coder Fulltime
...errors and denials from WQ or from other sources. Maintain AR processes to achieve organization's goals. Responsibilities: 1. Review and analyze records to identify and correct errors. 2. Prepare encounter and code correct medical billing claims generated from the...
Full time
Shift work
Bayhealth
Dover, DE
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!