AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour

Full-time

Bespoke Labs

Kenner, LA

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Apply

Vacancy posted 1 day ago

Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Kenner, LA vacancy

Export Documentation Specialist
$50k
Export Documentation Specialist MSC USA - New Orleans Looking for a career opportunity but have no experience? At MSC, we are looking for candidates that would like to challenge themselves in the world of logistics. We offer a diverse environment, and it is important for...
Suggested
Contract work
Temporary work
Work at office
Local area
Remote work
Overseas
Monday to Friday
Flexible hours
MSC Mediterranean Shipping Company SA
Metairie, LA
2 days ago
Document Processing Specialist
Job Summary The Document Processing Specialist is responsible for preparing, formatting, and editing a wide range of banking documents. This role requires advanced technical proficiency, a high degree of accuracy, and the ability to work efficiently in a fast‑paced, deadline...
Suggested
Work at office
Sustainable World, Inc.
Metairie, LA
5 days ago
Exit Booth Agent - New Orleans Airport
$14 per hour
...Exit Booth Agent Enterprise Mobility operates the Enterprise Rent-A-Car, National Car Rental, and Alamo car rental brands and has frequently been named one of the top places to work. Enterprise Mobility is the largest and fastest growing privately owned automotive rental...
Suggested
Hourly pay
Full time
Local area
Monday to Friday
Shift work
Weekend work
Enterprise Mobility Inc
Kenner, LA
2 days ago
Airport Agent - Ramp
About Our Company Evolution Parking & Guest Services is America’s exclusive provider of specialized valet parking and front door services exclusively for hotels and resorts nationwide. Our strict focus on only serving the hotel industry affords our clients the benefit ...
Suggested
Work experience placement
EVOLUTION PARKING & GUEST SERVICES
Harahan, LA
5 days ago
Special Agent, $40,000 Recruitment Incentive
...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...
Suggested
Overseas
The United States Secret Service
Harahan, LA
2 days ago
Special Agent
$93.75k - $133.2k
...The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, your career is defined by how you think, analyze, and act in moments that directly impact national security. By harnessing your background to transition into federal...
Work at office
Local area
Shift work
Federal Bureau of Investigation (FBI)
Harahan, LA
5 days ago
Senior Right of Way Agent
...is expanding. We’re looking for a full-time Senior Right of Way Agent to help advance public works projects in South Louisiana to construction... ...the ability to: Conduct title research, prepare abstracts, and review abstracts prepared by others for compliance, completeness and...
Full time
Temporary work
Work at office
Local area
Remote work
Relocation
CIVIX
Harahan, LA
1 day ago
Special Agent: Military/ Law Enforcement Expertise
$103.24k - $133.2k
...position advertised has been exempted from the federal civilian hiring freeze. Use your military background to become an FBI special agent! The transition from law enforcement to special agent is more than a career move—it’s an opportunity to upskill and become an expert...
Work experience placement
Work at office
Local area
Federal Bureau of Investigation (FBI)
Harahan, LA
1 day ago
Right OF Way Agent
...Civix Consulting Group in the Greater New Orleans Area seeks a Senior Right of Way Agent to lead land acquisition through pre-planning and project delivery. You will conduct title research, read engineering maps, draft agreements and coordinate with teams and clients across...
Remote work
CIVIX
Harahan, LA
2 days ago
Insurance Staff Position - State Farm Agent Team Member
$40k - $60k
...Future Opening: Insurance Staff Position - State Farm Agent Team Member Position Overview Successful State Farm Agent is seeking a qualified... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. As an...
For contractors
Work at office
Mia Delouise - State Farm Agent
Harahan, LA
5 days ago
Intern - State Farm Agent Team Member
$13 - $15 per hour
Mendy Dunn State Farm Agency Internship Join the Mendy Dunn State Farm Agency this summer for an enriching internship opportunity in the heart of Metairie, Louisiana. We are seeking energetic and enthusiastic college interns to be a part of our dynamic insurance sales...
Hourly pay
Full time
Internship
Summer internship
Work at office
Mendy Dunn
Metairie, LA
4 days ago
Special Agent: Psychology & Counseling Expert
...The Federal Bureau of Investigation (FBI) is seeking individuals to transition from psychology-related roles into Special Agent positions in New Orleans. The role emphasizes applying crisis intervention experience to investigations addressing cyber threats, terrorism,...
Work at office
Federal Bureau of Investigation (FBI)
Harahan, LA
3 days ago
Intern - State Farm Agent Team Member
$10 - $15 per hour
...Responsive Recruiter Barcley Steib - State Farm Agent is seeking an organized and efficient specialist to fulfill administrative duties and support our agency in gaining and keeping customers. As an Intern - State Farm Agent Team Member with our agency, your attention...
Hourly pay
For contractors
Internship
Work at office
Barcley Steib - State Farm Agent
Harahan, LA
4 days ago
Multi-Line Representative - State Farm Agent Team Member
$30k - $90k
...Multiline Representative at Tony Bordlee - State Farm Agent Benefits: ~401(k) matching ~ Flexible schedule ~ Paid time off You May Be a Great Fit if: You're the person people call first when life goes sideways - calm, steady, and ready to help....
For contractors
Work at office
Work from home
Flexible hours
Tony Bordlee - State Farm Agent
Metairie, LA
4 days ago
Criminal Investigator (Special Agent) (Open Continuous Announcement)
...Responsibilities The position is for a Criminal Investigator (Special Agent) with primary responsibility for prevention, detection, and... ...in operational planning, searches, seizures, arrests, reviewing evidence, and preparing case reports. GS‑12 – One year of specialized...
Work at office
Local area
ATF
Harahan, LA
2 days ago
Office Representative - State Farm Agent Team Member
$40k - $60k
...State Farm Agent Aspirant Are you driven by the idea of owning your own business one day and building a lasting career in financial services? As a State Farm Agent Aspirant, you'll be developed into the kind of leader who can run a successful State Farm agencycombining...
For contractors
Work at office
Mendy Dunn
Metairie, LA
3 days ago
Document Specialist
$20 - $30 per hour
Company Description Work with Us. Change the World. At AECOM, we're delivering a better world. Whether improving your commute, keeping the lights on, providing access to clean water, or transforming skylines, our work helps people and communities thrive. We are the...
Hourly pay
Temporary work
Work at office
Local area
Immediate start
Worldwide
Relocation package
Flexible hours
Night shift
AECOM
New Orleans, LA
3 days ago
Document Control Specialist
$70k - $75k
Job Description This Opportunity WSP is seeking a Document Control Specialist to join our Project Controls team. This position can be based out of any of our locations across the US. Your Impact Duties and Responsibilities: Support the Document Control...
Contract work
Work at office
Local area
Flexible hours
WSP
New Orleans, LA
4 days ago
Receptionist - State Farm Agent Team Member
$18 - $25 per hour
...Receptionist - State Farm Agent Team Member As a Receptionist - State Farm Agent Team Member with Chad Wood - State Farm Agent, you... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use...
Hourly pay
For contractors
Work at office
Chad Wood - State Farm Agent
Kenner, LA
9 hours ago
Future Opening: Receptionist - State Farm Agent Team Member
...Receptionist - State Farm Agent Team Member Successful State Farm Agent is seeking a qualified professional to join their winning... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. As an...
For contractors
Work at office
Immediate start
Kimberly Bergeron - State Farm Agent
Metairie, LA
4 days ago
Receptionist - State Farm Agent Team Member
$40k - $50k
...for advancement ~ Paid time off ~ Signing bonus ~ Vision insurance ROLE DESCRIPTION: Barcley Steib - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual...
For contractors
Work at office
Relocation package
Flexible hours
State Farm
Harahan, LA
5 days ago
Document Control Specialist
...and audits. Track MOC closeout deadlines and ensure the collection of required supporting files. Organize and support PSM program reviews and compliance verifications. Training & Certification Tracking Maintain training matrices and update employee training records....
For contractors
Work at office
Monday to Friday
UBE C1 Chemicals America
Waggaman, LA
5 days ago
Document Controls Specialist
This Opportunity WSP is seeking a Document Control Specialist to join our Project Controls team. This is a position located in either Texas/Arkansas/Louisiana locations. Job Description This Opportunity WSP is seeking a Document Control Specialist to join our Project Controls...
Contract work
Work at office
Local area
Flexible hours
WSP in the U.S.
New Orleans, LA
3 days ago
Document Control Specialist
Summary Crescent Power Systems is committed to providing high-quality integrated electrical buildings that meet industry standards and client specifications. The Document Control Specialist will play an essential role in ensuring that all project documents are organized...
Work experience placement
Crescent Power
New Orleans, LA
1 day ago
Agent's Executive Assistant - State Farm Agent Team Member
...Agent's Executive Assistant – State Farm Agent Team Member Position Overview Are you outgoing and customer-focused? Do you enjoy working with the public? If you answered yes to these questions, working for a State Farm independent contractor agent may be the career for...
Hourly pay
For contractors
Mia Delouise - State Farm Agent
Harahan, LA
5 days ago
Document Control Specialist
Optimal Field Services, a Crest Industries company, is seeking an experienced Document Controller to join our team. The Document Controller for Optimal Field Services serves as the primary clerical representative for field execution, quality control and quality assurance...
For subcontractor
Crestoperations
New Orleans, LA
2 days ago
Receptionist - State Farm Agent Team Member
...agency opened in 2018 and is a close-knit team of four focused on growth, service, and supporting one another. I began my career as an Agent Aspirant right out of college and worked as a team member for two years before opening my own agency. For our team, we offer a...
Work at office
Flexible hours
Taylor Lambert - State Farm Agent
Metairie, LA
12 days ago
SPECIAL AGENT
$55.1k - $57.91k
.... ** Pay above the minimum for superior qualifications may be granted under Civil Service Commission Rule IV Sec. 2.7 based on a review of the applicant pool, candidate's verified experience, education, and/or possession of related certifications. Kind of Examination...
Full time
Part time
Work experience placement
Work at office
Remote work
Flexible hours
Shift work
City of New Orleans, LA
New Orleans, LA
1 day ago
Special Agent
...Special Agent Position Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and protection. Special Agents play a critical role in...
United States Secret Service
New Orleans, LA
2 days ago
SPECIAL AGENT
...classification. Pay above the minimum for superior qualifications may be granted under Civil Service Commission Rule IV Sec. 2.7 based on a review of the applicant pool, candidate's verified experience, education, and/or possession of related certifications. Kind Of...
Full time
Part time
Work at office
City of New Orleans
New Orleans, LA
4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!