AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$20 - $30 per hour
...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent... ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every... ...at yet and judge it step by step. • Review — Take an existing annotation, written...SuggestedHourly payFull timeContract workImmediate startRemote work$80.5k - $137k
McCormick & Company, Inc., a global leader in the spice, flavor, and seasonings industry, is seeking an AI & Agent Product Manager, RDQR Digital Transformation. This new hire will work at the Technical Innovation Center located in Hunt Valley, MD. McCormick & Company,...SuggestedPermanent employmentWorldwide- ...Fusion Advanced Technology, LLC Quality Assurance Reviewer Baltimore, MD·Full time Company website Apply for Quality Assurance Reviewer Fusion Advanced Technology, LLC (Fusion) is seeking a motivated and mission-focused professional to perform case, payment, and service...SuggestedFull time
- Records And Registration Data ClerkReporting to the Associate Registrar, the Records and Registration Data Clerk is responsible for accurately inputting, updating, and managing data within the Office of Records and Registration. The Data Clerk handles the daily data received...SuggestedMonday to Friday
- ...we stand for as a firm.KPMG is currently seeking a Tax Return Reviewer, Global Mobility Services to join our Global Mobility Services... ...experience at an accounting firmLicensed CPA, JD/LLM or Enrolled Agent, in addition to others on KPMG's approved credential listing; any...SuggestedLocal area
$85k - $120k
...currently seeking an experienced professional Right of Way (ROW) Agent for our Baltimore, MD office. The position may involve on-site... ...transportation projects. This position is not supervisory but may assign and review the work of other real estate professionals while serving as...Full timeContract workTemporary workH1bWork at officeLocal areaRemote workRelocationRelocation packageFlexible hours- ...Hmong Chinese Language Document Reviewer Employment Type: Full Time, Mid-Level Department: Legal As a CGS Hmong Chinese Document Reviewer you will play a key role in supporting various aspects of the company's litigation portfolio including eDiscovery activities...Full timeWork at officeFlexible hours
- ...programme, project and cost management offering in markets around the world.Job DescriptionTurner & Townsend are seeking a Invoice Reviewer / Coordinator. The ideal candidate will have extensive experience reviewing construction invoices, schedules of values, supporting...Contract workFor contractorsWork at officeFlexible hours
- CGS Federal (Contact Government Services) is seeking a Mid-Level Hmong Chinese Language Document Reviewer to join its in-house legal team. The role focuses on supporting eDiscovery activities, workflow management, and litigation support within a fast-paced department....
- Job Description Job Description Benefits: ~401(k) ~401(k) matching ~ Dental insurance ~ Health insurance ~ Paid time off ~ Tuition assistance ~ Vision insurance Job description: I. Job Overview We are seeking a detail-oriented and highly ...Work at office
$103.24k - $133.2k
...Special Agent Job POSITION OVERVIEW The position advertised has been exempted from the federal civilian hiring freeze. Job Title SPECIAL... ...and/or exchange information. REQUIRED DOCUMENTATION Please review the list of required documents below and submit them with your...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate start- ...Online Cruise Booking Agent (Remote)Sparks, Maryland, United StatesAbout the JobWe are a travel services company dedicated to helping clients plan and book unforgettable cruise vacations. Partnering with leading cruise lines and travel providers, we deliver seamless booking...Remote workFlexible hours
$48.85k
...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step...$17 per hour
...General information Job Title Prospect Dispatch Agent - BWI Date Monday, September 28, 2026 Entity Prospect International Airport Services State Maryland City Baltimore Base Pay Rate: $ 17.00...Daily paidFull timeTemporary workPart timeSeasonal workWork at officeImmediate startShift workNight shiftRotating shiftWeekend work- ...to” carriers for reliability, integrity, and OUR IMPECIBLE on-time pickup & delivery record. We’re growing & looking for top notch Agents in the industry to partner with us! Interchange agreements with all ship & rail lines Dispatching & scanning software provided....Weekly pay
$80k - $100k
...including prospect activity, opportunity stages, expected close dates, and revenue forecasts. Participate in sales meetings, pipeline reviews, and strategic planning sessions. Provide market feedback and competitive insights to support the development of Klik Solutions'...Contract workFor contractorsLocal area$100k
...property damage claims. We are currently hiring Surveillance Agents to conduct stationary/mobile surveillance across the country.... ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and...Hourly payFull timeCurrently hiringFlexible hours$47 per hour
...to join our network of industry subject matter experts. The Agent may perform a variety of responsibilities based on the assignment... ..., as assigned. Secure the client's residential perimeter and review CCTV, manage access controls, and respond to alarms, as assigned...Full timeTemporary workWork at officeLocal areaFlexible hours- ...Airport Cruise Guest Service Agent This is a fast-paced high energy part-time position requiring a positive, efficient and professional image. Agents will work at the airport, directly with the public and are responsible for providing service for arriving and departing...Part timeImmediate startWeekday work
$65k
...improvement and innovation. We embrace emerging technologies, including AI, to help our employees work more efficiently, solve problems, and... ...of Dominion's lending operations. What You'll Do Track, review, and clear post-closing exceptions to ensure loan files are...Temporary workWork at office- ...ownership of their work, explore emerging technologies, including AI, and identify better ways to improve efficiency, enhance the... ...Risk Management & Compliance In This Role You Will: Track, review, and clear post-closing exceptions to ensure loan files are complete...Temporary workWork at office
$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift- Missions Dans le cadre d'un service public de proximit et dans une relation d'interface avec les lus, la secrtaire de mairie et la population, conduit l'ensemble des activits lies l'entretien et la valorisation des espaces publics et des btiments communaux. Savoir...Permanent employmentTemporary work
$10k
...ensuring your CV and skills are a good match.As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal Law... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the application...Work experience placementImmediate startNight shift- ...Final Expense Agent We're excited to invite ambitious, friendly go-getters to join our dynamic team as Final Expense Agents. This isn't a cookie-cutter jobit's a modern career path built for people who want flexibility, purpose, and real earning potential. You'll have...Remote workFlexible hours
$22.24 per hour
Bowhead seeks a General Clerk III in Baltimore, MD to provide document management and administrative support to ICE-OPLA. Responsibilities include mail handling, file interfiling, FOIA support, and data entry into PLAnet, with adherence to records schedules and security...Hourly payFull timeLocal area- ...Gate Agent Position As a Gate Agent with ProLog, Inc., you will play a vital role in providing Air Mobility Command (AMC) Gateway... ...Efficiently process passengers using automated and manual systems. Review military travel orders to determine eligibility, operate...Part timeWork at office
- ...night. We are currently looking for a genuine, driven Multi-line Agent to join our close-knit office. Our culture is built on... ...Owings Mills community. Manage policy renewals and conduct annual reviews to ensure plans remain aligned with client life changes. Maintain...Work at officeWork from homeSleeping nightsFlexible hours
- Customs and Border Protection (CBP) seeks Border Patrol Agent to join a large federal law enforcement team protecting the United States at and beyond the border. You will detect, question, and inspect at risk points, while enforcing immigration and customs laws with vigilance...
- Customs and Border Protection (CBP) is actively recruiting Border Patrol Agents to support national security and border integrity. The role involves detecting and questioning individuals, inspecting documents, and conducting patrols using diverse assets and night operations...Night shift
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!





