AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- Job Title Records and Registration Data Clerk Job Description Reporting to the Associate Registrar, the Records and Registration Data Clerk is responsible for accurately inputting, updating, and managing data within the Office of Records and Registration. The ...SuggestedMonday to Friday
Baltimore City Community College
Baltimore, MD3 days ago - ...programme, project and cost management offering in markets around the world.Job DescriptionTurner & Townsend are seeking a Invoice Reviewer / Coordinator. The ideal candidate will have extensive experience reviewing construction invoices, schedules of values, supporting...SuggestedContract workFor contractorsWork at officeFlexible hours
- ...project and cost management offering in markets around the world. Job Description Turner & Townsend are seeking a Invoice Reviewer / Coordinator . The ideal candidate will have extensive experience reviewing construction invoices, schedules of values,...SuggestedContract workFor contractorsWork at officeFlexible hours
Turner & Townsend
Baltimore, MD13 days ago $25.18 - $40.26 per hour
...based kitchen audits to reduce and eliminate loss of FNS assets due to lack of knowledge, skill, theft, or fraud through continual review and revision of policy and procedures. Incorporates constructive and beneficial methods of change developed throughout the school year...SuggestedHourly payTemporary workWork experience placementSummer workWork at officeWork visaFlexible hours- Core responsibilities Review customer, provider, case, and payment records against QA criteria and control standards. Document defects, observations, trends, and required follow-up actions clearly. Support 100% or targeted pre-authorization reviews where required....Suggested
$95k - $110k
...role ensures that drawings, specifications, submittals, RFIs, change orders, and correspondence are properly logged, distributed, reviewed, and archived in accordance with project requirements. The Document Control Specialist acts as the central hub for information flow...Full timeContract workFor contractorsFor subcontractorWork at officeStacy Witbeck
Baltimore, MD3 hours ago- Job Description Job Description Benefits: ~401(k) ~401(k) matching ~ Dental insurance ~ Health insurance ~ Paid time off ~ Tuition assistance ~ Vision insurance Job description: I. Job Overview We are seeking a detail-oriented and highly ...Work at office
DM Enterprises
Baltimore, MD9 days ago $28 - $38 per hour
Construction Document Control Specialist Full Time • Baltimore 401(k) Bonus based on performance Key Responsibilities Provide technical and administrative support Prepare, process, log, and file incoming and outgoing project documents in Prolog Input and retrieve ...Hourly payFull time$103.24k - $133.2k
...Special Agent Job POSITION OVERVIEW The position advertised has been exempted from the federal civilian hiring freeze. Job Title SPECIAL... ...and/or exchange information. REQUIRED DOCUMENTATION Please review the list of required documents below and submit them with your...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate startFederal Bureau of Investigation (FBI)
Baltimore, MD3 days ago$93.75k - $133.2k
...Special Agent Job Position Overview: The position advertised has been exempted from the federal civilian hiring freeze. Job Title:... ...and/or exchange information. Required Documentation Please review the list of required documents below and submit them with your application...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate startClearanceJobs
Baltimore, MD1 day ago- ...Special Agent Position Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and protection. Special Agents play a critical role in...
United States Secret Service
Baltimore, MD4 days ago $100k
...property damage claims. We are currently hiring Surveillance Agents to conduct stationary/mobile surveillance across the country.... ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and...Hourly payFull timeCurrently hiringFlexible hoursMarshall Investigative Group
Lutherville Timonium, MD5 days ago- ...integrity management, feasibility studies, surveys, constructability reviews, turnkey delivery services, and more. MasTec Professional... ...more innovative organization. Job Summary The Right of Way Agent’s main purpose is to assist the Right of Way (ROW) Department in...Permanent employmentTemporary workFor contractorsWork at officeLocal areaRelocationFlexible hours
MasTec Advanced Technologies
Baltimore, MD3 days ago - Air Cargo Ramp Clerk-Warehouse-BWIJoin the Worldwide Flight Services family and contribute to the timely delivery of cargo shipment, luggage, business to customer delivery, and on-time flights while operating safely and securely. We perform at the highest level for our...Full timePart timeFor contractorsWorldwideRotating shift
Worldwide Flight Services
Baltimore, MD5 days ago - ...services at over 300 locations in 65 countries, across 6 continents.But at the heart of our business is our people.Role PurposeThe Ramp Agent plays a vital role in ensuring top tier performance in all aspects of baggage handling, exceeding customer expectations while...
Menzies Aviation
Baltimore, MD2 days ago - ...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...Full timeOverseas
The United States Secret Service
Baltimore, MD3 days ago - ...grade level, you will have the opportunity to learn to perform these duties and receive training to help you grow in this position. Reviews proposed and final drafts of Spanish and English tax forms, instructions, schedules, public use forms, form letters, taxpayer...Work at office
- ...business units can be found at: Position(s) are to be filled in following area(s): TS - CARE-Customer Assistance, Relationship and Education Tax Forms & Publications - Other Tax Law Section REVIEW THE ADDITIONAL INFORMATION BELOW FOR FURTHER DETAILS US Government Jobs
$20 per hour
...Front Office Agent Live Your Passion. Add Your Magic. At Montage International, we are doing something different, something exciting and it takes passionate people to bring our vision to life. We have built a culture that leads with the important notion we live by...Hourly payLocal area$18 - $22 per hour
...Scheduling Agent Scheduling agents schedule appointments for LifeBridge Health Medical Group primary care practices. Facilitate messages to providers for referral requests, prescription refill requests and non-urgent medical concerns. Great opportunity to be part...Work at officeRemote workMonday to FridayShift work$17 - $23.58 per hour
...Contact Center Agent I The Contact Center Agent I uses protocols, guidelines, and appropriate software systems to handle customer requests coming into the Contact Center by phone, EMR portal, LBH website, or other unified communications. The agent handles requests in...- Job Opportunity At RDSolutions The pricing intelligence pioneer, RDSolutions is hiring immediately in your area! Are you retired, want extra income, have extra time on your hands and want to stay active? Retirees are welcome and encouraged to apply! We have part-time...Extra incomePart timeImmediate startFlexible hoursShift work
- Fifth Third Bank is seeking an Equipment Finance Documentation Specialist to manage the end-to-end documentation and closing of equipment finance transactions. You will collaborate with clients and internal deal team members to ensure policy, compliance, and timely funding...
$93.75k - $133.2k
The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, your career is defined by how you think, analyze, and act in moments that directly impact national security. By harnessing your background to transition into federal...Work at officeLocal areaShift work- About the job Online Cruise Booking Agent (Remote) About Us: We are a travel services company dedicated to helping clients plan and book unforgettable cruise vacations. Partnering with leading cruise lines and travel providers, we deliver seamless booking experiences and...Remote jobFlexible hours
$103.24k - $133.2k
...position advertised has been exempted from the federal civilian hiring freeze. Use your accounting background to become an FBI special agent! The transition from finance to special agent is more than a career move—it’s an opportunity to upskill and become an expert in...Work experience placementWork at officeLocal area$60k - $75k
...~ Experience managing relationships with carriers, vendors, and agents ~ This is an on-site position and candidates must reside in either... ...track cargo, update clients, and resolve operational issues Review, audit, and improve export processes to enhance efficiency and...Contract workWork from homeWorldwideRelocation$97.3k - $125.54k
As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau’s approach to safeguarding our nation. At the FBI, you will have the opportunity to channel your expertise...Work at officeLocal area$13 - $17 per hour
...Curbside Concierge - Airline Check-in Agent Bags, a Metropolis company, is on a mission to make every moment matter for a world on the go. As part of the largest mobility network in North America, we combine innovative technology with a customer-first mindset to transform...Hourly payTemporary workImmediate startFlexible hoursShift workWeekend workEarly shiftSP Plus Corporation
Halethorpe, MD2 days ago- ...About Crisis24 Crisis24 is a global, AI-enhanced provider of travel risk management, mass communications, critical event management,... ...profile clients, executives, and organizations across the globe. Our agents are not just security professionals, they are trusted protectors,...Work at office
Crisis24
Baltimore, MD1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
- state farm agent Baltimore, MD
- tsa agent Baltimore, MD
- operations agent Baltimore, MD
- import export agent Baltimore, MD
- commissioning agent Baltimore, MD
- remote chat agent Baltimore, MD
- agent Baltimore, MD
- executive protection agent Baltimore, MD
- special agent Baltimore, MD
- work from home chat agent Baltimore, MD