AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$100k
...property damage claims. We are currently hiring Surveillance Agents to conduct stationary/mobile surveillance across the country.... ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and...SuggestedHourly payFull timeCurrently hiringFlexible hoursMarshall Investigative Group
Lutherville Timonium, MD2 days ago- Job Title Records and Registration Data Clerk Job Description Reporting to the Associate Registrar, the Records and Registration Data Clerk is responsible for accurately inputting, updating, and managing data within the Office of Records and Registration. The ...SuggestedMonday to Friday
Baltimore City Community College
Baltimore, MD3 days ago - Fifth Third Bank is seeking an Equipment Finance Documentation Specialist to manage the end-to-end documentation and closing of equipment finance transactions. You will collaborate with clients and internal deal team members to ensure policy, compliance, and timely funding...Suggested
- ...programme, project and cost management offering in markets around the world.Job DescriptionTurner & Townsend are seeking a Invoice Reviewer / Coordinator. The ideal candidate will have extensive experience reviewing construction invoices, schedules of values, supporting...SuggestedContract workFor contractorsWork at officeFlexible hours
- Sheppard Pratt in Towson, MD is seeking a Unit Secretary for the General Adult Unit. You will serve as the receptionist, manage charts, schedule meetings, maintain records, and support daily unit operations on-site. The role requires a High School Diploma and at least one...SuggestedWork at office
$45k - $75k
...team environment Opportunity for long-term career growth within the agency Opportunity to work under one of the top producing agents in the region Opportunity to have a flexible work schedule depending on performance If you're an experienced, licensed...Base plus commissionFlexible hoursJason Griswold - State Farm Agent
Parkville, MD1 day ago- ...project and cost management offering in markets around the world. Job Description Turner & Townsend are seeking a Invoice Reviewer / Coordinator . The ideal candidate will have extensive experience reviewing construction invoices, schedules of values,...Contract workFor contractorsWork at officeFlexible hours
Turner & Townsend
Baltimore, MD13 days ago $25.18 - $40.26 per hour
...based kitchen audits to reduce and eliminate loss of FNS assets due to lack of knowledge, skill, theft, or fraud through continual review and revision of policy and procedures. Incorporates constructive and beneficial methods of change developed throughout the school year...Hourly payTemporary workWork experience placementSummer workWork at officeWork visaFlexible hours- Core responsibilities Review customer, provider, case, and payment records against QA criteria and control standards. Document defects, observations, trends, and required follow-up actions clearly. Support 100% or targeted pre-authorization reviews where required....
$95k - $110k
...role ensures that drawings, specifications, submittals, RFIs, change orders, and correspondence are properly logged, distributed, reviewed, and archived in accordance with project requirements. The Document Control Specialist acts as the central hub for information flow...Full timeContract workFor contractorsFor subcontractorWork at officeStacy Witbeck
Baltimore, MD9 hours ago$40k - $45k
...2022 and has quickly grown into a strong team of eight. I bring over 30 years of experience with State Farm, including roles as an agent, sales leader, and in operations, which has helped shape a well-rounded and growth-focused approach to leading the team. I'm a graduate...For contractorsWork at officeLocal areaFlexible hoursState Farm
Parkville, MD2 days ago- Job Description Job Description Benefits: ~401(k) ~401(k) matching ~ Dental insurance ~ Health insurance ~ Paid time off ~ Tuition assistance ~ Vision insurance Job description: I. Job Overview We are seeking a detail-oriented and highly ...Work at office
DM Enterprises
Baltimore, MD9 days ago $28 - $38 per hour
Construction Document Control Specialist Full Time • Baltimore 401(k) Bonus based on performance Key Responsibilities Provide technical and administrative support Prepare, process, log, and file incoming and outgoing project documents in Prolog Input and retrieve ...Hourly payFull time$35k - $50k
...Receptionist - State Farm Agent Team Member Haley Munson - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent...For contractorsWork at office- Job Description Job Description Full job description At Anderson Automotive Group, we value our employees and we treat them like family. Each team member can take advantage of our ongoing training programs, advancement opportunities and progressive culture. We...Part timeWork at office
- ...Special Agent Position Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and protection. Special Agents play a critical role in...
United States Secret Service
Baltimore, MD4 days ago $93.75k - $133.2k
...Special Agent Job Position Overview: The position advertised has been exempted from the federal civilian hiring freeze. Job Title:... ...and/or exchange information. Required Documentation Please review the list of required documents below and submit them with your application...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate startClearanceJobs
Baltimore, MD1 day ago$103.24k - $133.2k
...Special Agent Job POSITION OVERVIEW The position advertised has been exempted from the federal civilian hiring freeze. Job Title SPECIAL... ...and/or exchange information. REQUIRED DOCUMENTATION Please review the list of required documents below and submit them with your...Permanent employmentFull timeWork experience placementWork at officeLocal areaImmediate startFederal Bureau of Investigation (FBI)
Baltimore, MD3 days ago- ...integrity management, feasibility studies, surveys, constructability reviews, turnkey delivery services, and more. MasTec Professional... ...more innovative organization. Job Summary The Right of Way Agent’s main purpose is to assist the Right of Way (ROW) Department in...Permanent employmentTemporary workFor contractorsWork at officeLocal areaRelocationFlexible hours
MasTec Advanced Technologies
Baltimore, MD4 days ago - Air Cargo Ramp Clerk-Warehouse-BWIJoin the Worldwide Flight Services family and contribute to the timely delivery of cargo shipment, luggage, business to customer delivery, and on-time flights while operating safely and securely. We perform at the highest level for our...Full timePart timeFor contractorsWorldwideRotating shift
Worldwide Flight Services
Baltimore, MD5 days ago - ...services at over 300 locations in 65 countries, across 6 continents.But at the heart of our business is our people.Role PurposeThe Ramp Agent plays a vital role in ensuring top tier performance in all aspects of baggage handling, exceeding customer expectations while...
Menzies Aviation
Baltimore, MD2 days ago - Gordon Feinblatt LLC seeks an experienced attorney to join our energy and public utility practice in Maryland. This first-chair opportunity centers on guiding high-stakes regulatory matters with clients at the center of the state’s energy transition. The role involves ...Remote work
City of Plant City
Baltimore, MD4 days ago $80k - $120k
...Job Description Job Description Director of Growth & Agent Development Keller Williams Capital Properties | Towson, MD Keller... ...recruiting) — required ~ Tech-forward leader ready to utilize AI and tech tools to drive growth ~ Active Maryland real estate license...Work at officeKeller Williams Capital Properties
Towson, MDmore than 2 months ago- ...grade level, you will have the opportunity to learn to perform these duties and receive training to help you grow in this position. Reviews proposed and final drafts of Spanish and English tax forms, instructions, schedules, public use forms, form letters, taxpayer...Work at office
- ...business units can be found at: Position(s) are to be filled in following area(s): TS - CARE-Customer Assistance, Relationship and Education Tax Forms & Publications - Other Tax Law Section REVIEW THE ADDITIONAL INFORMATION BELOW FOR FURTHER DETAILS US Government Jobs
- ...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...Full timeOverseas
The United States Secret Service
Baltimore, MD3 days ago $18 - $22 per hour
...Scheduling Agent Scheduling agents schedule appointments for LifeBridge Health Medical Group primary care practices. Facilitate messages to providers for referral requests, prescription refill requests and non-urgent medical concerns. Great opportunity to be part...Work at officeRemote workMonday to FridayShift work$17 - $23.58 per hour
...Contact Center Agent I The Contact Center Agent I uses protocols, guidelines, and appropriate software systems to handle customer requests coming into the Contact Center by phone, EMR portal, LBH website, or other unified communications. The agent handles requests in...$20 per hour
...Front Office Agent Live Your Passion. Add Your Magic. At Montage International, we are doing something different, something exciting and it takes passionate people to bring our vision to life. We have built a culture that leads with the important notion we live by...Hourly payLocal area$93.75k - $133.2k
The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, your career is defined by how you think, analyze, and act in moments that directly impact national security. By harnessing your background to transition into federal...Work at officeLocal areaShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
