Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Baltimore, MD vacancy
  • $20 - $30 per hour

     ...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent...  ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every...  ...at yet and judge it step by step. • Review — Take an existing annotation, written... 
    Suggested
    Hourly pay
    Full time
    Contract work
    Immediate start
    Remote work

    Bespoke Labs

    Towson, MD
    1 day ago
  • $80.5k - $137k

    McCormick & Company, Inc., a global leader in the spice, flavor, and seasonings industry, is seeking an AI & Agent Product Manager, RDQR Digital Transformation. This new hire will work at the Technical Innovation Center located in Hunt Valley, MD. McCormick & Company,... 
    Suggested
    Permanent employment
    Worldwide

    McCormick & company

    Hunt Valley, MD
    2 days ago
  •  ...Fusion Advanced Technology, LLC Quality Assurance Reviewer Baltimore, MD·Full time Company website Apply for Quality Assurance Reviewer Fusion Advanced Technology, LLC (Fusion) is seeking a motivated and mission-focused professional to perform case, payment, and service... 
    Suggested
    Full time

    GoFusion

    Baltimore, MD
    3 days ago
  • Records And Registration Data ClerkReporting to the Associate Registrar, the Records and Registration Data Clerk is responsible for accurately inputting, updating, and managing data within the Office of Records and Registration. The Data Clerk handles the daily data received...
    Suggested
    Monday to Friday

    Baltimore City Community College

    Baltimore, MD
    3 days ago
  •  ...we stand for as a firm.KPMG is currently seeking a Tax Return Reviewer, Global Mobility Services to join our Global Mobility Services...  ...experience at an accounting firmLicensed CPA, JD/LLM or Enrolled Agent, in addition to others on KPMG's approved credential listing; any... 
    Suggested
    Local area

    KPMG

    Baltimore, MD
    1 day ago
  • $85k - $120k

     ...currently seeking an experienced professional Right of Way (ROW) Agent for our Baltimore, MD office. The position may involve on-site...  ...transportation projects. This position is not supervisory but may assign and review the work of other real estate professionals while serving as... 
    Full time
    Contract work
    Temporary work
    H1b
    Work at office
    Local area
    Remote work
    Relocation
    Relocation package
    Flexible hours

    Whitman, Requardt & Associates, LLP

    Baltimore, MD
    17 hours ago
  •  ...Hmong Chinese Language Document Reviewer Employment Type: Full Time, Mid-Level Department: Legal As a CGS Hmong Chinese Document Reviewer you will play a key role in supporting various aspects of the company's litigation portfolio including eDiscovery activities... 
    Full time
    Work at office
    Flexible hours

    Contact Government Services LLC

    Baltimore, MD
    17 hours ago
  •  ...programme, project and cost management offering in markets around the world.Job DescriptionTurner & Townsend are seeking a Invoice Reviewer / Coordinator. The ideal candidate will have extensive experience reviewing construction invoices, schedules of values, supporting... 
    Contract work
    For contractors
    Work at office
    Flexible hours

    Turner & Townsend

    Baltimore, MD
    4 days ago
  • CGS Federal (Contact Government Services) is seeking a Mid-Level Hmong Chinese Language Document Reviewer to join its in-house legal team. The role focuses on supporting eDiscovery activities, workflow management, and litigation support within a fast-paced department.... 

    CGS Federal

    Baltimore, MD
    17 hours ago
  • Job Description Job Description Benefits: ~401(k) ~401(k) matching ~ Dental insurance ~ Health insurance ~ Paid time off ~ Tuition assistance ~ Vision insurance Job description: I. Job Overview We are seeking a detail-oriented and highly ...
    Work at office

    DM Enterprises

    Baltimore, MD
    13 days ago
  • $103.24k - $133.2k

     ...Special Agent Job POSITION OVERVIEW The position advertised has been exempted from the federal civilian hiring freeze. Job Title SPECIAL...  ...and/or exchange information. REQUIRED DOCUMENTATION Please review the list of required documents below and submit them with your... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    Baltimore, MD
    1 day ago
  •  ...Online Cruise Booking Agent (Remote)Sparks, Maryland, United StatesAbout the JobWe are a travel services company dedicated to helping clients plan and book unforgettable cruise vacations. Partnering with leading cruise lines and travel providers, we deliver seamless booking... 
    Remote work
    Flexible hours

    HB Travels

    Sparks Glencoe, MD
    2 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    Baltimore, MD
    3 days ago
  • $17 per hour

     ...General information Job Title Prospect Dispatch Agent - BWI Date Monday, September 28, 2026 Entity Prospect International Airport Services State Maryland City Baltimore Base Pay Rate: $ 17.00... 
    Daily paid
    Full time
    Temporary work
    Part time
    Seasonal work
    Work at office
    Immediate start
    Shift work
    Night shift
    Rotating shift
    Weekend work

    Prospect International Airport Services

    Baltimore, MD
    17 hours ago
  •  ...to” carriers for reliability, integrity, and OUR IMPECIBLE on-time pickup & delivery record. We’re growing & looking for top notch Agents in the industry to partner with us! Interchange agreements with all ship & rail lines Dispatching & scanning software provided.... 
    Weekly pay

    PIONEER TRANSPORT , INC.

    Baltimore, MD
    3 days ago
  • $80k - $100k

     ...including prospect activity, opportunity stages, expected close dates, and revenue forecasts. Participate in sales meetings, pipeline reviews, and strategic planning sessions. Provide market feedback and competitive insights to support the development of Klik Solutions'... 
    Contract work
    For contractors
    Local area

    Klik.Solutions

    Towson, MD
    23 hours ago
  • $100k

     ...property damage claims. We are currently hiring Surveillance Agents to conduct stationary/mobile surveillance across the country....  ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and... 
    Hourly pay
    Full time
    Currently hiring
    Flexible hours

    Marshall Investigative Group

    Lutherville Timonium, MD
    17 hours ago
  • $47 per hour

     ...to join our network of industry subject matter experts. The Agent may perform a variety of responsibilities based on the assignment...  ..., as assigned. Secure the client's residential perimeter and review CCTV, manage access controls, and respond to alarms, as assigned... 
    Full time
    Temporary work
    Work at office
    Local area
    Flexible hours

    Pinkerton

    Owings Mills, MD
    4 days ago
  •  ...Airport Cruise Guest Service Agent This is a fast-paced high energy part-time position requiring a positive, efficient and professional image. Agents will work at the airport, directly with the public and are responsible for providing service for arriving and departing... 
    Part time
    Immediate start
    Weekday work

    SMS International Shore Operations

    Baltimore, MD
    2 days ago
  • $65k

     ...improvement and innovation. We embrace emerging technologies, including AI, to help our employees work more efficiently, solve problems, and...  ...of Dominion's lending operations. What You'll Do Track, review, and clear post-closing exceptions to ensure loan files are... 
    Temporary work
    Work at office

    Dominion Financial Services

    Baltimore, MD
    4 days ago
  •  ...ownership of their work, explore emerging technologies, including AI, and identify better ways to improve efficiency, enhance the...  ...Risk Management & Compliance In This Role You Will: Track, review, and clear post-closing exceptions to ensure loan files are complete... 
    Temporary work
    Work at office

    Dominion Financial Services

    Baltimore, MD
    18 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Baltimore, MD
    17 hours ago
  • Missions Dans le cadre d'un service public de proximit et dans une relation d'interface avec les lus, la secrtaire de mairie et la population, conduit l'ensemble des activits lies l'entretien et la valorisation des espaces publics et des btiments communaux. Savoir...
    Permanent employment
    Temporary work

    France Government

    Baltimore, MD
    4 days ago
  • $10k

     ...ensuring your CV and skills are a good match.As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal Law...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the application... 
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Baltimore, MD
    4 days ago
  •  ...Final Expense Agent We're excited to invite ambitious, friendly go-getters to join our dynamic team as Final Expense Agents. This isn't a cookie-cutter jobit's a modern career path built for people who want flexibility, purpose, and real earning potential. You'll have... 
    Remote work
    Flexible hours

    GIA Legacy Planning

    Baltimore, MD
    2 days ago
  • $22.24 per hour

    Bowhead seeks a General Clerk III in Baltimore, MD to provide document management and administrative support to ICE-OPLA. Responsibilities include mail handling, file interfiling, FOIA support, and data entry into PLAnet, with adherence to records schedules and security...
    Hourly pay
    Full time
    Local area

    UICGS / Bowhead Family of Companies

    Baltimore, MD
    17 hours ago
  •  ...Gate Agent Position As a Gate Agent with ProLog, Inc., you will play a vital role in providing Air Mobility Command (AMC) Gateway...  ...Efficiently process passengers using automated and manual systems. Review military travel orders to determine eligibility, operate... 
    Part time
    Work at office

    Prolog, Inc.

    Baltimore, MD
    4 days ago
  •  ...night. We are currently looking for a genuine, driven Multi-line Agent to join our close-knit office. Our culture is built on...  ...Owings Mills community. Manage policy renewals and conduct annual reviews to ensure plans remain aligned with client life changes. Maintain... 
    Work at office
    Work from home
    Sleeping nights
    Flexible hours

    The Atlantic District

    Owings Mills, MD
    5 days ago
  • Customs and Border Protection (CBP) seeks Border Patrol Agent to join a large federal law enforcement team protecting the United States at and beyond the border. You will detect, question, and inspect at risk points, while enforcing immigration and customs laws with vigilance... 

    Jobtome

    Kingsville, MD
    4 days ago
  • Customs and Border Protection (CBP) is actively recruiting Border Patrol Agents to support national security and border integrity. The role involves detecting and questioning individuals, inspecting documents, and conducting patrols using diverse assets and night operations... 
    Night shift

    Jobtome

    Maryland Line, MD
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!