Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Lake Worth, FL vacancy
  • Are you passionate about the patient experience? At HCA Healthcare, we are committed to caring for patients with purpose and integrity. We care like family! Jump-start your career as a Records Clerk today with HCA Florida JFK Hospital. Job Summary And Qualifications Under...
    Suggested
    Temporary work
    Work at office
    Flexible hours

    HCA Florida JFK Hospital

    Atlantis, FL
    3 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    U.S. Customs and Border Protection

    Atlantis, FL
    1 day ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Suggested
    Full time
    Work experience placement
    Immediate start
    Night shift

    Adzuna

    Lake Worth, FL
    1 day ago
  • $10k

     ...ensuring your CV and skills are a good match.As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal Law...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the application... 
    Suggested
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Palm Springs, FL
    2 days ago
  • $74k - $106k

    Think of this Documentation Specialist job at UnitedHealth Group as Coaching without the red tape, plus a West Palm Beach, FL team that actually ships. Think $74,000 - $106,000, think full-time hours, think 4 years of Organization turning into ownership you can actually...
    Suggested
    Full time
    Remote work
    Shift work

    Energy Innovation: Policy and Technology LLC

    West Palm Beach, FL
    4 days ago
  • Job Description Job Description Looking for licensed agent with a minimum 12 months experience in personal auto and homeowners insurance some commercial preferred.

    Eaton Insurance Inc

    Lake Worth, FL
    18 days ago
  • Multiline Representative at Nancy Wolfe-Smith - State Farm AgentAt Nancy Wolfe-Smith - State Farm Agent, we're here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support. Whether we're helping someone navigate... 
    Hourly pay
    Work at office

    NANCY WOLFE-SMITH INSURANCE AGENCY, INC

    Lake Worth, FL
    2 days ago
  • Manpower in Delray Beach, FL is seeking a detail-oriented Document Production Specialist to support formatting and assembling reports, manuscripts, proposals, and marketing materials using MS Word templates and styles. The role requires proficiency with Microsoft Office...
    Work at office

    Manpower

    Delray Beach, FL
    5 days ago
  • Clerical Position in Public Safety Department Responsible clerical work which involves moderately complex and usually varied work methods and problems in the records department of the Public Safety Department. Essential Job Duties The following duties are normal...
    Work at office
    Trial period
    Flexible hours

    Village of Palm Springs, FL

    West Palm Beach, FL
    5 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    West Palm Beach, FL
    1 day ago
  • Loss Prevention ManagerCoach and train associates and leaders on shrink awareness, theft deterrence, and customer service programs.Partner with Store and AP Leadership to identify theft trends, develop action plans, and promote awareness initiatives.Deliver SMART customer...
    Local area

    Lowe's

    West Palm Beach, FL
    2 days ago
  • $40 - $50 per hour

    FIELD ASSOCIATE - PALM BEACH AREA ABOUT THE ROLE We're seeking a Field Associate to join our team in the Palm Beach area. This contracted hourly position is a 1099 position that focuses on conducting property showings for rental properties with occasional opportunities...
    Hourly pay
    Local area
    Flexible hours

    Doorstead Inc

    Palm Beach, FL
    2 days ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    West Palm Beach, FL
    5 days ago
  • We're Hiring: Licensed Insurance Broker / Insurance Sales Advisor Valora Group | West Palm Beach, FL Valora Group is looking for a motivated, licensed insurance professional to join our growing team. We help individuals and families find life insurance that protects...
    Permanent employment
    Work at office

    Confidential

    West Palm Beach, FL
    25 days ago
  • $12 per hour

     ...nd shifts onlyPrevious 1-2+ years of Customer Service experience is preferred - Paid TrainingA Curbside Concierge - Airline Check-in Agent's responsibilities include handling and securing luggage according to the Transportation Security Administrations' (TSA) standards.... 
    Hourly pay
    Temporary work
    Immediate start
    Flexible hours
    Weekend work
    Day shift
    Afternoon shift
    Early shift

    SP Plus Corporation

    West Palm Beach, FL
    6 hours ago
  • $18 - $22 per hour

     ...Flexible schedule ROLE DESCRIPTION: Dan Cavin - State Farm Agent is seeking an organized and efficient specialist to fulfill...  ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use... 
    Hourly pay
    Full time
    For contractors
    Internship
    Work at office
    Flexible hours

    State Farm

    West Palm Beach, FL
    4 days ago
  • Palm Beach County School District is seeking a records management professional to ensure compliance with Florida Public Records law and to manage electronic and physical records using current software tools. The role supports district policies and the safety and efficiency...

    TrulyHired

    West Palm Beach, FL
    5 days ago
  • $18 - $22 per hour

     ...as a team and have fun while we are doing it!Intern - State Farm Agent Team MemberDan Cavin - State Farm Agent is seeking an organized...  ...submissions and billing clarification.Use a customer-focused, needs-based review process to educate customers about insurance options.Use... 
    Full time
    For contractors
    Internship
    Work at office

    Dan Cavin - State Farm Agent

    West Palm Beach, FL
    1 day ago
  • $45k

     ...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs...  ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    MarCom Group

    West Palm Beach, FL
    2 days ago
  • $10 - $12 per hour

     ...focused? Do you enjoy working with the public? If you answered yes to these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State Farm insurance and financial service products. Responsibilities... 
    Hourly pay
    For contractors
    Internship
    Flexible hours

    State Farm

    Boynton Beach, FL
    4 days ago
  • $16.5 per hour

     ...Seasonal Return Agent Enterprise Mobility operates the Enterprise Rent-A-Car, National Car Rental, and Alamo car rental brands and...  ...number, date, time and turn into lost and found. Advise and review rental charges; and provide an accurate receipt to the customer.... 
    Full time
    Seasonal work
    Flexible hours
    Night shift
    Weekend work

    Enterprise Mobility

    West Palm Beach, FL
    4 days ago
  • $16.5 per hour

     ...Exit Booth Agent Enterprise Mobility operates the Enterprise Rent-A-Car, National Car Rental, and Alamo car rental brands and has frequently been named one of the top places to work. Enterprise Mobility is the largest and fastest growing privately owned automotive rental... 
    Local area
    Immediate start

    Enterprise Mobility

    West Palm Beach, FL
    3 days ago
  •  ...fulfilling our protective legacy. Proven to be worthy of trust and confidence. Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and... 
    Overseas
    Trial period

    U.S. Secret Service

    West Palm Beach, FL
    4 days ago
  •  ...reports findings to management, as established. Assists guests with locating luggage and completing lost luggage forms, when necessary. Reviews and verifies validity of all travel documents presented by guests for check-in. Inputs guest travel information using the... 
    Weekday work

    SMS International Shore Operations

    West Palm Beach, FL
    4 days ago
  • Federal Bureau of Investigation (FBI) in the United States seeks a highly capable Special Agent to protect the nation from cyber-attacks, terrorism, fraud, and evolving threats. This role leverages your expertise to conduct high-stakes investigations and uncover crucial... 
    Work at office

    Federal Bureau of Investigation (FBI)

    West Palm Beach, FL
    2 days ago
  • $16.5 per hour

     ...Seasonal Exit Booth Agent Enterprise Mobility operates the Enterprise Rent-A-Car, National Car Rental, and Alamo car rental brands and has frequently been named one of the top places to work. Enterprise Mobility is the largest and fastest growing privately owned automotive... 
    Part time
    Seasonal work
    Local area
    Flexible hours
    Night shift
    Weekend work

    Enterprise Mobility

    West Palm Beach, FL
    4 days ago
  •  ...Security! Salary Range: $15.00 per hour 401K with company matching! ACTS-Aviation Security, Inc. has an immediate opening for Security Agents at the West Palm Beach International Airport in West Palm Beach, FL. Established in 2001, ACTS-Aviation Security has grown to... 
    Hourly pay
    Immediate start
    Weekend work
    Afternoon shift

    ACTS-Aviation Security

    West Palm Beach, FL
    4 days ago
  •  ...holidays • A desire to be part of a team that genuinely cares about each other and our guests We are seeking a Guest Experience  Agent  to join our team. As a key member of the Eau Resort & Spa team, you will play an essential role in delivering exceptional service... 
    Full time
    Flexible hours
    Night shift
    Weekend work

    Eau Resort & Spa Palm Beach

    Lake Worth, FL
    24 days ago
  •  ...Description Benefits: Bonus based on performance Competitive salary Flexible schedule Job Title: Business Development Agent – Preowned Car Dealership Location: Lake Worth FL Job Type: Full-Time About Us: At Jacobs Preowned, we take pride in offering... 
    Full time
    Flexible hours

    J And M Auto Specialists Llc

    Lake Worth, FL
    17 days ago
  • Benefits License reimbursement Bonus based on performance Competitive salary Flexible schedule Health insurance Opportunity for advancement Paid time off Training & development Location Delray Beach, FL 33483 Responsibilities Communicate with customers to understand their...
    Work at office
    Flexible hours

    Marion Fazio - State Farm Agent

    Delray Beach, FL
    6 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!