Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in San Diego, CA vacancy
  • $50 per hour

     ...Earn Up To $50/hr as an OpenClaw Agent Builder Shape the future of autonomous agents...  ...completely free ~ Over $200/month in AI access, included at no extra cost. Overview...  ...projects offer incentive payments. Please review the payment terms for each project.... 
    Suggested
    Weekly pay
    Extra income
    Contract work
    Freelance
    Internship
    Remote work
    Worldwide

    Outlier

    San Diego, CA
    7 hours ago
  •  ...families are still without adequate coverage. A wave of veteran agents is heading into retirement. And more clients than ever are...  ...discipline, and skill to directly shape their income and career trajectory. Remote options available. Full training provided. Licensing... 
    Suggested
    Remote work
    Shift work

    PSI Groups

    San Diego, CA
    4 days ago
  • $83k - $120k

    Bring your 3 years of experience to a Documentation Specialist role that rewards initiative and fresh thinking. The reward structure favors doers: $83,000 - $120,000 upfront, real general ownership, and a HFF team pulling the same direction. Key Responsibilities Convert...
    Suggested
    Full time
    Remote work

    HFF

    San Diego, CA
    2 days ago
  •  ...Claims Reviewer A Claims Reviewer is responsible for assisting the Claims Analyst and Supervisor in adjudication of all claims as it relates to San Diego PACE Program. Reviews participant statements to resolve outstanding participant responsibility. Initiates payment... 
    Suggested
    Weekly pay
    Contract work
    Work at office
    Local area
    Weekend work
    Afternoon shift

    San Ysidro Health

    San Diego, CA
    3 days ago
  • $18 - $22 per hour

    Orthodontic Dental Assistant / Records Clerk for San Bernardino GREAT Career Opportunity to learn about Orthodontic Assisting - We have a great training program Summary: The Dental Assistant provides direct support to the dentist by preparing treatment areas, assisting...
    Suggested

    Choice Healthcare Services

    San Diego, CA
    3 days ago
  •  ...with other departments and teams as needed to facilitate document scanning/retrieval. Prepares quality records prior to scanning, reviews scanning packet information to ensure clear and legible documentation and supports the Quality Management System requirements during... 
    Contract work
    Work at office

    BioTalent

    San Diego, CA
    5 days ago
  • $18 - $18.25 per hour

     ...Demonstrate proficiency in all areas of document preparation, including barcoding, prepping, scanning, and processing incoming mail. Review and analyze documents to determine the correct policy or claim number using information from multiple sources. Accurately enter... 
    Hourly pay
    Part time

    Anchor General Insurance Agency

    San Diego, CA
    3 days ago
  • $35 - $40 per hour

     ...ensuring compliance with industry standards. Key Responsibilities: Change Control Management: Oversee the change control system by reviewing, routing, assessing training impacts, and releasing change orders. Training Support: Assist in training related to the change... 
    Full time
    Temporary work
    Work at office

    Manpower, Inc.

    San Diego, CA
    1 day ago
  • $33.66 per hour

     ...Coordinator for the facility. Independently facilitate the Document Change Order and Deviations, including formatting, coordinating the review and approval process, tracking and routing controlled documents, and releasing documents in a timely manner. Ensure that any new... 
    Contract work
    Work at office
    Local area
    Immediate start
    Worldwide
    Relocation
    Work visa
    Monday to Friday
    Shift work

    GTT, LLC

    San Diego, CA
    21 days ago
  •  ...but is not limited to, meeting visual and hearing standards. Successfully complete approximately 19 weeks of employment as a Special Agent trainee, while housed at the FBI Academy at Quantico. Upon graduation from the FBI Academy, be available to transfer to one of the... 
    Full time
    Temporary work
    Traineeship
    Work experience placement
    Work at office
    Remote work

    Military, Veterans and Diverse Job Seekers

    San Diego, CA
    4 days ago
  • $48.85k

     ...Special AgentJoining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13, step... 

    US Government Jobs

    San Diego, CA
    7 hours ago
  • $93.75k - $133.2k

     ...JOB SUMMARY As an FBI Special Agent, you will have the opportunity to channel your expertise to protect our nation from cyber-attacks...  ...associated start and end dates. Uploaded resumes will not be reviewed or used for qualification purposes. Official or unofficial college... 
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Immediate start

    Federal Bureau of Investigation (FBI)

    San Diego, CA
    1 day ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Work experience placement
    Night shift

    U.S. Customs and Border Protection

    Chula Vista, CA
    12 hours ago
  • $19 per hour

     ...Pier Agent Hourly Pay Rate: $19.00 Per Hour. Metro Shore Services is recognized for its rich history of excellence in the cruise industry...  ...and timetables; reports findings to management, as established. Reviews and verifies validity of all travel documents and proof of... 
    Hourly pay
    Contract work
    Flexible hours

    Metro Cruise Services

    San Diego, CA
    6 days ago
  • $20 - $22 per hour

     ...environment for our guests, team members, and visitors. As a Security Agent, you will serve as a visible presence throughout the property,...  ...to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. #J-1... 
    Full time
    Shift work
    Night shift
    Weekend work
    Afternoon shift

    The Westgate Hotel

    San Diego, CA
    1 day ago
  •  ...Bring your energy to the role of American Airlines Gate Agent to perform responsibilities as a Gate Agent. Stay adaptable to changing priorities and business needs. Work with your team to maintain efficiency and high standards. Perks include competitive pay, flexible scheduling... 
    Flexible hours

    TradeJobsWorkforce

    Coronado, CA
    5 days ago
  • $51.63k - $124.72k

     ...Job Description Border Patrol Agent (BPA)\n With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    Spring Valley, CA
    7 hours ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    La Mesa, CA
    1 day ago
  • $45k

     ...Job DescriptionJob DescriptionJob Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO)About U.S. Customs...  ...documentation.General Employment RequirementsCandidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    MarCom Group

    Chula Vista, CA
    2 days ago
  •  ...At Connie Health, we're revolutionizing the way older Americans navigate Medicare, but we're doing it together. As an independent agent, you'll have the freedom to build your own business with the full support of our team. We've built a culture of collaboration, innovation... 
    Hourly pay
    Shift work

    Connie Health

    San Diego, CA
    3 days ago
  • Job Opportunity With RDSolutions The pricing intelligence pioneer, RDSolutions is hiring immediately in your area! Are you retired, want extra income, have extra time on your hands and want to stay active? Retirees are welcome and encouraged to apply! We have part-time...
    Extra income
    Part time
    Immediate start
    Flexible hours
    Shift work

    RDSolutions

    San Diego, CA
    2 days ago
  •  ...Remote Document Signing Agent Certified Mobile Notary Service is where dependable notaries turn skills into a thriving business. We have an immediate opening for a Remote Document Signing Agent in Chula Vista, CA (91910). You will enjoy the autonomy of independent work... 
    Remote job
    Immediate start
    Home office

    Certified Mobile Notary Service

    Chula Vista, CA
    3 days ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Linkup Inc

    La Mesa, CA
    7 hours ago
  • $20 per hour

     ...position requiring an enthusiastic, efficient and professional image. Agents working for SMS International Shore Operations US, Inc., direct...  ...luggage and completing lost luggage forms, when necessary. Reviews and verifies validity of all travel documents presented by... 
    Hourly pay
    Extra income
    Part time
    Local area
    Immediate start
    Weekday work

    SMS International Shore Operations

    San Diego, CA
    5 days ago
  • Job Title Join a team committed to providing outstanding service at Envoy Air. Job Description Would you like to work for a stable, secure, and fast-growing airline where you will be stimulated, challenged, and have the opportunity to develop your career? Come...
    Night shift
    Weekend work

    Envoy Air

    San Diego, CA
    2 days ago
  •  ...Night Audit Agent Tower23 Hotel is seeking an experienced and service-driven part-time Night Audit Agent to join our team. As San Diego's premier boutique lifestyle hotel on the beach, we pride ourselves on delivering exceptional guest experiences with style, professionalism... 
    Part time
    Local area
    Flexible hours
    Night shift
    Weekend work

    Tower23 Hotel

    San Diego, CA
    3 days ago
  •  ...disability, veteran status, or other characteristics protected by applicable law. Specific licensing requirements, fees, compensation, and contractor terms may vary by state and will be reviewed with candidates during the hiring and onboarding process. #J-18808-Ljbffr AIL
    Hourly pay
    Weekly pay
    For contractors
    Remote work
    Work from home
    Flexible hours

    AIL

    San Diego, CA
    3 days ago
  • $45k - $65k

     ...Remote Hotel Booking Agent Destination Knot is a travel company dedicated to helping clients plan seamless hotel stays and personalized travel experiences. Our mission is to provide exceptional service while creating unforgettable journeys. We value flexibility, integrity... 
    Remote job
    Immediate start
    Flexible hours

    Destination Knot

    San Diego, CA
    4 days ago
  •  ...Hotel Front Office Agent The ancestors of the Sycuan Band of the Kumeyaay Nation existed many centuries ago as a community of peoplea Tribe living together, farming, hunting and fishing to survive. Much has changed since that time. Today the Sycuan Tribe governs its... 
    Shift work

    Sycuan

    El Cajon, CA
    2 days ago
  •  ...Part Time Passport Acceptance Agent The City of National City is a full-service, general law city serving a vibrant and socioeconomically...  ...related tasks not listed here. Assist clients by accepting, reviewing, and ensuring completeness of passport applications and... 
    Contract work
    Part time
    Work at office
    Local area
    Immediate start

    City of National City

    National City, CA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!