Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in New Brunswick, NJ vacancy
  • $25 per hour

     ...About the gig Join our distributed team as a Remote AI Task Reviewer and help improve AI systems by completing short, style online tasks. This is an easy-to-learn role with full training and ongoing support. No prior experience is required. What you'll do Log... 
    Suggested
    Second job
    Immediate start
    Remote work
    Work from home
    Flexible hours

    RLDG

    Milltown, NJ
    19 days ago
  • $150k - $180k

     ...approximately 6,165 employees and has served more than 200,000 customers across over 100 countries and regions. Job Title: Enterprise AI Agent Development Engineer Supervisor: Head of AI for Industry, Digital & Intelligent Technology Center Location: Piscataway, NJ... 
    Suggested
    Temporary work
    Work experience placement
    Flexible hours

    GenScript

    Piscataway, NJ
    19 days ago
  • $85k - $110k

     ...Job Description Job Description Document Reviewer – Steel Construction Location: Somerset, New Jersey Salary: $85,000–$110,000 per year, depending on experience Benefits: Health insurance, 401(k), PTO, and bonus Job Description An established steel... 
    Suggested
    For contractors
    Work at office

    Royalty Talent Group

    Somerset, NJ
    26 days ago
  • $17 - $21 per hour

    A leading consulting firm is seeking an entry-level Data Information / Input Assistant to support their team from home. The role involves updating and verifying data, organizing digital files, and ensuring data privacy. The ideal candidate is detail-oriented and willing...
    Suggested
    Hourly pay
    Remote work

    Wadley Donovan Gutshaw Consulting

    Bridgewater, NJ
    15 hours ago
  • $18 - $25 per hour

     ...Warehouse AgentThe Warehouse Agent is responsible for loading and unloading deliveries, documenting inventory, organizing stock, and verifying orders for stocking and shipping. This role involves processing orders in the shipping system, creating packing slips and labels... 
    Suggested
    Hourly pay
    Flexible hours
    Weekend work

    SHI GmbH

    Piscataway, NJ
    4 days ago
  • $51.63k - $124.72k

     ...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S...  ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before... 
    Full time
    Work experience placement

    Marcom Group

    South River, NJ
    8 days ago
  •  ...State Farm Insurance Agent PositionState Farm Insurance Agent located in Somerset, NJ is seeking an outgoing, career-oriented professional to join their team. As a State Farm team member for Vipul Verma - State Farm Agent, you will build and develop customer relationships... 
    Hourly pay
    For contractors
    Local area

    State Farm

    Somerset, NJ
    3 days ago
  • $51.63k - $124.72k

    Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Somerset, NJ
    3 days ago
  • $10k

     ...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal...  ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    New Brunswick, NJ
    1 day ago
  • $51.63k - $124.72k

     ...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    Customs and Border Protection

    South River, NJ
    1 day ago
  • $18 - $25 per hour

     ...Warehouse Agent The Warehouse Agent is responsible for loading and unloading deliveries, documenting inventory, organizing stock, and verifying orders for stocking and shipping. This role involves processing orders in the shipping system, creating packing slips and... 
    Hourly pay
    Flexible hours
    Weekend work

    SHI

    Piscataway, NJ
    5 hours ago
  •  ...Hiring!Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office.ResponsibilitiesEstablish customer relationships and follow up with customers, as needed.Provide prompt, accurate, and friendly... 
    Part time
    For contractors
    Work at office
    Local area

    State Farm

    Piscataway, NJ
    4 days ago
  • $20 per hour

    City, State: Basking Ridge, New Jersey Title: Room Inspector Location: Basking Ridge, NJ FLSA: Non-Exempt Status: Full time Reports to: Housekeeping Supervisor Supervises: Housekeeping Department Pay Range: ...
    Hourly pay
    Full time
    Work at office
    Flexible hours
    Afternoon shift

    Stonebridge Companies

    Basking Ridge, NJ
    3 days ago
  •  ...East Brunswick, NJ is urgently seeking 2 experienced Senior Tax Reviewers to join our growing team. This is an exciting opportunity to...  ...federal and multi-state tax compliance ~ CPA license or Enrolled Agent (EA) designation strongly preferred ~ Proficiency in tax... 

    Sai Cpa Services

    East Brunswick, NJ
    17 days ago
  • $24 - $28 per hour

    A financial institution in New Jersey is seeking a Document Specialist to monitor daily operations in the Credit Control Unit. Candidates must have a Bachelor's degree and 5-10 years of experience in corporate banking documentation. The role involves supporting Vault Custody...
    Remote work

    Confidential

    Woodbridge, NJ
    1 day ago
  • $24 - $28 per hour

    Overview Document Specialist - must have banking experience Base pay range: $24.00/hr - $28.00/hr Bill Rate: $39.48/hr Work location: Iselin, NJ Work schedule: 9-5 Monday to Friday as needed in office with the possibility of some remote work (WFH). End date: 1/14/2026 Job...
    Temporary work
    Work at office
    Remote work
    Work from home
    Monday to Friday

    Confidential

    Woodbridge, NJ
    1 day ago
  •  ...Hiring! Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed. Provide prompt,... 
    Part time
    For contractors
    Work at office
    Local area

    Vipul Verma - State Farm Agent

    North Brunswick, NJ
    2 days ago
  • $20 per hour

     ...PM education: No Degree Required Responsibilities Reviewing, sorting, and organizing stored documents and records. Supporting...  ...in matching great people with great companies. Our experienced agents will listen carefully to your employment needs and then work... 
    Hourly pay
    Permanent employment
    Temporary work
    Work experience placement
    Work at office
    Relocation
    Shift work
    Piscataway, NJ
    16 days ago
  •  ...State Farm Insurance Agent State Farm Insurance Agent located in Somerset, NJ is seeking an outgoing, career-oriented professional to join their team. As a State Farm team member for Vipul Verma - State Farm Agent, you will build and develop customer relationships within... 
    Hourly pay
    For contractors
    Local area

    Vipul Verma - State Farm Agent

    Somerset, NJ
    3 days ago
  •  ...industry is experiencing unprecedented demand. Millions of families still need financial protection while thousands of experienced agents are retiring each year—creating a tremendous opportunity for motivated sales professionals. At The Bill Lampe Group Inc. , we... 
    Remote work
    Flexible hours

    The Bill Lampe Group Inc.

    Woodbridge, NJ
    3 days ago
  • $30 - $40 per hour

     ...Know what its like to work and grow at Brillio: Click here We may use artificial intelligence (AI) tools to support parts of the hiring process such as reviewing applications analyzing resumes or assessing responses and identifying potential inconsistencies or verification... 
    Remote work

    Brillio

    Edison, NJ
    3 days ago
  • Holiday Inn South Plainfield-Piscataway is seeking a Housekeeping Inspector responsible for ensuring the highest standards of cleanliness and guest satisfaction. This role involves inspecting rooms, verifying cleanliness, and training staff while monitoring performance...
    Flexible hours

    Holiday Inn South Plainfield-Piscataway

    South Plainfield, NJ
    2 days ago
  • $22.25 per hour

    Job Duties Provides accurate information and assistance to customers, which includes responding to inquiries, problem-resolution and selling of company services. Performs administrative support functions necessary for efficient station operations. This is a non-driving...
    Hourly pay
    Minimum wage

    Shaker Fedex

    Sayreville, NJ
    4 days ago
  • $750 per week

     ...time Enjoy free samples from our sponsors and partners in exchange for your honest feedback of their products. You get to review and use new products or services before they are released to the public. You must apply on our website and complete a set of... 
    Extra income
    Full time
    Temporary work
    Part time
    Local area
    Remote work
    Work from home
    Flexible hours

    Apex Focus Group

    New Brunswick, NJ
    15 hours ago
  •  ...Agent Damage Te apasiona el sector de la automocin y el servicio al cliente? En SIXT, buscamos un Agent Damage que se encargue de revisar los archivos de daos generados en nuestras diversas sucursales a nivel nacional. Tu objetivo ser recopilar toda la informacin necesaria... 
    Flexible hours

    SIXT

    Avenel, NJ
    15 hours ago
  • Job Opportunity We've got the breakdown You've got the solution Full-time | Remote from Spain | Working for the SIXT team in Rostock, Germany Stranded drivers need a voice they can trust Be that voice When something goes wrong on the road, our customers count...
    Full time
    Remote work
    Work from home
    Home office
    Shift work

    SIXT

    Avenel, NJ
    5 hours ago
  • Job Description Job Description Outbound Call Center Sales Rep- High Earning Potential! Grapevine Pro is seeking an Outbound Sales Rep to call out to our customers across New Jersey. This friendly professional will call out to our amazing customers to follow...
    Shift work

    Grapevine Pro

    Rahway, NJ
    15 days ago
  • $40k - $55k

     ...the sale.Build lasting relationships through follow-up, policy reviews, and great service.What We're Looking ForOutgoing, confident, and...  ...insuranceLife insuranceRetirement planningState Farm agents are independent contractors who hire their own employees. State... 
    For contractors
    Work at office

    Marc Hughes - State Farm Agent

    Metuchen, NJ
    4 days ago
  •  ...~ Competitive salary ~ Dental insurance You May Be a Great Fit as a Multiline Representative at Kevin Baez - State Farm Agent if: You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care... 
    Work at office

    Njfreequote

    Montgomery, NJ
    5 days ago
  •  ...advancement Paid time off Position Overview State Farm Insurance Agent located in Bound Brook, NJ is seeking an outgoing, career-...  ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop leads... 
    Hourly pay
    For contractors
    Work at office
    Local area

    Marisella Gallardo - State Farm Agent

    Bound Brook, NJ
    5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!