Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Trajectory Annotator and Reviewer

$20 - $30 per hour
Full-time

Bespoke Labs

Type: Contract, hourly

Location: Remote

Hours: 20–30 per week

Pay: $20–30/hour, based on experience and language coverage

Start: Immediate

ABOUT THE ROLE

We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.

You will do two jobs, and you should expect either on any given day.

•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.

• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.

TASKS YOU'LL SEE

• Feature build — Add a working feature to a live library without breaking anything that already worked.

• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.

• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.

Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.

WHAT YOU JUDGE IN A TRAJECTORY

• Was the command right for the state the environment was actually in?

• Did the agent read the previous output correctly?

• Was the step wrong, or only inefficient — these are scored differently.

• Where did the run first go off course — usually earlier than where it visibly broke.

• Did the agent notice its own mistake and recover, or keep building on a false assumption?

• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?

WHAT WE NEED FROM YOU

• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.

• Languages — Strong in Python or Go, and able to read a language you've never used.

• Linux — Comfortable with logs, running processes, build failures, and containers.

• Workflow — Everyday Git, diffs, pull requests, and issue tracking.

• Debugging — Able to work with no test suite and no error message pointing at the cause.

• Focus — Able to hold context across a long run, because step 74 can depend on step 12.

• Writing — Clear English, since every judgement needs an explanation another engineer can check.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Agent Trajectory Annotator and Reviewer in Memphis, TN vacancy
  • $20 - $30 per hour

     ...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent...  ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every...  ...at yet and judge it step by step. • Review — Take an existing annotation, written... 
    Suggested
    Hourly pay
    Full time
    Contract work
    Immediate start
    Remote work

    Bespoke Labs

    Southaven, MS
    1 day ago
  •  ...Purpose : Provide expertise in all Company transactional issues to include risk, compliance, insurance and other issues presented. Review, draft exceptions, advise, assist in negotiation of, obtain approval and finalize all contracts of the Company within Company... 
    Suggested
    Contract work
    Flexible hours

    Barnhart

    Memphis, TN
    4 days ago
  • $51.63k - $124.72k

     ...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S...  ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before... 
    Suggested
    Full time
    Work experience placement

    Marcom Group

    Memphis, TN
    12 days ago
  • Job Opportunity at Millstone Medical Come join Millstone Medical and become part of a high growth company where you will have the opportunity to learn and grow in an employee-centric culture and see the importance of your work! Millstone is a fast growing, fast paced medical...
    Suggested
    Work at office
    Work from home

    Millstone Medical

    Olive Branch, MS
    3 days ago
  •  ...document control processes using Aconex, Procore, Bluebeam, SharePoint, and other EDMS platforms Perform advanced document markup, review, and collaboration using Bluebeam Revu while enforcing version control, numbering, and revision tracking Process incoming and... 
    Suggested
    Work at office

    SpaceXSI

    Southaven, MS
    6 days ago
  • $48.85k

     ...Special Agent Joining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13,... 

    US Government Jobs

    Memphis, TN
    3 days ago
  •  ...it for you? Travel for free with your family and friends on flights across the American Airlines global network. Comprehensive benefits package which includes health, dental, prescription, and vision coverage so you stay healthy. Participat Agent, Airline, Insurance... 

    Envoy Air

    Memphis, TN
    1 day ago
  • $19.34 per hour

    Job Duties Ensures safe and legal handling of dangerous goods including: proper separation, palletizing and aircraft/truck loading. Ensures related dangerous goods paperwork is accurate, complete, and timely, including proper distribution to all necessary parties. Performs...
    Hourly pay
    Full time
    Monday to Friday
    Shift work

    FedEx

    Memphis, TN
    14 hours ago
  • $100k

     ...property damage claims. We are currently hiring Surveillance Agents to conduct stationary/mobile surveillance across the country....  ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and... 
    Hourly pay
    Full time
    Currently hiring
    Flexible hours

    Marshall Investigative Group

    Memphis, TN
    1 day ago
  • $12 per hour

     ...through their journeys—saving time, reducing hassle, and ensuring every step feels effortless and efficient.This is a Curbside Check-in Agent, also known as a Skycap, position starting at $12.00 (+ Tips $$$$) Part-Time Positions Available Must have the ability and... 
    Hourly pay
    Temporary work
    Part time
    Immediate start
    Flexible hours
    Shift work
    Night shift
    Weekend work
    Early shift

    SP Plus Corporation

    Memphis, TN
    3 days ago
  • $51.63k - $124.72k

    Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law...  ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the... 
    Full time
    Work experience placement
    Immediate start
    Night shift

    US Customs and Border Protection

    Southaven, MS
    2 days ago
  • $125 per hour

     ...identities of signers, witness their signatures, and administer oaths or affirmations as necessary. Collaborate closely with real estate agents, lenders, and title companies to coordinate signings and address any specific requirements. Maintain accurate records of... 

    Certified Mobile Notary Service

    Olive Branch, MS
    3 days ago
  • Job Description Job Description Description: JOB TYPE: PRN Who We Are: As the only independent and physician-led faculty practice plan of the University of Tennessee Health Science Center, University Clinical Health (UCH) offers best-in-class clinical care...
    For contractors
    Relief

    University Clinical Health

    Memphis, TN
    20 days ago
  •  ...based on performance ~ Competitive salary You May Be a Great Fit as a Multiline Representative at Shantorra Owens - State Farm Agent if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care... 
    Work at office

    Shantorraismyagent

    Memphis, TN
    3 days ago
  •  ...advancement Paid time off Profit sharing Training & development About our Agency With over 43 years as a State Farm agent, I’ve built a stable, experienced agency focused on consistency, service, and long-term success. Our team of three works closely... 
    Work at office
    Flexible hours

    Charles Giordano

    Germantown, TN
    3 days ago
  • $70k - $80k

     ...schedule About our Agency: I opened my agency in September of 2015 after working as a State Farm team member for another agent, which gave me a strong foundation for building a high-performing, people-focused office. I'm married, born and raised in Memphis,... 
    For contractors
    Work at office
    Local area
    Flexible hours

    State Farm

    Germantown, TN
    3 days ago
  • $75k - $85k

     ...About Our Agency: I opened my agency in 2013 after working as a State Farm Agency Aspirant, gaining hands-on experience with another agent before stepping into ownership. Today, our team has grown to 15 members, and we operate with a strong growth mindset and an "all in... 
    For contractors
    Work at office
    Flexible hours

    Zach Jaworski - State Farm Agent

    Memphis, TN
    4 days ago
  • $45k

     ...for successful careers. If you want to work in an environment that is fun, challenging, and rewarding, then Elvin Taylor - State Farm Agent may be the right fit for you! If you want to make an impact, see immediate success, we could be the place for you! If you're willing... 
    For contractors
    Work at office
    Immediate start

    Elvin Taylor - State Farm Agent

    Memphis, TN
    4 days ago
  • $50k - $100k

     ...Do you aspire to someday run your own business, be an advisor looked to in your community, and lead a team? Will Patton - State Farm Agent is searching for an outgoing and customer-focused individual who can recognize opportunities and strategically turn leads into long-... 
    For contractors
    Work at office
    Flexible hours

    Will Patton - State Farm Agent

    Southaven, MS
    14 hours ago
  • $55k - $75k

     ...of two. I've been with State Farm since 2005 and come from a long line of insurance professionals as a third-generation State Farm agent. That background helped shape my passion for the industry and for building strong relationships with customers. I'm a proud dad of... 
    For contractors
    Work at office
    Flexible hours

    State Farm

    Olive Branch, MS
    20 hours ago
  • $16 - $17 per hour

     ...State Farm Independent Contractor Agent Are you outgoing and customer-focused? Do you enjoy working with the public? If you answered...  ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Work with... 
    Hourly pay
    For contractors

    Wesley Channels - State Farm Agent

    Memphis, TN
    3 days ago
  • $40k

     ...businesses, and your communities. As a criminal Investigator (Special Agent), you'll receive paid training, develop investigative skills,...  ...your selections. Pick the job that matches your location. Review job description and click apply Have a copy of your resume... 

    2020 People

    Memphis, TN
    12 days ago
  •  ...to consider working in the office of Colin Vickrey - State Farm Agent. As a member of our agency team, you have an opportunity to experience...  .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work... 
    For contractors
    Work at office
    Flexible hours

    Colin Vickrey - State Farm Agent

    Memphis, TN
    7 days ago
  • $40k - $60k

     ...Acquisitions Specialist at Shantorra Owens - State Farm Agent Build a career with a respected Memphis insurance leader. Shantorra Owens is a highly experienced State Farm agent serving Memphis and the surrounding region for over 17 years, known for delivering personalized... 
    For contractors
    Work at office

    Shantorra Owens - State Farm Agent

    Memphis, TN
    3 days ago
  • $40k - $80k

     ...results you create, we want to hear from you. APPLY TODAY We review all applications within 24 hours because we respect your time....  ...Life insurance Retirement planning State Farm agents are independent contractors who hire their own employees. State... 
    Full time
    For contractors

    State Farm

    Memphis, TN
    2 days ago
  • $750 per week

     ...time Enjoy free samples from our sponsors and partners in exchange for your honest feedback of their products. You get to review and use new products or services before they are released to the public. You must apply on our website and complete a set of... 
    Extra income
    Full time
    Temporary work
    Part time
    Local area
    Remote work
    Work from home
    Flexible hours

    Apex Focus Group

    Bartlett, TN
    14 hours ago
  •  ...Job Description Job Description Air General, a national cargo handling company, is looking for a full time Warehouse Agent at our Memphis Ocean facility in Memphis, TN to handle the transfer of cargo to and from our client and their customers. This position focuses... 
    Full time
    Flexible hours
    Shift work
    Night shift
    Weekend work

    Air General Incorporated

    Memphis, TN
    2 days ago
  •  ...fast-paced environment and takes pride in maintaining exceptional attention to detail. Essential Duties & Responsibilities Review incoming emails and shipment documentation. Verify receipt of all required import and customs documents. Open files in the company... 
    Hourly pay
    Full time
    Work at office

    Carmichael International Service

    Memphis, TN
    4 days ago
  • About the job Data Entry Assistant Junior - Remote How To Apply Please send an email to ****@*****.*** with the subject " Applications " and your resume in order to receive the steps to continue the process . Thank you. CHECK YOUR EMAIL, INCLUDING YOUR ...
    Remote work

    Seekmate

    Memphis, TN
    14 hours ago
  •  ...time off ~ Training & development ~401(k) ~ Bonus based on performance ROLE DESCRIPTION: As aTelemarketer - State Farm Agent Team Member for Zach Jaworski - State Farm Agent, your creativity and strategy promote the continued growth of our agency. Your diversified... 

    Zach Jaworski - State Farm Agent

    Memphis, TN
    15 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!