AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$20 - $30 per hour
...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent... ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every... ...at yet and judge it step by step. • Review — Take an existing annotation, written...SuggestedHourly payFull timeContract workImmediate startRemote work- ...Purpose : Provide expertise in all Company transactional issues to include risk, compliance, insurance and other issues presented. Review, draft exceptions, advise, assist in negotiation of, obtain approval and finalize all contracts of the Company within Company...SuggestedContract workFlexible hours
$51.63k - $124.72k
...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S... ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before...SuggestedFull timeWork experience placement- Job Opportunity at Millstone Medical Come join Millstone Medical and become part of a high growth company where you will have the opportunity to learn and grow in an employee-centric culture and see the importance of your work! Millstone is a fast growing, fast paced medical...SuggestedWork at officeWork from home
- ...document control processes using Aconex, Procore, Bluebeam, SharePoint, and other EDMS platforms Perform advanced document markup, review, and collaboration using Bluebeam Revu while enforcing version control, numbering, and revision tracking Process incoming and...SuggestedWork at office
$48.85k
...Special Agent Joining the Secret Service as a Special Agent will allow you to perform critical protective and investigative assignments. The Special Agent position starts at a base salary of $48,854.00 (GL-07, step 1), with promotion potential to $118,204.00 (GS-13,...- ...it for you? Travel for free with your family and friends on flights across the American Airlines global network. Comprehensive benefits package which includes health, dental, prescription, and vision coverage so you stay healthy. Participat Agent, Airline, Insurance...
$19.34 per hour
Job Duties Ensures safe and legal handling of dangerous goods including: proper separation, palletizing and aircraft/truck loading. Ensures related dangerous goods paperwork is accurate, complete, and timely, including proper distribution to all necessary parties. Performs...Hourly payFull timeMonday to FridayShift work$100k
...property damage claims. We are currently hiring Surveillance Agents to conduct stationary/mobile surveillance across the country.... ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and...Hourly payFull timeCurrently hiringFlexible hours$12 per hour
...through their journeys—saving time, reducing hassle, and ensuring every step feels effortless and efficient.This is a Curbside Check-in Agent, also known as a Skycap, position starting at $12.00 (+ Tips $$$$) Part-Time Positions Available Must have the ability and...Hourly payTemporary workPart timeImmediate startFlexible hoursShift workNight shiftWeekend workEarly shift$51.63k - $124.72k
Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift$125 per hour
...identities of signers, witness their signatures, and administer oaths or affirmations as necessary. Collaborate closely with real estate agents, lenders, and title companies to coordinate signings and address any specific requirements. Maintain accurate records of...- Job Description Job Description Description: JOB TYPE: PRN Who We Are: As the only independent and physician-led faculty practice plan of the University of Tennessee Health Science Center, University Clinical Health (UCH) offers best-in-class clinical care...For contractorsRelief
- ...based on performance ~ Competitive salary You May Be a Great Fit as a Multiline Representative at Shantorra Owens - State Farm Agent if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care...Work at office
- ...advancement Paid time off Profit sharing Training & development About our Agency With over 43 years as a State Farm agent, I’ve built a stable, experienced agency focused on consistency, service, and long-term success. Our team of three works closely...Work at officeFlexible hours
$70k - $80k
...schedule About our Agency: I opened my agency in September of 2015 after working as a State Farm team member for another agent, which gave me a strong foundation for building a high-performing, people-focused office. I'm married, born and raised in Memphis,...For contractorsWork at officeLocal areaFlexible hours$75k - $85k
...About Our Agency: I opened my agency in 2013 after working as a State Farm Agency Aspirant, gaining hands-on experience with another agent before stepping into ownership. Today, our team has grown to 15 members, and we operate with a strong growth mindset and an "all in...For contractorsWork at officeFlexible hours$45k
...for successful careers. If you want to work in an environment that is fun, challenging, and rewarding, then Elvin Taylor - State Farm Agent may be the right fit for you! If you want to make an impact, see immediate success, we could be the place for you! If you're willing...For contractorsWork at officeImmediate start$50k - $100k
...Do you aspire to someday run your own business, be an advisor looked to in your community, and lead a team? Will Patton - State Farm Agent is searching for an outgoing and customer-focused individual who can recognize opportunities and strategically turn leads into long-...For contractorsWork at officeFlexible hours$55k - $75k
...of two. I've been with State Farm since 2005 and come from a long line of insurance professionals as a third-generation State Farm agent. That background helped shape my passion for the industry and for building strong relationships with customers. I'm a proud dad of...For contractorsWork at officeFlexible hours$16 - $17 per hour
...State Farm Independent Contractor Agent Are you outgoing and customer-focused? Do you enjoy working with the public? If you answered... ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Work with...Hourly payFor contractors$40k
...businesses, and your communities. As a criminal Investigator (Special Agent), you'll receive paid training, develop investigative skills,... ...your selections. Pick the job that matches your location. Review job description and click apply Have a copy of your resume...- ...to consider working in the office of Colin Vickrey - State Farm Agent. As a member of our agency team, you have an opportunity to experience... .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work...For contractorsWork at officeFlexible hours
$40k - $60k
...Acquisitions Specialist at Shantorra Owens - State Farm Agent Build a career with a respected Memphis insurance leader. Shantorra Owens is a highly experienced State Farm agent serving Memphis and the surrounding region for over 17 years, known for delivering personalized...For contractorsWork at office$40k - $80k
...results you create, we want to hear from you. APPLY TODAY We review all applications within 24 hours because we respect your time.... ...Life insurance Retirement planning State Farm agents are independent contractors who hire their own employees. State...Full timeFor contractors$750 per week
...time Enjoy free samples from our sponsors and partners in exchange for your honest feedback of their products. You get to review and use new products or services before they are released to the public. You must apply on our website and complete a set of...Extra incomeFull timeTemporary workPart timeLocal areaRemote workWork from homeFlexible hours- ...Job Description Job Description Air General, a national cargo handling company, is looking for a full time Warehouse Agent at our Memphis Ocean facility in Memphis, TN to handle the transfer of cargo to and from our client and their customers. This position focuses...Full timeFlexible hoursShift workNight shiftWeekend work
- ...fast-paced environment and takes pride in maintaining exceptional attention to detail. Essential Duties & Responsibilities Review incoming emails and shipment documentation. Verify receipt of all required import and customs documents. Open files in the company...Hourly payFull timeWork at office
- About the job Data Entry Assistant Junior - Remote How To Apply Please send an email to ****@*****.*** with the subject " Applications " and your resume in order to receive the steps to continue the process . Thank you. CHECK YOUR EMAIL, INCLUDING YOUR ...Remote work
- ...time off ~ Training & development ~401(k) ~ Bonus based on performance ROLE DESCRIPTION: As aTelemarketer - State Farm Agent Team Member for Zach Jaworski - State Farm Agent, your creativity and strategy promote the continued growth of our agency. Your diversified...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
- tsa agent Memphis, TN
- import export agent Memphis, TN
- cruise agent Memphis, TN
- telemarketer - state farm agent team member Memphis, TN
- agent Memphis, TN
- executive protection agent Memphis, TN
- work from home chat agent Memphis, TN
- airport agent Memphis, TN
- state farm agent Memphis, TN
- remote chat agent Memphis, TN





