AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...program. If you’re ready to make an impact with a stable industry leader, we want to hear from you. Your Next Career Opportunity - AI Agent Developer We are seeking a talented and passionate AI Agent Developer to join our innovative team. This role involves designing,...SuggestedWork at office
$25 per hour
About the Role Join our distributed team and help evaluate online platforms through interactive testing assignments. This is a flexible, remote opportunity with full training provided and no prior experience required. Reel Edge is a gambling technology company focused on...SuggestedRemote workWork from homeFlexible hours- ...Commercial Appraisal Reviewer Join Mercantile Bank as a Commercial Appraisal Reviewer in Lansing, Michigan where you will be at the forefront of property valuation in the banking industry. This role offers the unique opportunity to engage in detailed analysis, honing...SuggestedWork at officeRemote workMonday to FridayFlexible hours
- ...advancing people within the organization. For over a century they have focused on building exceptional partnerships with independent agents to achieve the best outcomes for its mutual customers. Since 1912 they have been the industry leader due to consistency and process...SuggestedHourly payWeekly payCasual workLocal areaMonday to Friday
FasTemps
Lansing, MI1 day ago £28k - £32k per year
Document Controller / Administrator Job Description EnergIs is currently recruiting for a Document Controller for our Co Antrim based client who operates in the construction sector. The successful Document Controller will manage quality documentation, key information/documentation...Suggested- ...INDEPENDENT FREIGHT AGENT R.B. Humphreys, Inc. — Rome, NY (Remote) ROLE OVERVIEW As an Independent Freight Agent with R.B. Humphreys, you will build and manage your own book of business while operating under our established authority. Backed by more than 60 years...Remote workRelocation package
RB Humphreys
Lansing, MI3 days ago - ...Repossession Agent RMS is seeking experienced and dependable Repossession Agents to join our growing team. This field-based position is ideal for individuals who thrive in a fast-paced environment, enjoy working independently, and possess strong problem-solving skills...Shift work
RMS
Lansing, MI1 day ago - ...essential administrative and applicant-processing support at various Military Entrance Processing Station (MEPS) sites. The role involves reviewing and processing a range of applicant documentation, as well as communicating with Army Guidance Counselors and other personnel. The...
$16 - $30 per hour
...Insurance Staff Position - State Farm Agent Team Member Full Time in Buchanan - MI Replies within 24 hours Paid time off Position Overview... ...and billing clarification. Use a customer‑focused, needs‑based review process to educate customers about insurance options. Work with...Hourly payFull timeFor contractorsMonday to FridayFlexible hoursJoejilek
Lansing, MI4 days ago$17 - $30 per hour
...Insurance Staff Position - State Farm Agent Team Member Full Time in Washington Township, MI 401(k) matching Bonus based on performance Opportunity for advancement Paid time off Role Description Adam Klemp - State Farm Agent is searching for a competitive Producer who...Hourly payFull timeFor contractorsWork at officeMiinsuranceguy
Lansing, MI2 days ago- Job Title Must be available mornings: 6 hours a week 3 hour shift on Wednesday and Saturday Must be able to work a flexible schedule including nights, weekends, and holidays. Hours will vary based upon flight schedules. Number of weekly hours may increase ...Seasonal workFlexible hoursShift workNight shiftDay shiftAfternoon shift
$97.3k - $125.54k
JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau'...Work at officeLocal area$15.5 - $20.15 per hour
As a Geek Squad Agent, you’ll be the first point of contact for people seeking technology support, knowledge and solutions for computers and other devices at our local Best Buy store. We’ll provide the training and resources you need to assess customers’ needs, perform...Hourly payFull timePart timeLocal areaWork from homeFlexible hoursNight shift$40k - $60k
Insurance Staff Position - State Farm Agent Team Member Part Time in Kalamazoo, MI Benefits 401(k) matching Bonus based on performance Free uniforms Opportunity for advancement Paid time off Position Overview Looking for a part-time job that provides meaningful work and...Part timeFor contractorsWork at officeFlexible hours- ...Agent in Training | Joey Wetoskey Agency Mason, MI Your Sales Skills Could Take You Further If you’re good with people, motivated by... ...with existing clients through regular communication and policy reviews Tracking leads, conversations, and follow-up activity using CRM...Base plus commissionFlexible hours
Farmers District 40
Lansing, MI3 days ago - ...Criminal Investigator (Special Agent) - $40,000 Recruitment Incentive Join to apply for the Criminal Investigator (Special Agent) - $40,000 Recruitment Incentive role at U.S. Secret Service . Get AI-powered advice on this job and more exclusive features. Recruitment Incentive...Full timeOverseas
US Secret Service
Lansing, MI1 day ago $20 - $30 per hour
...Products Representative - State Farm Agent Team Member Part Time in Novi, MI Position Overview Looking for a part-time job that provides... ...'s office. Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work with...Hourly payPart timeFor contractorsWork at officeGreatrates247
Lansing, MI2 days ago$60k - $90k
...Agency ManagerAt Josh Orler - State Farm Agent, we're here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support. Whether we're helping someone navigate a claim or personalize a plan, we believe in doing...For contractorsWork at officeJosh Orler - State Farm Agent
Lansing, MI2 days ago$45k - $75k
...Agent in Training Farmers Insurance | Lansing, MI Build a Career Where Your Effort Creates Opportunity If you're motivated by goals, enjoy working with people, and want a career where your performance can directly impact your income and future, insurance could be the opportunity...Farmers District 40 & 94
Lansing, MI4 days ago$40k - $80k
...Overview Office Representative - State Farm Agent Team Member. Full Time in Jackson, MI. Position Overview Do you have aspirations to... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Work with...Full timeWork at officeJoemunie
Lansing, MI3 days ago$45k - $70k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...For contractorsState Farm
East Lansing, MI1 day ago$40k - $60k
...Receptionist - State Farm Agent Team Member Josh Orler - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team...For contractorsWork at officeRelocation packageFlexible hoursJosh Orler - State Farm Agent
Lansing, MI5 days ago$45k - $60k
...Insurance Agent Team Member Trinesha launched the agency on January 1, 2011 and has since built a strong team of nine licensed agents... ...Responsibilities: Use a customer-focused, needs-based review process to educate customers about insurance options. Work with...Hourly payFor contractorsWork at officeTrinesha Goebel - State Farm Agent
Lansing, MI5 days ago- Office Representative - State Farm Agent Team Member Full Time in Manistee, MI Benefits 401(k) matching Opportunity for advancement Paid... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Hourly payFull timeFor contractorsWork at office
$32 - $42 per hour
...single patient's request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of... ...facility. Experience with significant level of coding quality review feedback Experience in computerized encoding and abstracting...Hourly payReliefRemote workRelocation packageFlexible hoursDatavant
Lansing, MI5 days ago$28 per hour
...and assignment. Interested candidates may apply for next steps. We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but...Remote jobBi-weekly payHourly payWeekly payFull timeCurrently hiringWork from home$15 - $22 per hour
...State Farm Independent Contractor Agent Are you outgoing and customer-focused? Do you enjoy working with the public? If you answered... ...customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Hourly payFor contractorsTrinesha Goebel - State Farm Agent
East Lansing, MI5 days ago- ...Description Job Description Position Overview State Farm Insurance Agent located in Bath, MI is seeking an outgoing, career-oriented... .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Work...For contractorsRelocation package
Charles Schnetzler - State Farm Agent
Bath, MI9 days ago - ...Agent in Training Holly Benson Agency | Farmers Insurance Dimondale, MI Ready for a Career That Rewards Your Drive? If you're someone who enjoys talking with people, likes working toward goals, and wants an opportunity to grow professionally, the Holly Benson Agency would...Base plus commission
Farmers District 40
Dimondale, MI5 days ago $55k - $85k
...Agent in Training | Farmers Insurance | Nikolai Jump Agency Build Your Sales Career. Grow Your Income. Create Your Future. Looking for a career where your hard work can directly impact your income and your future? The Nikolai Jump Agency is looking for a motivated, people...Base plus commissionFarmers District 40 & 94
Okemos, MI4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
- state farm agent Lansing, MI
- tsa agent Lansing, MI
- import export agent Lansing, MI
- commissioning agent Lansing, MI
- remote chat agent Lansing, MI
- agent Lansing, MI
- executive protection agent Lansing, MI
- work from home chat agent Lansing, MI
- cruise agent Lansing, MI
- telemarketer - state farm agent team member Lansing, MI