AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$25 per hour
...About the gig Join our distributed team as a Remote AI Task Reviewer and help improve AI systems by completing short, style online tasks. This is an easy-to-learn role with full training and ongoing support. No prior experience is required. What you'll do Log in...SuggestedSecond jobImmediate startRemote workWork from homeFlexible hoursRLDG
Flint, MI7 days ago$25 per hour
About the Role Join our distributed team and help evaluate online platforms through interactive testing assignments. This is a flexible, remote opportunity with full training provided and no prior experience required. Reel Edge is a gambling technology company ...SuggestedRemote workWork from homeFlexible hours$97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...SuggestedWork at officeLocal areaFederal Bureau of Investigation (FBI)
Flint, MI3 days ago- ...Contact Center Agent I The External Agent I is responsible for providing exceptional member service, achieving sales goals, and meeting quality assurance (QA) benchmarks by applying excellent product knowledge, strong problem-solving skills, and a commitment to delivering...SuggestedRemote work
ELGA Credit Union
Grand Blanc, MI4 days ago - ...and personal achievement. If you want to work in an environment that is fun, challenging, and rewarding, then Bob Love - State Farm Agent may be the right fit for you! About Our Agency Our mission is to help people manage the risks of everyday life, recover from...SuggestedFor contractorsWork at office
Bob Love - State Farm Agent
Flushing, MI2 days ago $45k - $65k
...Competitive salary ~ Free food & snacks You May Be a Great Fit as a Multiline Representative at Ryan Lukovich - State Farm Agent if: You're the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care...Hourly payFor contractorsWork at officeState Farm
Grand Blanc, MI2 days ago- Data Entry Operator | Junior (Remote) Flint, MI, United States Job Openings Data Entry Operator | Junior (Remote) About the Job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying, Check Your Inbox or Spam...Remote job
$750 per week
ApexFocusGroup collaborates with research firms, universities, and trusted brands to gather genuine consumer insights. We help connect everyday people with flexible remote market research opportunities that influence real products, services, and customer experiences. Role...Part timeCasual workRemote workFlexible hoursAFG
Burton, MI4 days ago$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office$16 - $20 per hour
Ace Hardware Stockers and Receivers are responsible for receiving shipments, unloading trucks, organizing inventory in the stockroom, stocking shelves, building product displays, performing inventory counts, and keeping the warehouse and sales floor organized. Overnight...Hourly payFull timeNight shiftAce Hardware
Flint, MI3 days ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Burton, MI1 day ago - NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Flint, MI1 day ago - Position Summary: Create an outstanding customer experience through exceptional service. Establish and maintain a safe and clean environment that encourages our customers to return. Assist the department manager in reaching sales and profit goals est
Kroger
Flint, MI5 days ago $23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFlexible hours- Supply Chain Clerk Under the direct supervision of the Supply Chain Manager, provides for receiving, storing and issuing of hospital merchandise and assisting with inventory. Responsible for the staging of the excess materials to be shipped and put away at off site ...
- Medical Records Clerk Genesee County - Flint, MI 48502 Overview Position Type: Full Time Job Shift: Day Education Level: High School Category: Health Care Description This position is responsible for the maintenance of all medical records which includes ...Full timeWork at officeImmediate startShift work
Hamilton Community Health Network Inc
Flint, MI1 day ago - ...members, in person, by phone, or by mail. Answers questions, resolves problems, and explains membership, warehouse, and credit programs. Reviews membership, warehouse, and credit programs to keep knowledge up to date. Physically checks item availability in the warehouse,...Temporary work
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly pay- ...Hurley Family Standards of Behavior. Works under the general supervision of the departmental director or designee who assigns and reviews work for completeness and accuracy. Responsibilities Processes in-person, telephone, and written requests for the release of...Work at office
- Join a Team That Makes a Difference At Dort Financial Credit Union, we're more than a workplacewe're a mission-driven team committed to enriching lives. That includes our members, employees, and the communities we serve. Our vision is to be a leader in our industry ...Full timeFlexible hours
Dort Financial Credit Union
Flint, MI1 day ago $1,000 per month
About the Organization Sovita Credit Union is a locally owned and operated financial institution serving education and healthcare workers, their families and their communities. Since 1934, Sovita Credit Union's focus is on raising up the life's work of those who work hardest...Full timeContract workPart timeMonday to ThursdayFlexible hoursShift workNight shiftDay shiftSovita Credit Union
Flint, MI2 days ago- Bank TellerPosition Summary: To provide exceptional, high-quality service and direct the delivery of products/services through sales efforts through face to face and drive thru services. Must be a team player with a can-do-attitude, display a high level of integrity, and...Part timeWork at officeFlexible hoursShift work
Financial Plus Credit Union - Michigan
Flint, MI5 days ago $40.12k
Job Description Job Description Starting Pay: $19.2885 Hourly ($40,120)Step A: $20.4630 Hourly ($42,563)Step B: $23.6817 Hourly ($49,258)Step C: $24.2563 Hourly ($50,453)Step D: $24.8553 Hourly ($51,699)Step E: $25.4817 Hourly ($53,002)Step F: $26.1317 Hourly ($54,...Hourly payGenesee County, MI
Flint, MI3 days ago- ...Accurately post payments, adjustments, and remittances into Traumasoft and internal billing systems. Invoice production Generate, review, and submit invoices for ambulance transports, ensuring compliance with payer requirements. Traumasoft data management Add and...Remote job
- Float Csr/Teller - Pontiac/Clarkston - Full-Time Make banking a Fifth Third better We connect great people to great opportunities. Are you ready to take the next step? Discover a career in banking at Fifth Third Bank. GENERAL FUNCTION: Provide excellent customer ...Full timeWork at office
- ...Medical Center's Courtesy Performance Standards. Works under the direction of the Director of Patient Business Services or designee who reviews and inspects work for conformance to established standards and procedures. Exercises working supervision over employees engaged in...Full timeWork experience placementShift work
$23.03 - $31.16 per hour
...Other benefits: optional legal and pet insurance, transportation savings and more How You'll Make An Impact In This Role Review participant medical records and accurately assign diagnosis and procedure codes for Medicare and Medicaid, while meeting...Hourly payFull timeLocal areaRemote workAscension
Flint, MI5 days ago- Job Description Job Description: Tellers provide our members with the best personal experience possible, processing member banking transactions accurately and efficiently. Tellers cross sell credit union products and services and perform other duties as assigned. Job/Task...Part timeMonday to ThursdayFlexible hoursShift workNight shiftDay shift
- Oak Street Health is seeking a Medical Scribe to support primary care providers with real-time documentation, enabling clinicians to focus on patient care. You will observe visits, document encounters, and help with coding and post-visit summaries in a fast-paced clinic...
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekend work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
