AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$100k
...Surveillance Agents Wanted Marshall Investigative Group (MIG), the industry's leading investigative firm, seeks self-motivated individuals... ...preparation for and during surveillance operations Prepare, review, and submit detailed reports of findings, incidents, and...SuggestedHourly payFull timeCurrently hiringFlexible hoursMarshall Investigative Group
Olympia, WA5 days ago- ...Online Cruise Booking Agent Olympia, Washington, United States About the Job About Us: We are a travel services company dedicated to helping clients plan and book unforgettable cruise vacations. Partnering with leading cruise lines and travel providers, we deliver...SuggestedRemote workFlexible hours
$70k - $100k
...advancement ~ Paid time off ~ Vision insurance You May Be a Great Fit as an Agency Manager at Kevin Hayward - State Farm Agent if: You take ownership and follow through until the issue is fully resolved. You communicate clearly and confidently ,...SuggestedFor contractorsWork at officeState Farm
Lacey, WA5 days ago$45k - $60k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...SuggestedFor contractorsWork at officeFlexible hoursState Farm
Olympia, WA2 days ago$50k - $70k
...rewarding, this could be the right fit for you. You May Be a Great Fit as a Multiline Representative at Melanie Bakala - State Farm Agent if: You're the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about...SuggestedFor contractorsWork at officeLocal areaFlexible hoursMelanie Bakala - State Farm Agent
Olympia, WA3 days ago- ...rewarding, this could be the right fit for you. You May Be a Great Fit as a Multiline Representative at Melanie Bakala - State Farm Agent You’re the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about your community...Work at officeLocal areaFlexible hours
Melanie Bakala - State Farm Agent
Olympia, WA4 days ago - ...greater confidence in the insights they rely on. For HDR, it means enterprise data that is ready to support analytics, automation, and AI in a way that reflects how we work.HDR is seeking a strategic-minded Resources Business Group Data Steward to shape how our data is...
- ...other records. Request clarification when document identifiers or indexing information are unclear. Perform quality-control reviews to ensure scanned images are complete, legible, properly oriented, and readable. Verify that electronic documents are accurately...Full timeContract workRemote workMonday to Friday
$18 per hour
...from the unexpected and realize their dreams. If You Want a Career, Not a Job, Then We Encourage You to Apply. State Farm agents are independent contractors who hire their own employees. State Farm agents' employees are not employees of State Farm. Agents are...Hourly payFor contractors- ...snacks ~ Paid time off ~ Vision insurance You May Be a Great Fit as a Multiline Representative at Tom Kuhlmann - State Farm Agent if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about...Work at office
- ...Benefits: Bonus based on performance Paid time off Training & development ROLE DESCRIPTION: Jarron Janski - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Work at office
- ...role that is both demanding and rewarding, this could be the right fit for you. ROLE DESCRIPTION: Melanie Bakala - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual...Work at officeLocal areaFlexible hours
- ...Training & development ~ Vision insurance You May Be a Great Fit as an Acquisitions Specialist at Tom Kuhlmann - State Farm Agent if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about your...Work at officeRelocation package
$17.38 - $21.9 per hour
Create an outstanding customer experience through exceptional service. Establish and maintain a safe and clean environment that encourages our customers to return. Assist the department manager in reaching sales and profit goals established for the department, and monitor...Hourly paySeasonal workLocal areaShift workKroger Family of Companies
Lacey, WA1 day ago$16 - $20 per hour
Ace Hardware Stockers and Receivers are responsible for receiving shipments, unloading trucks, organizing inventory in the stockroom, stocking shelves, building product displays, performing inventory counts, and keeping the warehouse and sales floor organized. Overnight...Hourly payFull timeNight shiftAce Hardware
Tumwater, WA5 days ago- ...Overview Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed. Work with the agent to...Hourly payPart timeFor contractorsWork at officeFlexible hours
- Job Description Job Description Benefits: Bonus based on performance Paid time off Flexible schedule Opportunity for advancement Training & development Job Description: We are currently seeking a dynamic and enthusiastic Telemarketer to join ...Flexible hours
- Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFull timeFlexible hours
$18 per hour
...invoices; disburses overpayments on true-up invoices. Independently reviews and interprets sold documentation and completes Billing set-up.... .... The company's expertise, combined with the most advanced AI-enabled technology available, sets the standard for solutions in...Part timeWork at officeLocal areaShift workSedgwick
Olympia, WA1 day ago$20 - $22.88 per hour
Heritage Bank has an exciting opportunity to join our organization! At Heritage Bank we are committed to delivering exceptional financial services to our customers. The Tumwater Branch is seeking a bank teller to help customers meet their financial goals by providing...Hourly payFull timePart timeWork at officeFlexible hoursShift workHeritage Bank
Olympia, WA2 days ago$20 - $22.88 per hour
Heritage Bank has an exciting opportunity to join our organization! At Heritage Bank we are committed to delivering exceptional financial services to our customers. The Lacey Branch is seeking an experienced bank teller to help customers meet their financial goals...Hourly payFull timePart timeWork at officeMonday to FridayFlexible hoursShift workHeritage Bank
Lacey, WA2 days ago$18 - $22 per hour
About the Role:As the first point of contact for most clients, Tellers play a key role in delivering on the bank's Breakthrough Client Service Standards. With guidance from the Branch Manager and Assistant Branch Manager, the Teller's primary responsibility is to ensure...Night shiftColumbia Banking System
Olympia, WA1 day ago$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
Lacey, WAmore than 2 months ago$17 per hour
Company Description HealthCare Support Staffing, Inc. (HSS), is a proven industry-leading national healthcare recruiting and staffing firm. HSS has a proven history of placing talented healthcare professionals in clinical and non-clinical positions with some of the...Full timePrivate practiceWork at officeMonday to FridayShift workHealth Care Support Staffing
Olympia, WA1 day ago$20 - $22.88 per hour
Job Description Job Description Heritage Bank has an exciting opportunity to join our organization! At Heritage Bank we are committed to delivering exceptional financial services to our customers. The Lacey Branch is seeking an experienced bank teller to...Hourly payFull timeMonday to FridayHERITAGE BANK
Lacey, WA20 days ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements:...For contractorsRemote workAmerican Logistics Authority
Olympia, WA4 days ago$1,200 - $2,500 per week
A logistics company is seeking an Entry-Level Freight Dispatcher in Olympia, WA. This independent contractor role supports owner-operators by coordinating freight and managing dispatch operations. Candidates should possess strong communication and organizational skills...For contractorsRemote workAmerican Logistics Authority
Olympia, WA2 days ago$1,200 - $2,500 per week
Entry-Level Freight Dispatcher – $1,200–$2,500 WeeklyWe are seeking reliable and organized individuals for an Entry-Level Freight Dispatcher opportunity. This is an independent contractor role supporting owner-operators with daily dispatch operations.Estimated Weekly Earnings...For contractorsAmerican Logistics Authority
Olympia, WA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
