AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$20 - $30 per hour
...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent... ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every... ...at yet and judge it step by step. • Review — Take an existing annotation, written...SuggestedHourly payFull timeContract workImmediate startRemote workBespoke Labs
Pearl City, HI1 day ago$30 per hour
Seeds Renewables is seeking a personnel for waste management who will be responsible for packing, transporting, and disposing of waste material while adhering to environmental policies. You will need to be in rotation for overnight work on neighbor islands, which requires...SuggestedHourly payCasual workNight shiftSeeds Renewables
Pearl City, HI4 days ago- Job Opportunity At Rdsolutions The pricing intelligence pioneer, RDSolutions is hiring immediately in your area! Are you retired, want extra income, have extra time on your hands and want to stay active? Retirees are welcome and encouraged to apply! We have part-time...SuggestedExtra incomePart timeImmediate startFlexible hoursShift work
RD SOLUTIONS INC
Waipahu, HI4 days ago - ...focused? Do you enjoy working with the public? If you answered yes to these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State Farm insurance and financial service products. Responsibilities...SuggestedHourly payFor contractorsWork at officeFlexible hours
$18 per hour
...Receptionist - State Farm Agent Team Member MALIA JANSSON - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent...SuggestedHourly payWork at office- Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFull timeFlexible hours
- Clinical Assistant The Clinical Assistant (CA) is responsible for performing a variety of duties while working collaboratively with a multidisciplinary health care team to deliver optimal patient care services and manage care within the framework of the Patient Centered...
Waianae Coast Comprehensive Health Center
Waipahu, HI2 days ago - DutiesHelpCounts change fund, sets up cash drawers, and verifies the accuracy of each drawer.Turns on the master cash control processor and sets up the office terminal(s) for the day's business.Issues change to cashiers. Operates cash register, as needed.Periodically checks...Work at office
US Department of War
Schofield Barracks, HI4 days ago - Teller Position Tellers control and account for all funds received and deposited daily within the Defense Commissary Agency (DeCA) system, performing a variety of duties including setting up cash drawers, receiving funds from cashiers, and processing customer refunds...
US Government Jobs
Schofield Barracks, HI2 days ago - Performs relief teller services at various APFCU branches not limited to both teller services and member services in accordance with quality service objectives and credit union policies and procedures (transaction processing, account maintenance, product cross-sales, balancing...RelocationFlexible hours
Aloha Pacific Federal Credit Union
Waipahu, HI5 days ago - USA Government Jobs - - Responsibilities: Counts change fund, sets up cash drawers and verifies accuracy; Operates cash register as needed; Bags/seals cash and checks; prepares change for secure pickup; Verifies cashier records against change funds and sales records; Monitors...
USA Government Jobs
Schofield Barracks, HI3 days ago $22.91 per hour
...penalties and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our... ...made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools. #J-18808-Ljbffr...Full timeContract workTemporary workImmediate start3 days per weekTEKsystems
Kapolei, HI3 days ago$3,141 per month
Kipapa Elementary in Mililani, Hawaii is hiring for Office Assistant II and III. This posting covers both levels; the authorized title is Office Assistant III, with the option to hire at Office Assistant II if recruitment challenges occur. Salary ranges shown are $3,14...Full timeWork at officeGovernmentJobs.com
Mililani, HI4 days ago$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
Schofield Barracks, HImore than 2 months ago- Job Title Actively greets the members and provides a high level of member service while assisting members on the showroom sales floor. Answers member questions, demonstrates merchandise, and assists in processing member's Costco.com orders. Performs cash register and...Temporary work
$18.65 per hour
POSITION SUMMARY: Unit Office Clerk's work are self-directed and able to work independently to service management in their main or various Zippy's store locations! Individuals in these Unit Office Clerk roles are responsible for accurately verifying, receiving and...Full timePart timeWork at officeImmediate startMonday to FridayShift workFch Enterprises
Pearl City, HI1 day ago- ...Training & development Bonus based on performance You May Be a Great Fit as a Multiline Representative at Sean Huh - State Farm Agent if: You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care about...Work at officeWork from homeFlexible hours
- ...AFMIS Knowledge of Microsoft Office or similar office applications Strong organizational and recordkeeping skills Ability to review reports and identify discrepancies Ability to protect passwords and sensitive information Ability to meet system-access,...Full timeContract workWork at officeRotating shiftAfternoon shiftEarly shift
LADGOV CORPORATION
Schofield Barracks, HI5 days ago - ...and finance in and out-processing. Peripheral task requirements include processing and management of voucher documents. Extension reviews include expenses as indicated on the voucher, supporting documents and travel orders to ensure service members are paid correctly....Work at officeMonday to FridayWeekend work
$31.44 per hour
Production Control Clerk AGL is looking for a Production Control Clerk to work in Schofield Barracks, HI. To join our team of outstanding professionals, apply today! The Production Control Clerk must have a comprehensive knowledge of the applicable supply and maintenance...Hourly payFor contractorsWork at officeRemote work$20 per hour
...insurance Opportunity for advancement Paid time off Training & development ROLE DESCRIPTION: Thomas Jansson - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Work at officeFlexible hours$22.91 per hour
...penalties and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of... ...final decisions are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools....Full timeContract workTemporary workImmediate start3 days per weekKapolei, HI22 days ago$1,200 - $2,500 per week
Entry-Level Freight Dispatcher – Weekly Earnings $1,200–$2,500We are seeking reliable and organized individuals for an Entry-Level Freight Dispatcher opportunity. This is an independent contractor role supporting owner-operators with daily dispatch operations.Estimated...For contractorsAmerican Logistics Authority
Waipahu, HI2 days ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
- ...Fit as a Customer Retention Specialist at Sean Huh - State Farm Agent if: You’re the person people call first when life goes... ...existing customers regarding renewals and coverage options. Review accounts and assist with updates or changes as needed. Identify...Work at officeWork from homeFlexible hours
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
Schofield Barracks, HImore than 2 months ago- Job Type: Full-Time / Contract Position Work Schedule: Rotating shifts based on delivery, receiving, storage, inventory, and meal-production requirements. Work may include early mornings, evenings, weekends, holidays, and emergency feeding operations within the dining...Full timeContract workRotating shiftAfternoon shiftEarly shift
Ladgov Corporation
Schofield Barracks, HIa month ago - Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekend work
- Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and ...Hourly payWeekend work
$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...Hourly payFull timeWork at officeNight shiftUSPS
Mililani, HI2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
