AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$20 - $30 per hour
...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent... ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every... ...at yet and judge it step by step. • Review — Take an existing annotation, written...SuggestedHourly payFull timeContract workImmediate startRemote workBespoke Labs
Waipahu, HI19 hours ago- Valid8 Financial, Inc. is looking for a General Clerk III to provide administrative and applicant-processing support at various CONUS Military Entrance Processing Station (MEPS) sites. The role involves tracking enlistment records, maintaining applicant files, and coordinating...SuggestedFull timeWork at officeLocal areaMonday to Friday
VALID8 Financial
Pearl City, HI4 days ago - Valid8 Financial, Inc. is looking for a General Clerk III to provide administrative and applicant-processing support at various CONUS Military Entrance Processing Station (MEPS) sites. The role involves tracking enlistment records, maintaining applicant files, and coordinating...SuggestedFull timeWork at officeLocal areaMonday to Friday
VALID8 Financial
Pearl City, HI4 days ago - ...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...SuggestedOverseas
The United States Secret Service
Pearl City, HI19 hours ago - Job Opportunity At RDSolutions The pricing intelligence pioneer, RDSolutions is hiring immediately in your area! Are you retired, want extra income, have extra time on your hands and want to stay active? Retirees are welcome and encouraged to apply! We have part-time...SuggestedExtra incomePart timeImmediate startFlexible hoursShift work
- Data Entry Operator | Junior (Remote) Wahiawa, Hawaii, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...Remote work
Only Data Entry
Wahiawa, HI5 days ago - ...focused? Do you enjoy working with the public? If you answered yes to these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State Farm insurance and financial service products. Responsibilities...Hourly payFor contractorsWork at officeFlexible hours
Thomas Jansson - State Farm Agent
Aiea, HI3 days ago $18 per hour
...Receptionist - State Farm Agent Team Member MALIA JANSSON - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent...Hourly payWork at officeMalia Jansson - State Farm Agent
Aiea, HI3 days ago$18 per hour
...Receptionist - State Farm Agent Team MemberMALIA JANSSON - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team...Work at officeMalia Jansson - State Farm Agent
Aiea, HI22 hours ago$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
Wahiawa, HIa month ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Waipahu, HI2 days ago $31.44 per hour
Production Control Clerk AGL is looking for a Production Control Clerk to work in Schofield Barracks, HI. To join our team of outstanding professionals, apply today! The Production Control Clerk must have a comprehensive knowledge of the applicable supply and maintenance...Hourly payFor contractorsWork at officeRemote workAkima
Wahiawa, HI1 day ago$42.95 per hour
...ISO 9001 quality standard certification through the Performance Review Institute (PRI). About this position: Production Control Clerk... ...systems • Occasional exposure to dust, odors, cleaning agents, and building materials may occur • Requires interaction...Contract workFor contractorsWork experience placementWork at officeBering Straits Native Corporation
Wahiawa, HI5 days ago- ...and finance in and out-processing. Peripheral task requirements include processing and management of voucher documents. Extension reviews include expenses as indicated on the voucher, supporting documents and travel orders to ensure service members are paid correctly....Work at officeMonday to FridayWeekend work
Strategic Resources Inc
Schofield Barracks, HI1 day ago $17.5 - $21 per hour
...Licenses & Certifications Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor. #J-18808-Ljbffr...Hourly payFull timeLocal areaNight shiftTSB Corporation
Mililani, HI4 days ago- Benefits 401(k) Dental insurance Health insurance Paid time off Profit sharing Vision insurance Company Description Wahiawa Physical Therapy, LLC is a local practice committed to providing high-quality physical therapy services to the community. The team prioritizes compassionate...Full timeWork at officeLocal area
Wahiawa Physical Therapy LLC
Wahiawa, HI4 hours ago $18.65 per hour
POSITION SUMMARY: Unit Office Clerk's work are self-directed and able to work independently to service management in their main or various Zippy's store locations! Individuals in these Unit Office Clerk roles are responsible for accurately verifying, receiving and...Full timePart timeWork at officeImmediate startMonday to FridayShift workFch Enterprises
Pearl City, HI19 hours ago$22.91 per hour
...penalties and civil liability. Use of Artificial Intelligence (AI) We may use Artificial Intelligence (AI) to support parts of our... ...made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools. #J-18808-Ljbffr...Contract workTemporary workMonday to FridayTEKsystems
Kapolei, HI3 days ago- ...applicant files and suspense actions, and coordinates with military, civilian, recruiting, and medical personnel. Primary Duties Receive, review, organize, and process applicant records, enlistment packets, medical documentation, waivers, source documents, and other...Civilian ContractorFull timeFor contractorsWork at officeLocal areaMonday to Friday
VALID8 Financial
Pearl City, HI5 days ago - POSITION SUMMARY: The Clinical Assistant (CA) is responsible for performing a variety of duties while working collaboratively with a multidisciplinary health care team to deliver optimal patient care services and manage care within the framework of the Patient Centered...
Waianae Coast Comprehensive Health Center
Waipahu, HI4 days ago - ...Training & development Bonus based on performance You May Be a Great Fit as a Multiline Representative at Sean Huh - State Farm Agent if: You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care about...Work at officeWork from homeFlexible hours
- Cashier Or Headcount Clerk The Cashier or Headcount Clerk will verify customer eligibility, record meals served, process authorized payments, and maintain accurate meal, headcount, and cash-accountability records. Responsibilities include operating the dining facility...Immediate start
$22.91 per hour
...penalties and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of... ...final decisions are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools....Full timeContract workTemporary workImmediate start3 days per weekKapolei, HI1 day ago- ...AFMIS Knowledge of Microsoft Office or similar office applications Strong organizational and recordkeeping skills Ability to review reports and identify discrepancies Ability to protect passwords and sensitive information Ability to meet system-access,...Contract workWork at officeRotating shiftAfternoon shiftEarly shift
LADGOV CORPORATION
Schofield Barracks, HI3 days ago $20 per hour
...insurance Opportunity for advancement Paid time off Training & development ROLE DESCRIPTION: Thomas Jansson - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Work at officeFlexible hours$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFlexible hours$1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements: Prior...For contractorsRemote workAmerican Logistics Authority
Waipahu, HI3 days ago- NorthStar Memorial Group is seeking a Clerical & Admin Specialist at Mililani Memorial Park and Mortuary in Waipio, Hawaii. The role supports administrative operations throughout the client family life cycle, from the initial meeting with families to final arrangements...
NorthStar Memorial Group
Waipahu, HI1 day ago $1,200 - $2,500 per week
A logistics company located in Waipahu, Hawaii is seeking an Entry-Level Freight Dispatcher as an independent contractor. This role involves coordinating freight, communicating with brokers and drivers, and maintaining dispatch records. Candidates should have strong organizational...For contractorsRemote workAmerican Logistics Authority
Waipahu, HI1 day ago$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly pay
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
