AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...SuggestedWork at officeLocal areaFederal Bureau of Investigation (FBI)
Iowa City, IA3 days ago- A customer service agency in Iowa City is seeking experienced customer service professionals for their Winter Campaign Project. The role requires candidates with a minimum of 6 months of formal telephone experience and a typing speed of 40 WPM or higher. This project offers...Suggested
ACD Direct
Iowa City, IA3 days ago $16 - $18 per hour
...Intern - State Farm Agent Team Member Carolina Espinoza - State Farm Agent is seeking an organized and efficient specialist to fulfill... ...billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use...SuggestedHourly payFor contractorsInternshipCarolina Espinoza - State Farm Agent
Iowa City, IA2 days ago$50k - $75k
...before eventually pursuing my dream of agency ownership. I'm married with three daughters, and my husband — a third-generation insurance agent — now works alongside me in the business. I'm bilingual and a proud graduate of St. Ambrose University, where I earned both my B.A....SuggestedFor contractorsCasual workWork at officeFlexible hoursCarolina Espinoza - State Farm Agent
Iowa City, IA2 days ago$50k - $150k
Multiline Representative At Chad Burtch State Farm At Chad Burtch State Farm, we're here to help people protect what matters most. We take pride in serving our community with integrity, care, and real support. Whether we're helping someone navigate a claim or personalize...SuggestedWork at officeChad Burtch - State Farm Agent
Iowa City, IA5 days ago$45k - $50k
...win and achieve results, please consider joining our team. Come work with an energetic, fun team at Carolina Espinoza - State Farm Agent! About Our Agency Our mission is to help people manage the risks of everyday life, recover from the unexpected and realize...Work at officeCarolina Espinoza - State Farm Agent
Iowa City, IA3 days ago$20 per hour
...Documentation Specialist will play an important role in preparing the organization for Iowa Medicaid and Managed Care Organization (MCO) reviews and audits by supporting the development, implementation, and ongoing adherence to compliance policies and procedures....Daily paidFull timeWork at officeMonday to FridayMonday to ThursdayVibrant Homecare
North Liberty, IA1 day ago- ...Were looking for a American Airlines Gate Agent to perform daily responsibilities with dedication. Ensure compliance with company and safety standards. Work with your team to maintain efficiency and service quality. Perks include competitive pay, flexible schedules, hands...Flexible hours
TradeJobsWorkForce
Riverside, IA5 days ago - About the job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying , Check Your Inbox or Spam Folder For next steps. A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures...Remote work
Only Data Entry
Iowa City, IA2 days ago - Job Description Job Description Position: Data Entry Specialist Job Type: (Remote) SIKKY MANUFACTURING is a leading manufacturer of high-quality aftermarket automotive parts. With over 20 years of experience, we have established ourselves as a trusted brand in...Full timeWork at officeRemote work
Hillyard Inc
Iowa, Johnson County, IA6 days ago $750 per week
ApexFocusGroup collaborates with research firms, universities, and trusted brands to gather genuine consumer insights. We help connect everyday people with flexible remote market research opportunities that influence real products, services, and customer experiences. Role...Part timeCasual workRemote workFlexible hoursAFG
Coralville, IA4 days ago- Job Description Job Description Benefits: ~ Health insurance ~ Paid time off ~ Training & development ~ Vision insurance ~401(k) ~ Bonus based on performance ~ Dental insurance You May Be a Great Fit as an Acquisitions Specialist at Chad Burtch State...Work at office
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Coralville, IA1 day ago - ...at the delicatessen counter. Works with co-workers as a team to ensure customer satisfaction and a pleasant work environment. Reviews orders for the day. Understands and practices proper sanitation procedures and ensures the work area is always clean and neat....Part timeWork experience placement
Hy-Vee Food Stores
Coralville, IA1 day ago - ...development ROLE DESCRIPTION: Carolina Espinoza - State Farm Agent is seeking an organized and efficient specialist to fulfill... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary...InternshipFlexible hours
- Hy-Vee - - Responsibilities: Wash dishes and order product and supplies; Stock service areas with supplies including coffee, food, tableware, and linens; Maintain strict adherence to personal hygiene, dress standards, and store guidelines; Work with co-workers as a team...
Hy-Vee
Coralville, IA4 days ago - Clerk III - Guest Services (50%) Iowa Health Care Department of Guest Services is seeking a 50% Clerk III. Hours to be scheduled between 6:00 AM - 8:15PM Monday-Friday, 7:00a.m. - 6:00p.m. Weekends and rotating holidays. Additional hours as needed. Under general supervision...Full timePart timeWork at officeMonday to FridayWeekend work
University of Iowa
Iowa City, IA3 days ago - Hy-Vee - - Responsibilities: Greet customers and take Hy-Chi orders; Prepare and replenish Hy-Chi products; Maintain cleanliness and sanitation of the department; Operate the department register and handle payments; Assist with stocking and inventory tasks
Hy-Vee
Coralville, IA1 day ago - ...greeting every customer with a smile, taking orders at the Hy-Chi counter, and assisting them with their needs throughout the store Review orders for the day and ensure an adequate food supply is ready and on hand by developing or following a production list Prepare,...Shift work
- Server At Hy-Vee our people are our strength. We promise "a helpful smile in every aisle" and those smiles can only come from a workforce that is fully engaged and committed to supporting our customers and each other. About Hy-Vee At Hy-Vee, we believe the shortest...Work experience placementShift work
- ...faces, and replenishes merchandise and ensures highest quality standards are met. Checks in product, puts product away, and may review invoices. Maintains displays in the department and throughout the store. Understands and follows local, state, and company policies...Work experience placementLocal area
- ...person, by phone, or by mail. Answers questions, resolves problems, and explains membership, warehouse, and credit programs. Reviews membership, warehouse, and credit programs to keep knowledge up to date. Physically checks item availability in the warehouse, directs...Temporary work
$17 per hour
Teller The starting wage for this position is $17.00/hr. with a comprehensive and competitive benefits package. This position is eligible for a Spanish proficiency pay differential. Schedule: Full-time; Monday - Friday with availability between (8:15 am – 5:45 pm...Full timeBank staffWork at officeMonday to FridayNight shiftDay shiftHills Bancorporation
Iowa City, IA2 days ago$17 per hour
Hills Bank and Trust Company is seeking a full-time Teller in Iowa City, IA, to deliver exceptional customer service by processing transactions, maintaining cash drawers, and managing customer inquiries. Duties include handling deposits and withdrawals, balancing cash ...Full timeHills Bancorporation
Iowa City, IA3 days ago- Why Us? With a mantra of Empowering Human Potential, Hanger, Inc. is the world's premier provider of orthotic and prosthetic (O&P) services and products, offering the most advanced O&P solutions, clinically differentiated programs and unsurpassed customer service. ...Local area
Hanger
Iowa City, IA4 days ago $11 - $13 per hour
Box Office Attendant | Part-Time | Xtream Arena and GreenState Family Fieldhouse Under supervision of Box Office Director, the Box Office Attendant is responsible for ticket sales and customer service for all ticketed events at Xtream Arena & Greenstate Family Fieldhouse...Hourly payPart timeWork at officeLocal areaFlexible hoursNight shiftWeekend workEarly shift- ...eventually pursuing my dream of agency ownership. I’m married with three daughters, and my husband — a third-generation insurance agent — now works alongside me in the business. I’m bilingual and a proud graduate of St. Ambrose University, where I earned both my B.A....Casual workWork at officeFlexible hours
- Ehsan Islamic School, an Islamic faith-based educational institution in Iowa City, Iowa, is seeking an Office Assistant to provide comprehensive administrative support. Key Responsibilities Serve as the primary point of contact at the school's front desk for students, parents...Work at office
$17.75 per hour
Job Description Job Description The National Association of State Departments of Agriculture (NASDA) is seeking part-time, highly motivated individuals with a positive attitude and strong work ethic. Ideal candidates are self-motivated, detail-oriented, organized, ...Part timeFlexible hoursNational Association of State Departments of Agriculture
Iowa, Johnson County, IA6 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
