AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...you? Jockey is seeking Part-time Seasonal Consumer Care Order Agents to join our team. Jockey’s culture reflects the values of our... ...pursuant to federal employment laws. For further information, please review the Know Your Rights ( notice from the Department of Labor....SuggestedPart timeSeasonal workWork at officeMonday to FridayShift workWeekend work
Jockey International
Kenosha, WI4 days ago $97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...SuggestedWork at officeLocal areaFederal Bureau of Investigation (FBI)
Kenosha, WI3 days ago- Sanmina Corporation is looking for a Document Control Specialist in Kenosha, Wisconsin. This role involves managing documentation for engineering and manufacturing by setting up and maintaining systems in AGILE and ORACLE. Candidates should have 4-6+ years of experience...Suggested
- Senior Electrical QA/QC Agent Overview Join a Team Where Engineering Excellence Drives Opportunity. At Affiliated Engineers, Inc. (AEI) , we bring together expertise and innovation to solve complex challenges. Since 1978, we have supported mission-driven organizations...SuggestedFor contractorsWork from homeFlexible hours
- Affiliated Engineers, Inc. (AEI) is hiring an Electrical QA/QC Agent to resolve quality control challenges and implement QC plans across mission-critical facilities. The role requires a technical or associate degree in an electrical-related field and at least five years...SuggestedRemote work
- The Federal Bureau of Investigation (FBI) is seeking a Special Agent to safeguard national security. You will apply your expertise to cyber, fraud, and public safety investigations while transitioning from a specialized career into federal law enforcement. This role offers...Work at office
- Electrical QA/QC Agent Job ID: 2026-5687 Location: US‑WI‑Kenosha Open in Texas, Georgia, Wisconsin, Ohio, and Arizona. Responsibilities Resolve quality control challenges and complete tasks including developing and implementing quality control process delivery plans...For contractorsFlexible hours
$16 - $18 per hour
A logistics service provider in Waukegan is looking for a Documentation Specialist who will be responsible for basic documentation tasks such as filing, sorting, and typing. The role involves auditing importing/exporting documents for errors and entering data into spreadsheets...Hourly payContract work$29 - $31 per hour
...supplying training and support to users as well as assist in Validation of System through upgrades and workflow changes. Will maintain and review consistency checks for document formats and contents. Will coordinate with the Change Review Team to prioritize change requests...Contract workPlanet Pharma
Waukegan, IL3 days ago- Eightelevengroup is seeking an LKC Document Administrator in Pleasant Prairie, Wisconsin. This role involves reconciling controlled forms and organizing quality documentation in compliance with QA procedures. The ideal candidate will have strong attention to detail and ...
- Sanmina-SCI Systems de México is looking for a Document Control Specialist to manage documentation processes between Engineering and Manufacturing teams. This role requires 4-6+ years in Document Control, knowledge of AGILE and ORACLE systems, and effective communication...
- LKC Document Administrator Location: Pleasant Prairie, Wisconsin Key Responsibilities Reconcile controlled forms using MasterControl and spreadsheets File and organize paper quality documentation Maintain and organize the documentation control room Prepare files for...Local area
$16 - $18 per hour
The individuals selected for this role will be doing basic documentation specialist work. They will be filing, sorting, and typing. The Documentation Specialist will be auditing importing/exporting documentation for mistakes and data entry mistakes. They may also be putting...Hourly payContract work- All Fleet Inc. is seeking a customer service professional to grow into dispatch and operations. No prior trucking or dispatch experience required, we will teach you the rest. You will handle calls from drivers and fleet customers, gather breakdown information, dispatch ...Day shift
$17 - $20 per hour
...Insurance, Business Insurance, Health Insurance, and Renters Insurance. We look forward to speaking with you! State Farm agents are independent contractors who hire their own employees. State Farm agents' employees are not employees of State Farm. Agents are...Hourly payFor contractorsWork at officeState Farm
Waukegan, IL3 days ago- ...POSITION Dispensary Agent (Full-Time) RESPONSIBILITIES Dispensary Agents may be assigned to various shifts including Front Desk, Order Picking, and Sales Floor. Responsibilities in these roles include: Management of lobby area, customer queue, ID verification...Full timeShift work
- ...Job Description Job Description POSITION IS ON-SITE BONUS OPPORTUNITIES Job Summary: Our Electronic Monitoring Agents are responsible for providing accurate, timely, and ethical monitoring of individuals, fostering program compliance while contributing...
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Kenosha, WI1 day ago - Position Summary: Create an outstanding customer experience through exceptional service. Establish and maintain a safe and clean environment that encourages our customers to return. Assist the department manager in reaching sales and profit goals est
Kroger
Kenosha, WI3 days ago - Meijer, Inc. - [Retail Sales / Team Member] As a Gas Station Clerk at Meijer, you'll: Build rapport with customers; Display a friendly and outgoing attitude through good eye contact and body language; Help customers with any questions they may have; Stock product and maintain...Immediate start
Meijer
Kenosha, WI3 days ago - Tata Consultancy Services Limited seeks an Applications Portfolio Management Data Steward to maintain data integrity in the APM system of record. You will validate data, enforce enterprise standards, and coordinate with technology owners and governance teams to ensure proper...
- Wine And Spirits Clerk/Cashier Location: Kenosha - 80th Street Position Type: Part-Time ABOUT FESTIVAL FOODS: We are a Wisconsin grocer that is committed to giving back to the communities it serves and providing guests with exceptional service and value. We...Hourly payWeekly payFull timePart timeLocal areaImmediate startDay shiftAfternoon shift
- About the job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying , Check Your Inbox or Spam Folder For next steps. A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures...Remote work
Only Data Entry
Waukegan, IL2 days ago $16.06 - $19.27 per hour
Employment Type: Full-Time, On-site #LI-ONSITE Entry Salary Range: $16.06-$19.27 per hour with an incentive opportunity based on individual performance. Comprehensive Benefits Package We’re committed to supporting your well-being and work-life balance through a robust ...Hourly payFull timeAdvia Credit Union
Kenosha, WI3 days ago$750 per week
ApexFocusGroup collaborates with research firms, universities, and trusted brands to gather genuine consumer insights. We help connect everyday people with flexible remote market research opportunities that influence real products, services, and customer experiences. Role...Part timeCasual workRemote workFlexible hoursAFG
Racine, WI3 days ago$16.06 - $19.27 per hour
Employment Type: Part-Time, On-site #LI-ONSITE Entry Salary Range: $16.06-$19.27 per hour with an uncapped incentive opportunity based on individual performance. Comprehensive Benefits Package We're committed to supporting your well-being and work-life balance through ...Hourly payPart timeAdvia Credit Union
Kenosha, WI5 days ago- About the job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying , Check Your Inbox or Spam Folder For next steps. A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures...Remote work
Only Data Entry
Waukegan, IL3 days ago - Job Description Job Description The Data Entry Specialist plays a critical role in maintaining accurate and organized data systems. As an individual contributor, this position involves routine daily tasks driven by deadlines, ensuring data integrity and supporting ...
Total Paving & Brick
Waukegan, IL6 days ago - The Rosen Automotive Group has proudly served the residents of Illinois and Wisconsin for over 30 years. We are currently seeking dedicated and talented individuals to join our growing team! The goal of our store is to meet the needs and requirements of our customers. To...Part timeWork at officeLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
