AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- Job Opportunity Our client is seeking a skilled Document Control Specialist for a 6-month contract opportunity with Northern Natural Gas. This position is onsite, offers a competitive hourly pay rate, and candidates must have the legal right to work in the US without...SuggestedHourly payContract work
BizTek People
Omaha, NE3 days ago $18.16 - $20 per hour
Job Description Job Description Position: Product Document Administrator I Location: Centris HQ (Records Department) Hours: Monday - Friday, 8:30am to 5:00pm. Hourly Pay Rate: $18.16 - $20 per hour General Summary: The primary purpose of this position...SuggestedHourly payWork at officeMonday to FridayCentris Federal Credit Union
Omaha, NE4 days ago$20.78 per hour
...PositionResponsibilities Key Responsibilities Brief Description of Duties Administration Procedures Salary-$20.78-$20.79/hourly Review all incoming information for student records to ensure accuracy and completeness of each form Assist the Records Supervisor in scheduling...SuggestedHourly payContract workTemporary workFor contractorsWork at officeImmediate startDay shift$18 per hour
Job Description: GardaWorld Security Services is Now Hiring a Tactical Security Officer! Ready to suit up as a Tactical Security Guard? What matters most in a role like this is your ability to read the environment, anticipate risk, and act accordingly. Tell...SuggestedHourly payFull timeCasual workLocal areaImmediate startShift workGardaWorld Security Services U.S.
Omaha, NEjust now$22 - $28 per hour
...penalties and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our... ...our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools. #J-18808-Ljbffr ActalentSuggestedContract workTemporary workMonday to FridayShift work$93.75k - $133.2k
...Special Agent Job Posting Position Overview: The position advertised has been exempted from the federal civilian hiring freeze. Job... ...and/or exchange information. Required Documentation Please review the list of required documents below and submit them with your application...Permanent employmentFull timeTraineeshipWork experience placementWork at officeLocal areaImmediate startClearanceJobs
Omaha, NE3 days ago$22 - $28 per hour
...penalties and civil liability. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of... ...final decisions are made by our hiring team. By applying, youacknowledge and agree that your application may be reviewed using AI tools....Contract workTemporary workMonday to FridayShift workActalent
Omaha, NE5 days ago- ...Job Description As a Geek Squad Agent, you'll be the first point of contact for people seeking technology support, knowledge and solutions for computers and other devices at our local Best Buy store. We'll provide the training and resources you need to assess customers...Hourly payFull timePart timeLocal areaWork from homeFlexible hoursNight shift
Best Buy
Omaha, NE4 days ago - ...Special Agent Position Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and protection. Special Agents play a critical role in...
United States Secret Service
Omaha, NE1 day ago $20.13 - $20.52 per hour
Job Description: GardaWorld Security Services is Now Hiring a Surveillance Security Officer! Ready to suit up as a Surveillance Security Guard? What matters most about a role like this is your sharp eye, capturing every detail as you scan a series of screens....Hourly payFull timeCasual workLocal areaImmediate startGardaWorld Security Services U.S.
Omaha, NE1 day ago$21.65 per hour
Job Description: GardaWorld Security Services is Now Hiring a Response Security Officer! Ready to suit up as a Special Response/Flex Security Guard What matters most in a role like this is your ability to adapt from one mission to the next. Tell us about...Hourly payDaily paidFull timeCasual workLocal areaImmediate startFlexible hoursNight shiftGardaWorld
Council Bluffs, IA4 days ago- ...Benefits: Flexible schedule About the Role: Phantom Recovery LLC is looking for skilled and driven Repossession Agents to join our team in Omaha, NE. This is a fast-paced, field-based role where no two days are the same — perfect for self-starters who thrive...Flexible hoursNight shiftWeekend work
Phantom Recovery
Omaha, NE5 days ago $93.75k - $133.2k
The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, your career is defined by how you think, analyze, and act in moments that directly impact national security. By harnessing your background to transition into federal...Work at officeLocal areaShift work$97.3k - $125.54k
JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau'...Work at officeLocal area- ...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...Full timeOverseas
The United States Secret Service
Omaha, NE5 days ago - Job Title Must be able to work a flexible schedule including nights, weekends, and holidays. Hours will vary based upon flight schedules. Number of weekly hours may increase or decrease due to seasonal flight changes. Starting wage 18.00 Paid Time Off Must...Seasonal workFlexible hoursNight shiftAfternoon shift
$50k - $100k
Job Title American Senior Benefits is looking for a dedicated professional to serve those in need of insurance, especially seniors. Responsibilities Conduct an initial interview to determine the needs of the client. Introduce and clearly explain all available ...Traineeship- ...Online Cruise Booking Agent Omaha, Nebraska, United States About the Role We are seeking an Online Cruise Booking Agent to assist... ...help travelers explore cruise options, understand itineraries, review requirements, and coordinate all booking details. You will...Remote workFlexible hours
$103.24k - $133.2k
...position advertised has been exempted from the federal civilian hiring freeze. Use your mathematics background to become an FBI special agent! The transition from data to special agent is more than a career move—it’s an opportunity to upskill and become an expert in your...Work experience placementWork at officeLocal area- ...QA/QC Agent - Data Center Cedar Rapids, IA; Indianapolis, IN; Jackson, MS; Kansas City, MO; Monroe, LA; Omaha, NE Company Description We are Olsson. We engineer and design solutions that improve the world around us. As a company, we promise to always be responsive...Full timeFlexible hours
$103.24k - $133.2k
...position advertised has been exempted from the federal civilian hiring freeze. Use your psychology background to become an FBI special agent! The transition from counseling to special agent is more than a career move—it’s an opportunity to upskill and become an expert in...Work at officeLocal area$103.24k - $133.2k
The Federal Bureau of Investigation (FBI) is seeking candidates for the role of special agent. This opportunity allows individuals with a psychology background to apply their skills in investigative work that protects our nation from various threats. The position demands...Work at office$40k - $60k
...Office Representative - State Farm Agent Team Member Successful State Farm Agent is seeking a qualified professional to join their... .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Hourly payFor contractorsWork at officeGreta Ritchie - State Farm Agent
Omaha, NE2 days ago$35k - $60k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...For contractorsWork at officeFlexible hoursCandace Specht - State Farm Agent
Omaha, NE2 days ago$40k - $60k
...Position Overview Successful State Farm Agent is seeking a qualified professional to join their winning team for the role of Office... .... Responsibilities Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Hourly payFor contractorsWork at officeState Farm
Omaha, NE3 days ago$40k - $60k
...Office Representative - State Farm Agent Team Member Full Time in Omaha, NE Position Overview Successful State Farm Agent is seeking... ...potential. Responsibilities Use a customer‑focused, needs‑based review process to educate customers about insurance options. Develop leads...Hourly payFull timeFor contractorsWork at officeGretaritchie
Omaha, NE1 day ago$18 - $21 per hour
...interested in working part-time in a busy office? If you answered yes to these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State Farm insurance and financial service products....Hourly payPart timeFor contractorsWork at officeFlexible hoursState Farm
Omaha, NE4 days ago- ...about their development. If you want to work in an environment that is fun, challenging, and rewarding, then Tracy Hough - State Farm Agent may be the right fit for you! If you're willing to work hard and you expect to get results from yourself and those around you, we...For contractorsWork at officeAfternoon shift
Tracy Hough - State Farm Agent
Council Bluffs, IA5 days ago $40k - $60k
...development ABOUT OUR AGENCY: Our agency opened in 2019 and has grown into a small but close-knit team of three. Before becoming an agent, I spent eight years in appliance sales before a recruiter from State Farm reached out and set me on this path. I graduated from the...For contractorsWork at officeLocal areaImmediate startFlexible hoursState Farm
Omaha, NE4 days ago$35k - $60k
Future Opening: Office Representative - State Farm Agent Team Member Full Time • Omaha - NE Replies within 24 hours Position Overview... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Full timeFor contractorsWork at officeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
