AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$30 per hour
...autonomous database and industry’s broadest and deepest suite of AI-powered cloud applications. The following facts and figures highlight... ...Responsibilities We are seeking a highly motivated AI Agent Intern to join Oracle’s Supply Chain Applications (SCM) Solution...SuggestedHourly payTemporary workInternshipFlexible hours- ...Palmer Group is partnering with a growing Accounting Firm looking for a candidate interested in a Tax Reviewer position. This position will review simple to complex tax returns and projects. We are looking for an experienced individual with the skill set to provide...Suggested
- ...each other. CLA is seeking to hire an experienced Seasonal Tax Reviewer to join our talented tax team. The position offers flexibility... ...or related field is preferred. CPA is required (JD or Enrolled Agent is acceptable in lieu). #LI-NM1 Equal Opportunity Employer/Protected...SuggestedCasual workSeasonal work
- ...CLA is seeking an experienced Seasonal Tax Reviewer in Des Moines, IA to ensure the accuracy and compliance of tax returns for individuals, businesses, partnerships, or trusts. The role serves as a technical resource for complex tax questions and interfaces with client...SuggestedSeasonal work
$106.9k - $200.6k
...to build a better working world. The opportunity As a Agent Dev Architect, you’ll have the chance to shape innovative solutions... ...engineering teams, leverage cutting-edge technologies—including AI and cloud platforms—and help clients transform their operations....SuggestedFull timeSummer holidayFlexible hours$70k - $85k
...Job Description Job Description Tax Reviewer Location: West Des Moines, IA (In Person) Salary: $70,000–$85,000 per year Job Type: Full-Time Position Overview We are seeking an experienced, highly detail-oriented Tax Reviewer to join our growing...Full timeSummer workWork at officeMonday to FridayFlexible hours$144k - $374k
...Architect to lead the design and implementation of high autonomy Agentic AI systems in the enterprise. The scope of this role if across... ...candidate will lead and mentor a team building advance multi-agent capabilities for EY businesses and customers. Your key responsibilities...Full timeSummer holidayFlexible hours- ...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...Immediate startOverseas
$97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...Work at officeLocal area- ...Special Agent Position Joining the United States Secret Service (USSS) as a Special Agent offers a unique and rewarding career dedicated to serving the nation through our dual mission of criminal investigations and protection. Special Agents play a critical role in...
- ...Wheelchair Agent People. Passion. Pride. This is what has driven our teams since 1833. Since that time, we have developed to become a critical partner in the global aviation industry, delivering time-critical logistics services at over 300 locations in 65 countries...Flexible hoursNight shift
- ...The Federal Bureau of Investigation (FBI) seeks qualified individuals to become special agents. This position directly supports national security by investigating cyber threats, terrorism, fraud, and other crimes, using your technical and analytical background. Candidates...Work at office
$12 per hour
...Wheelchair Agent People. Passion. Pride. This is what has driven our teams since 1833. Since that time, we have developed to become... ...to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor....Hourly payFlexible hoursShift workNight shift$16.61 per hour
Job Title Join Envoy Air where you will be stimulated, challenged, and have the opportunity to develop your career. We offer: Amazing employee flight privileges within the American Airlines global network Training and development programs to take your career...Full timeImmediate startFlexible hoursNight shiftRotating shift- Agent Opportunity Learn more about the agent opportunity and hear testimonials from our agents. FBL Financial Group, Inc.Work at office
- ...Job Description Job Description Our client seeks a Tax Accountant/Reviewer to work in their office in West Des Moines, Iowa. This is a full-time, direct hire opportunity where you will earn a salary based on your experience. You will enjoy a competitive...Full timeWork at officeLocal areaImmediate startMonday to Friday
- ...in West Des Moines looking for a candidate who is interested in a Tax Manager position for one of our partners. This position will review simple to complex tax returns and projects. We are looking for an experienced Tax Manager that has the skillset to provide coaching...
$40k - $60k
...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop... ...needs, and market appropriate products and services. As an Agent Team Member, you will receive... ~401K ~ Salary plus...For contractorsWork at officeFlexible hours$21.4 per hour
Entry-Level Loan Servicing/Document SpecialistNo prior experience needed. Location: Des Moines, IA – Onsite (6:00 AM – 2:30 PM CST) / (alternate start times may be available at 6:30 AM or 7:00 AM). Job type: Contract. Pay rate: $21.40/hr.Grow your financial career! We ...Contract workWork at office- ...Summary Elevate your career by becoming a Buyers Agent. We provide extensive real estate knowledge, in-depth market insights, personalized mentorship, and advanced resources such as Brivity Technology and Brivity Marketer to fuel your success. Get ready to attract high...
- Document Fulfillment Specialist Ampcus Inc. is a certified global provider of a broad range of Technology and Business consulting services. We are in search of a highly motivated candidate to join our talented team. Location(s): Urbandale, IA Job Purpose Under...Shift workAfternoon shift
- ...greater confidence in the insights they rely on. For HDR, it means enterprise data that is ready to support analytics, automation, and AI in a way that reflects how we work.HDR is seeking a strategic-minded Resources Business Group Data Steward to shape how our data is...
$15 - $22 per hour
...Receptionist - State Farm Agent Team Member Corey Chancellor - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent...Hourly payFor contractorsWork at office$35k - $50k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...up with customers, as needed.Use a customer-focused, needs-based review process to educate customers about insurance options.Develop leads...For contractorsFlexible hours- ...personal achievement. If you want to work in an environment that is fun, challenging, and rewarding, then Terry Taylor - State Farm Agent may be the right fit for you! If you're willing to work hard and you expect to get results from yourself and those around you, we want...For contractorsWork at office
$35k - $50k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Hourly payFor contractorsFlexible hours$35.36k - $49.92k
...based on performance Competitive salary Opportunity for advancement ROLE DESCRIPTION: Glenn Waterhouse - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Full timeWork at office$32 - $42 per hour
...single patient’s request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of... ...facility. Experience with significant level of coding quality review feedback Experience in computerized encoding and abstracting...Hourly payFull timeRemote workRelocation packageFlexible hours$6,240 per month
...you have aspirations to run your own business? If so, you may want to consider working in the office of Glenn Waterhouse - State Farm Agent. As a member of our agency team, you have an opportunity to experience first-hand what it takes to be a State Farm Agent. You will...For contractorsInternshipSummer internshipWork at office- ...you have aspirations to run your own business? If so, you may want to consider working in the office of Glenn Waterhouse - State Farm Agent. As a member of our agency team, you have an opportunity to experience first-hand what it takes to be a State Farm Agent. You will...For contractorsWork at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
- state farm agent Des Moines, IA
- tsa agent Des Moines, IA
- import export agent Des Moines, IA
- commissioning agent Des Moines, IA
- remote chat agent Des Moines, IA
- agent Des Moines, IA
- freight agent no experience Des Moines, IA
- executive protection agent Des Moines, IA
- work from home chat agent Des Moines, IA
- cruise agent Des Moines, IA


