AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$20 - $30 per hour
...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent... ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every... ...at yet and judge it step by step. • Review — Take an existing annotation, written...SuggestedHourly payFull timeContract workImmediate startRemote workBespoke Labs
Chandler, AZ2 hours ago- ...headquarters. You will be the voice of Carvana, guiding customers through their car-buying or selling journeys with empathy, backed by an AI co-pilot that handles routine notes and case summaries. The role offers a clear path for growth within Carvana’s Leadership, Strategy...Suggested
$18 - $20.7 per hour
...scanning, uploading, and digitizing files. Perform accurate alpha-numeric data entry with strong attention to detail. Receive, review, and organize application materials and relevant documents to ensure completeness and correctness. Sort, file, and distribute...SuggestedHourly payWeekly payTemporary workLocal areaFlexible hoursShift workAdecco
Tempe, AZ4 days ago- Position Overview FUJIFILM Electronic Materials, U.S.A., Inc., is a global leader in chemical solutions which enable the semiconductor industry and the digital universe. We have an exciting opportunity at our Mesa, Arizona facility for a Document Control and Training...SuggestedFull timeWork at officeLocal area
FUJIFILM Biotechnologies
Mesa, AZ12 days ago - Description This position involves bidding, engaging and reviewing single family residential appraisals covering all states. The qualified candidate will become familiar, and then competent, with RIMS which is an internet based appraisal management system. The position...SuggestedWork at office
MidFirst Bank
Chandler, AZ4 days ago $26 - $29 per hour
JLM Strategic Talent Partners Job Post We partner with National & International prime contractors to provide them with qualified talent they can trust. We accomplish this by sourcing & vetting high level career seeking candidates in the industry and match them with ...Hourly payFor contractorsJLM Strategic Talent Partners
Apache Junction, AZ1 day ago$34.57 - $46.11 per hour
...every stage of your career. Try new things, learn new skills and discover what you excel at—all from Day One. Job Description Reviews appraisals and reports including desktop valuations from a wide variety of bank channels. Reviewer Appraiser positions review all types...Temporary workWork at officeLocal area3 days per weekU.S. Bank
Tempe, AZ2 days ago$50k - $65k
...construction-related documents. Additionally, you will process documents for incoming and outgoing transmittals. You will need to review technical reports, specifications, drawings, and other documents. You will need to identify and use the correct tools, systems, and...For contractorsMonday to FridayFlexible hoursAccess Sciences
Chandler, AZ2 days ago- ...students and families we serve. A Typical Day Looks Like While no two days are exactly the same, you can expect to: Review, process, and maintain student records and enrollment documentation in accordance with applicable requirements. Enter and update...Full timeWork at officeRemote workVisa sponsorshipFlexible hours
StrongMind
Chandler, AZ2 days ago - ...subscription to three fundamental tenets: Partnership, Integrity, and Innovation. Purpose Aldridge Pite, LLP is seeking a Title Reviewer. The Title Reviewer is responsible for reviewing the chain of title and ensuring the firm is clear to proceed with foreclosure....Full timeLocal areaRemote workWork from homeFlexible hours
Aldridge Pite LLP
Mesa, AZ25 days ago - ...securitized titles (repossessions, moves, adding or deleting names from title, name changes, and flips) including, but not limited to, reviewing incoming paperwork, prepping, and sending out all necessary paperwork to third-party title vendor, the DMV and/or customer...Full timeTemporary workWork at office
$1,000 per month
...Document Verification Specialist The Document Verification Specialist is responsible for review, validation, and execute legal documents related to the foreclosure process. The Document Verification Specialist must be well organized and detail-oriented with a specific...Casual workWork at officeWeekend workAfternoon shift- U.S. Bank’s Government Leasing & Finance Group seeks a detail‑oriented Leasing Specialist 5 to support lease and loan transactions, ensuring accurate documentation, lien perfection, and timely funding. Responsibilities include coordinating with sales, credit, legal, and...
- MSC USA in Tempe is seeking a Customer Service Specialist for Export Documentation to handle export documentation, communicate with customers, and coordinate with data-entry teams. The role requires on-site presence five days a week in Tempe. Ideal candidates have a bachelor...
- ...to remarketing, ensuring our customer partners the confidence of unparalleled professionalism in lender solutions. Repossession Agent/Tow Truck Driver Skills: Towing, Repossession Paramount Recovery Service is one of the nation's top repossession companies and...Hourly payFull timeNight shiftWeekend workDay shift
Paramount Recovery Services LLC
Mesa, AZ3 days ago - Duties and ResponsibilitiesTerminal Operations Agents support safe, secure, and smooth day-to-day operations of our Commercial Passenger Terminal and landside airport property by working with passengers, employees, tenants, contractors and the public.For contractors
Mesa Gateway Airport
Mesa, AZ4 days ago $750 per week
ApexFocusGroup collaborates with research firms, universities, and trusted brands to gather genuine consumer insights. We help connect everyday people with flexible remote market research opportunities that influence real products, services, and customer experiences. Role...Part timeCasual workRemote workFlexible hoursAFG
Apache Junction, AZ4 days ago- Bechtel Corporation seeks a Commissioning System/CX Software Admin in Chandler, AZ, to set up, track, walkdowns, and audit project data in Smart Completion and Procore systems. You will support project teams, coordinating with commissioning, site automation, and engineering...
Bechtel Corporation
Chandler, AZ4 days ago - ...countries, and on 5 continents. Are you ready to take off on your next career with us? Job Summary The Aircraft Appearance Agent is responsible for ensuring the interior of aircraft is clean, well-stocked, and flight-ready in accordance with airline and safety...Full timeContract workPart timeWorldwideRotating shift
Worldwide Flight Services
Mesa, AZ2 days ago - ...MaintenanceOpening Date: 07/23/2026Closing Date: ContinuousTerminal Operations Agents support safe, secure, and smooth day-to-day operations of our... ...passengers, employees, tenants, contractors and the public. Review job details at How many years of training, education, or work...For contractorsWork experience placement
GovernmentJobs.com
Mesa, AZ2 days ago - ...legacy. Proven to be worthy of trust and confidence. Be tomorrow's Secret Service. During the course of their careers, special agents carry out assignments in both investigations and protection and may be assigned to multiple duty stations throughout the U.S. and abroad...Overseas
The United States Secret Service
Mesa, AZ1 day ago - ...Job Description As a Geek Squad Agent, you'll be the first point of contact for people seeking technology support, knowledge and solutions for computers and other devices at our local Best Buy store. We'll provide the training and resources you need to assess customers...Hourly payFull timePart timeLocal areaWork from homeFlexible hoursNight shift
Best Buy
Tempe, AZ3 days ago $24 per hour
...Armed Security Agent PLEASE NOTE, THIS IS A SAFETY SENSITIVE ROLE: This position has been classified as safety-sensitive. As such, the duties of this role directly impact the safety of others. Individuals hired into this position will be subject to drug testing, including...Part timeRelocationMonday to FridayShift workNight shiftWeekend workDay shiftFortified Risk Group
Tempe, AZ1 day ago$1,000 per month
...certifications. Address inquiries from vendors, investors, Portfolio Managers, etc. professionally and in a timely fashion. Review daily task queue and exception reports to ensure completion of assigned duties within specific SLAs. Update and annotate all actions...- ...records systems as student data is received for new enrollments in accordance with established enrollment procedures. Assist with IEP review processes to ensure IEPs are reviewed by ESS Leadership and required accommodations are communicated to Registrars in a timely...Full timeWork at office
- ...supporting exhibits used in collections and recovery actions. You will redact sensitive information, ensure accuracy, and perform quality reviews with cross-functional partners to keep packages moving through required checkpoints and deadlines. This role requires meticulous...
$17 per hour
...Commissary Agent Date: Sep 1, 2026 Location: Mesa, Arizona, US Company: Worldwide Flight Services, Inc. Compensation: $17/hr Join our Worldwide Flight Services and Oxford family and contribute to the timely delivery of cargo shipments, luggage, business to customer...Full timePart timeWorldwideFlexible hoursNight shift- ...Domestic Agent I The Domestic Agent I has high visibility, is the primary contact for designated customers, and responsible for the outcome of assigned projects including customer service, tracking, routing, problem resolution, vendor compliance and customer interaction...Work at officeLocal areaWorldwideMonday to FridayFlexible hoursShift work
$17 per hour
...insurance, compensated time off, paid holidays, and 401 (K) with match: no phone calls, no agencies, EOE, drug-free workplace. Please review our Applicant Privacy Notice: -applicant-privacy-notice/ Each posted position will be active for at least five (5) business...Hourly payFull timeWork at officeFlexible hoursZwicker & Associates, P.C.
Tempe, AZ12 days ago$10k
...to succeed in this opportunity Find out below. Border Patrol Agent (BPA) - Experienced (GL-9 GS-11) - New Hire Sign-On and Retention... ...employment site, to complete your application. Be certain to review ALL details of the job opportunity announcement and follow all instructions...Full timeLocal areaImmediate startRelocationNight shiftCustoms and Border Protection
Chandler, AZ2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
