AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$89.2k - $209.5k
Job Description Oracle Health is seeking a Senior AI Agent Engineer to build production AI agents and workflow automation capabilities that accelerate analytics delivery, improve insight generation, and reduce manual engineering work across our analytics and reporting...SuggestedTemporary workFlexible hours- Flexible part time jobs now available in your area! Perfect for stay at home Moms and Dads! Looking for an independent part time job while your children are in school? As your children go back to school this fall, this is a perfect opportunity to earn some extra income...SuggestedExtra incomePart timeFlexible hoursShift work
RetailData
Concord, NH4 days ago - ...Border Patrol Agent (BPA) – in the Federal Security and Public Safety Sector (Entry Level) A MISSION WORTHY OF A CAREER! But, if you'... ...official employment site, to complete your application. Be certain to review ALL details of the job opportunity announcement and follow all...SuggestedFull timeWork experience placementImmediate startRelocationNight shift
US Customs and Border Protection
Concord, NH2 days ago - ...keep up with their dental health, affordability, transparency, and access. Work from home as a Spanish Bilingual Scheduling Center Agent where you will partner with our patients to schedule their appointment(s), providing them critical access to care and improved dental...SuggestedHourly payFull timePart timeImmediate startWork from homeShift workWeekend workAfternoon shift
- ...Recruiter account is now being activly monitored and your resumes will be reviewed. Granite Security Service has changed this job posting and is looking to hire Armed/Unarmed per diem security agents to help meet our security needs. Please keep in mind before applying...SuggestedDaily paidOverseasNight shiftWeekend work
Granite Security Service LLC
Concord, NH3 days ago - ...development Flexible schedule Free food & snacks You May Be a Great Fit as a Multiline Representative at Aron Compton - State Farm Agent if: You’re the person people call first when life goes sideways - calm, steady, and ready to help. You genuinely care about your...Work at officeFlexible hours
$25 - $26.7 per hour
...single patient's request for their medical records to powering the AI revolution in healthcare, Datavanters are building the future of... .... As an HCC (Hierarchical Condition Category) coder you will review medical records to identify and code diagnoses using a standardized...Hourly payRemote workFlexible hoursDatavant
Concord, NH1 day ago- Are you tired of being unsure how much your delivery/driver job will pay? Will the customer tip? We have a great side hustle job for you! Our jobs are preplanned with a flexible schedule, and the faster you get at the job the quicker you are in and out! Flexible when you...Hourly payExtra incomeTemporary workPart timeSecond jobFlexible hoursShift work
RetailData
Hooksett, NH1 day ago $3,000 per month
...No prior experience is needed and the hours are flexible—perfect for those looking for a part-time job they can do remotely. Roles include data entry clerk, customer service agent, nurse or medical assistant - just choose what suits your skills best and start earning!...Full timeTemporary workPart timeImmediate startRemote workWork from homeFlexible hoursFocusGroupPanel
Concord, NH5 days ago- Data Entry Operator | Junior (Remote) Concord, NH, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...Remote job
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- ...Job Description Job Description We are seeking a dedicated and customer-focused Agent Team Member to join our State Farm office. In this role, you will support the agent by providing exceptional customer service, managing daily office operations, and assisting clients...Full timeWork at officeFlexible hours
- ...documentation in the medical record. The CDS facilitates clarification of clinical documentation through extensive medical record review, interaction with physicians, nursing staff and other caregivers utilizing coding and clinical documentation expertise. The CDS will...Permanent employmentFull timeMonday to FridayShift work
Direct Staffing Inc
Concord, NH2 days ago $21 - $23 per hour
...with internal stakeholders, risk groups and outside vendors. Performs group enrollment audits and makes eligibility adjustments. Reviews enrollment data for accuracy by comparing and analyzing group reports and performing audits. Collaborates with Third Party...Full timeContract workTemporary workWork at officeMonday to Friday2 days per weekNortheast Delta Dental
Concord, NH6 days ago- Concord Hospital Health System is seeking an individual to manage and create patient accounts for services rendered, working collaboratively with patients, families, and the care team. Responsibilities include verifying patient demographic information, supporting financial...
- The Office Administrator (OA) is responsible for providing general office support with a variety of clerical activities and related tasks. Work as liaison between the local Region and RedBull Distribution Company’s (RBDC’s) Finance and Human Resources Service Center. OAs...Permanent employmentWork at officeLocal area
$1,800 - $3,200 per week
Entry-Level Freight DispatcherWe are currently hiring motivated and detail-oriented individuals to join our team as Independent Freight Dispatchers.What "Entry-Level" Means: If you already have basic freight dispatching experience, we consider that a strong advantage and...Weekly payCurrently hiringFlexible hoursAmerican Logistics Authority
Concord, NH5 days ago$1,500 - $3,000 per week
Freight Dispatchers Truck Driver Nation is growing, and we're looking for freight dispatchers ready to take control of their financial future. Experienced dispatchers put your skills to work with a proven system. Entry-level candidates training available for motivated...Immediate startAmerican Logistics Authority
Concord, NH2 days ago$6,000 per month
Freight Dispatchers WantedTruck Driver Nation is expanding and we are hiring both experienced and entry-level freight dispatchers who are ready to take charge of their financial future.This is not an hourly job. As an independent freight dispatcher, you earn 8%–10% of ...Hourly payImmediate startAmerican Logistics Authority
Concord, NH1 day ago$15 - $16 per hour
Mail Room Clerk Concord, NH, United States $ 15.00 - 16.00 (US Dollar) Mail Room Clerk needs to be reliable, punctual Mail Room Clerk requires: High school diploma or equivalent Proficient in relevant computer applications such as MS Office Knowledge...Global Channel Management
Concord, NH5 days ago$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFlexible hoursPostal Jobs Resource
Concord, NHmore than 2 months ago- POSTAL MAIL PROCESSOR NO EXPERIENCE REQUIRED - PAID TRAINING PROVIDED - JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience...
US Postal Service
Pembroke, NH3 days ago $23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
Pembroke, NHmore than 2 months ago- Company description Terry Soot Management Group (TSMG) is a field data collection company founded in 2017 in Europe. We collect data where automation is not possible. We count features, take pictures, make videos, record speech, and scan areas for every detail you need ...Monday to FridayFlexible hours
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
- A logistics company is seeking a Freight Dispatcher (Independent Contractor) to work remotely in the U.S. This performance-based role involves booking loads, negotiating rates, and managing workflows independently. Ideal candidates are organized and self-motivated, with...Remote jobFor contractors
- Hours Every Monday and Friday 11 AM - 9:30 PM, 20 hours per week Summary Under the direction of the Supervisor Laboratory Non-Technical, the Clinical Lab Assistant performs phlebotomy using various venipuncture techniques to collect blood directly from patients...Part timeMonday to FridayAfternoon shift
Concord Hospital
Concord, NH3 days ago $1,800 - $4,500 per week
Freight Dispatcher (Independent Contractor) — $1,800 to $4,500+ Weekly Potential Employment Type: 1099 / Independent Contractor Location: Remote (U.S. Based Only) This is a performance-based dispatch role working with owner-operators under their own authority. Earnings ...For contractorsRemote work$8,000 per month
Freight Dispatchers Wanted (Experienced & Entry-Level) - Work From Home & Earn $8,000 to $15,000+ Per Month Truck Driver Nation is hiring motivated individuals ready to build a high-paying career from the comfort of home. Whether you’re an experienced dispatcher or brand...Hourly payWork from home
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
