AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...Paid time off Training & development About our Agency I opened my agency in 2025 and currently lead a team of four. Before becoming an agent, I spent six years working as a State Farm team member, where I developed a passion for helping customers and building a strong team...SuggestedWork at officeLocal area
- ...time off Training & development ABOUT OUR AGENCY: I opened my agency in 2025 and currently lead a team of four. Before becoming an agent, I spent six years working as a State Farm team member, where I developed a passion for helping customers and building a strong team...SuggestedWork at officeLocal areaFlexible hours
- ...Opportunity for advancement Training & development You May Be a Great Fit as an Acquisitions Specialist at Ryan Powell - State Farm Agent if: You’re the person people call first when life goes sideways — calm, steady, and ready to help. You genuinely care about your community...SuggestedWork at officeFlexible hours
$10k
...Border Patrol Agent (BPA) GL-5/7 grade levels - New Hire Sign-On and Retention Incentives A MISSION WORTHY OF A CAREER! If you... ...official employment site, to complete your application. Be certain to review ALL details of the job opportunity announcement and follow all...SuggestedFull timeWork experience placementImmediate startRelocationNight shiftCustoms and Border Protection
Kyle, TX3 days ago$40k - $60k
...Health insurance ~Opportunity for advancement ~Paid time off ~Training & development ROLE DESCRIPTION: Cory Pack - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...SuggestedFull timeFor contractorsWork at officeFlexible hoursCory Pack - State Farm Agent
Kyle, TXmore than 2 months ago$40k - $60k
Receptionist - State Farm Agent Team Member job at Cory Pack - State Farm Agent. Kyle, TX. Responsive recruiter Position Overview Are... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...Hourly payFor contractorsFlexible hours$36k - $47k
Data Entry ClerkSharpcontra is a fast-growing company specializing in cutting-edge solutions for various industries, helping clients achieve efficiency and success through innovative approaches. Our team is committed to providing top-tier service and technology-driven solutions...Work at office$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
San Marcos, TXmore than 2 months ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
- Epic Piping has an exciting opportunity available for a Clerk to be an integral part of our Pipe Support team. Epic takes pride in our quality of work and has a goal of building our workforce with the most talented employees. You’ll have an opportunity to work on a wide...Work at office
EPIC Piping
San Marcos, TX3 days ago - Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFull timeFlexible hours
- Teller – San MarcosLocation: 904 Bugg Ln. San Marcos, TX, 78666 Schedule: Monday – Friday 8:00am – 5:45pm and rotating Saturdays: 8:45am – 12:15pm Starting Pay: $41,600 per year FLSA Status: Non-exemptAt Austin Telco Federal Credit Union, we're dedicated to making a positive...Work at officeMonday to Friday
Austin Telco FCU
San Marcos, TX1 day ago - ...Composes and prepares routine correspondence for signature. Takes and transcribes dictation. Independently prepares recurring reports for review and signature. Sends and retrieves messages via email. Faxes and retrieves incoming messages on a regular basis. Maintains a safe...Work at officeLocal area
- Austin Telco Federal Credit Union in San Marcos, TX seeks a Teller to deliver friendly, accurate service at the branch. You will handle member transactions, answer questions, and support daily cash operations. The role requires a high school diploma and at least 6 months...
Austin Telco FCU
San Marcos, TX11 hours ago $40.06k - $41.27k
Government Jobs - Full-Time Regular - $40,063.00 - $41,265.00 Annually Essential functions, as defined under the Americans with Disabilities Act, include the responsibilities, knowledge, skills, and other characteristics listed below. This list of responsibilities is ILLUSTRATIVE...Full time- ...insurance ~ Opportunity for advancement ~ Paid time off ~ Training & development ROLE DESCRIPTION: Cory Pack - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic...Work at officeFlexible hours
$70k
...maintaining a high level of confidentiality regarding client and company matters. Key Responsibilities: Create, review, and edit proformas in response to attorney and administrative requests. Execute complex bills in a timely manner, including multiple...Permanent employmentFull timeWork at officeLocal areaAddison Group
San Marcos, TX19 days ago- Merry Maids Team MemberProvides general facilities administration support which may include data entry, system updates, and weekly payroll submission. Confirms customer appointments. Creates and updates schedules with new customer information and provide team members with...Work at office
Merry Maids
San Marcos, TX10 hours ago - First Lockhart Bank is seeking an Universal Banker at the San Marcos branch to deliver exceptional customer service while handling teller and new accounts responsibilities. The role involves routine financial transactions and promoting bank products. As a key customer-facing...
- Ace Relocation Systems, Inc. in San Marcos, TX seeks an OA/DA Coordinator to coordinate schedules for customers and vendors, ensuring a seamless relocation experience. You will manage OA/DA and Military move bookings and document customer interactions. The role requires...Full timeWork at officeRelocationDay shift
- Ace Relocation Systems in San Marcos, TX is seeking an OA/DA Coordinator to schedule and survey customer moves, coordinating with internal teams and military shipments. You will maintain records in MoverSuite and communicate with customers by phone and email to ensure a...Relocation
$42.29 per hour
Summary Responsible for the safe, orderly, and expeditious flow of air traffic control duties in a Visual Flight Rules (VFR) Air Traffic Control Tower. Dimensions of Position Works under the general supervision of the Tower Manager who relies on the incumbent ...Hourly payWork at officeImmediate startRobinson Aviation
San Marcos, TX2 days ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...Hourly payFull timeWork at officeNight shiftUSPS
San Marcos, TX4 days ago- Berry Aviation seeks an aviation maintenance technician with dual role as flight attendant/loadmaster for a government contract. You will perform maintenance on fixed-wing turboprops under FAA Part 135, and provide safe transport of passengers and cargo in austere environments...Contract work
- Vitalogy Skincare in San Marcos, TX is seeking a Medical Assistant to support licensed providers across clinic and surgical settings. This role involves preparing patients, measuring vital signs, assisting with procedures, and maintaining accurate medical records under ...
$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekend workPostal Jobs Resource
San Marcos, TXmore than 2 months ago- ...members, in person, by phone, or by mail. Answers questions, resolves problems, and explains membership, warehouse, and credit programs. Reviews membership, warehouse, and credit programs to keep knowledge up to date. Physically checks item availability in the warehouse,...Temporary work
- PEIMS Clerk - High School Reports to: Principal Pay Grade: PP02, 8h, 210 days Dept./School: High School Assigned Date Revised: February 2022 Primary Purpose: Responsible for maintaining student academic records at the campus level under minimal supervision. Perform...Work at office
- ABOUT Act as our eyes and ears in the sky, mapping coasts and ports for Navy intelligence. Gain extensive training in radar, sonar, and air traffic communications. If you feel more at home above the sea than on it, a career in aviation might be for you. Below are some...Full timePart time
U.S. Navy
San Marcos, TX10 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
