AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$17 - $20 per hour
Hotel Cerro in San Luis Obispo, CA 93401 is hiring for an Entry Level Housekeeping role with flexible Full-Time/Part-Time hours and $17.00-$20.00 hourly pay. This position requires thorough daily inspections of guest rooms, public areas, and back-of-house spaces to ensure...SuggestedHourly payFull timePart timeFlexible hoursHotel Cerro
San Luis Obispo, CA3 days ago- Azul Hospitality is seeking an Entry-level Housekeeping Supervisor at Hotel Cerro in San Luis Obispo, CA. The role ensures top cleanliness standards across guest rooms and public areas and collaborates with Room Attendants to maintain brand quality. Responsibilities include...Suggested
Azul Hospitality
San Luis Obispo, CA3 days ago $97.3k - $125.54k
JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau'...SuggestedWork at officeLocal area$20k - $40k
...Future Opening: Office Representative - State Farm Agent Team Member Part Time • San Luis Obispo - CA Position Overview Are you outgoing... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...SuggestedPart timeFor contractorsWork at officeFlexible hoursCatherine Riedstra
San Luis Obispo, CA2 days ago$100k
...Join the Keller Williams Realty Team! New and experienced agents alike are encouraged to apply to join the KW family. Applicants without a real estate license are invited to apply . You need a license to become a real estate agent before being considered for the...SuggestedHourly payFull timeFor contractors- A data services company is seeking a Remote Data Entry Operator to manage large volumes of data while ensuring accuracy and completeness. The candidate will be involved in data verification, quality checks, and collaborations. This entry-level position emphasizes attention...Remote job
- A leading firm in data management seeks a focused Data Entry Clerk to update and maintain company databases. The ideal candidate will be diligent and energetic, with excellent communication skills to collaborate effectively with employees and customers. Responsibilities...
- About the job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying , Check Your Inbox or Spam Folder For next steps. A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures...Remote job
- Data Entry Operator | Junior (Remote) San Luis Obispo, CA, United States Job Openings Data Entry Operator | Junior (Remote) About the Job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying, Check Your Inbox...Remote job
- We are looking for a focused data entry clerk to continuously update our company's databases. The data entry clerk will liaise with and follow up with employees within the company as well as with customers to collect information. The data entry clerk will capture the data...
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
San Luis Obispo, CA4 days ago $26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
San Luis Obispo, CAa month ago- ...organizations including Tesla, American Medical Response, Mercedes Benz, and Mustang Cat. Similar Career Fields Include: Cargo and Freight Agent, Cargo Inspector, Vehicle Operator. About Our Organization: The U.S. Army is wealth of possibilities for your future whether you...Contract workPart timeImmediate startRelocation packageFlexible hours
U.S. Army Reserve
San Luis Obispo, CA10 hours ago - ...receive a confirmation email. If you do not receive a confirmation email, check your spam or junk folder. Please ensure that all documents are attached, as you cannot proceed in the recruitment without the required attachments. We look forward to reviewing your application!...Full timeWork at officeMonday to Friday
San Luis Coastal School District
San Luis Obispo, CA5 days ago $16.9 - $20 per hour
Clerk SLO Headquarters - San Luis Obispo, CA 93401 Overview Salary Range $16.90 - $20.00 Hourly Position Type Full Time Job Shift Day Travel Percentage Negligible Category Admin - Clerical Description JOB SUMMARY: Provide operational, customer service and...Hourly payPermanent employmentFull timeTemporary workWork at officeShift work- ...person, by phone, or by mail. Answers questions, resolves problems, and explains membership, warehouse, and credit programs. Reviews membership, warehouse, and credit programs to keep knowledge up to date. Physically checks item availability in the warehouse, directs...Temporary work
$25 - $28.59 per hour
Job Summary And Responsibilities As our Medical Records Professional, you will ensure the integrity, confidentiality, and accessibility of patient health information, meticulously managing records in compliance with all regulatory standards and organizational policies....Dignity Health
San Luis Obispo, CA1 day ago$16.9 - $21.5 per hour
...ordinances. Applicant Privacy Notice To learn more about how we collect, use, and retain applicant personal information, please review our Applicant Privacy Notice at: Employment-At-Will Employment with PAQ, Inc. is at will. Either the employee or the...Hourly payFull timeLocal areaImmediate startShift workWeekend workAfternoon shiftFood 4 Less
San Luis Obispo, CA2 days ago- Reservation Office Clerk The Madonna Inn is seeking a candidate for its front desk as a Front Desk Clerk. This is a full time position, 5 shifts/40 hours per week. Duties include handling guest check-ins and check-outs, collecting payments, answering multi-line phones...Full timeWork at officeShift workNight shiftWeekend work
Madonna Inn
San Luis Obispo, CA3 days ago $23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFlexible hoursPostal Jobs Resource
San Luis Obispo, CAmore than 2 months ago$42.51 per hour
Summary Responsible for the safe, orderly, and expeditious flow of air traffic control duties in a Visual Flight Rules (VFR) Air Traffic Control Tower. Dimensions of Position Works under the general supervision of the Tower Manager who relies on the incumbent ...Hourly payWork at officeImmediate startRobinson Aviation
San Luis Obispo, CA4 days ago$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
San Luis Obispo, CAmore than 2 months ago- ...Straatsma, LLP (GMS), a professional workers' compensation firm, is seeking a Mail Clerk to join our San Luis Obispo office. Please review the key responsibilities below and reach out if you feel this position would be a good fit for you. Key Responsibilities:...Work at office
- HealthCare Support Staffing, Inc. (HSS), is a proven industry-leading national healthcare recruiting and staffing firm. HSS has a proven history of placing talented healthcare professionals in clinical and non-clinical positions with some of the largest and most prestigious...Private practiceWork at officeLocal areaMonday to Friday
- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Arroyo Grande, CA4 days ago $23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekend workPostal Jobs Resource
San Luis Obispo, CAmore than 2 months ago$18 - $21 per hour
Teller PositionAs the first point of contact for most clients, Tellers play a key role in delivering on the bank's Breakthrough Client Service Standards. With guidance from the Branch Manager and Assistant Branch Manager, the Teller role's primary responsibility is to ...Night shiftColumbia Bank (WA, OR & ID)
Arroyo Grande, CA5 days ago- ABOUT Act as our eyes and ears in the sky, mapping coasts and ports for Navy intelligence. Gain extensive training in radar, sonar, and air traffic communications. If you feel more at home above the sea than on it, a career in aviation might be for you. Below are some...Full timePart time
U.S. Navy
San Luis Obispo, CA2 days ago $21.25 - $23.75 per hour
Why do we say working at SESLOC is the #BestJobEver? Working at SESLOC-a not-for-profit organization-means you're educating, empowering and enriching the lives of our members AND supporting our local community. Whether lending to your neighbor next door, promoting an informative...Hourly payLocal areaSESLOC Credit Union
Arroyo Grande, CA2 days ago$25 - $28.63 per hour
Job Summary and Responsibilities As our Clerk Receptionist, you will be the welcoming face and organizational hub of our clinic, ensuring a seamless and positive experience for every patient. Every day, you will expertly manage patient registration and scheduling...Work experience placementDignity Health
Arroyo Grande, CA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
