AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...SuggestedWork at officeLocal areaFederal Bureau of Investigation (FBI)
Huntsville, AL5 days ago$51k
...Job Description Border Patrol Agent U.S. Customs and Border Protection (CBP) Multiple Duty Locations – Northern and Southern U.S. Borders U.S. Customs and Border Protection (CBP) is seeking Border Patrol Agents to help secure the nation's borders, prevent the illegal entry...SuggestedWork experience placementLocal areaRemote workRelocationLeoforce
Huntsville, AL1 day ago$10k
...U.S. Customs and Border Protection (CBP) is seeking Border Patrol Agents to secure the nation’s borders, prevent illegal entry, and enforce federal laws along the Northern and Southern borders. Training includes six months at the Border Patrol Academy in Artesia, NM,...SuggestedLeoforce
Huntsville, AL1 day ago- ...U.S. Customs and Border Protection (CBP) is seeking Border Patrol Agents to secure the nation's borders and enforce federal laws. Duties include patrolling, apprehending individuals, and coordinating with agencies across remote and challenging environments. Ideal candidates...SuggestedWork experience placementRemote workRelocation
Leoforce
Huntsville, AL1 day ago $25k
...or seasonal employees. Please consult with the Human Resources Department for a full list of benefits. Madison County periodically reviews and when fiscally able, adjusts salary rates and our benefits package to remain competitive in our area. Benefits Summary Healthcare...SuggestedFull timePart timeSeasonal workWork at officeLocal areaFlexible hoursNight shift- Job Title Under direction, employee maintains log of all radio traffic and computer inquires. Checks for criminal history when application is made for pistol permits. Conducts warrant checks and maintains jail sheets. Establishes and maintains files, keeps records and...Work at officeLocal areaNight shift
$40k - $45k
...Responsive Recruiter Ken Davis - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team Member. Our ideal candidate...For contractorsWork at officeImmediate startFlexible hours- ...eligibility, coverages, policy changes, transfers, claim submissions, and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Ensure the office is well-maintained and continuously equipped to...Work at officeFlexible hours
Shon Henry - State Farm Agent
Huntsville, AL26 days ago - ...Job Description Job Description Agent Aspirant – Future Agency Owner Madalynn Williams State Farm | Huntsville, AL We are looking for a highly motivated, competitive, and growth-minded Agent Aspirant who wants more than just a job. This opportunity is designed...Work at officeImmediate start
Madalynn Williams - State Farm Agent
Huntsville, AL2 days ago $16 - $22 per hour
...specialist to fulfill administrative duties and support insurance agents in gaining and keeping customers. As a Intern with our agency,... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use...Hourly payFor contractorsInternshipFlexible hoursState Farm
Owens Cross Roads, AL4 days ago$12.5 - $17.5 per hour
...and Renters Insurance. Our office is located in Scottsboro, AL . Our office is open 9:00-5:00. Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member. Our agency has received awards including: Chairman's Circle,...Hourly payPart timeFor contractorsInternshipWork at officeFlexible hoursState Farm
Madison, AL4 days ago- ...planning, and renters insurance. Our office is located in Scottsboro, AL. Our office is open 9:00-5:00. Before becoming a State Farm Agent, I was previously employed as a State Farm Agent Team Member. Our agency has received awards including: Chairman's Circle,...InternshipWork at office
David Perkins
Madison, AL4 days ago $16 per hour
...Job Description Job Description Defective Parts Agent Automation Personnel Services is looking for a skilled Defective Parts Agent at LG Electronics based in Huntsville, AL. In this role, you will be responsible for calling DMST (technicians) and request that defective...Weekly payFull timeTemporary workRemote work- ...about their development. If you want to work in an environment that is fun, challenging, and rewarding, then Ken Davis - State Farm Agent may be the right fit for you! If you want to make an impact, see immediate success, we could be the place for you! If you're willing...For contractorsWork at officeImmediate start
- ...advancement ROLE DESCRIPTION: Madalynn Williams - State Farm Agent is seeking an organized and efficient contributor to fulfill... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use...Full timeInternshipWork at office
- ...and currently lead a team of three. Before becoming a State Farm agent, I worked as a physician recruiter and later as an Office... ...life balance. We're proud to have earned more than 1,200 five-star reviews by treating every customer like family. Our culture is laid-back...Work at officeFlexible hours
- ...insurance Opportunity for advancement Paid time off Training & development ROLE DESCRIPTION: Ken Davis - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual...Work at officeFlexible hours
- ...their development. If you want to work in an environment that is fun, challenging, and rewarding, then Diana Lockmiller - State Farm Agent may be the right fit for you! If you're willing to work hard and you expect to get results from yourself and those around you, we...For contractorsWork at office
Diana Lockmiller - State Farm Agent
Madison, AL4 days ago - ...development ROLE DESCRIPTION: As a Receptionist - State Farm Agent Team Member with Tobias Kelley - State Farm Agent, you will... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Use exemplary...Hourly payWork at officeRelocation package
- ...numerous work such as online data entry, performing e-mail response, reviews, studies and also various other online jobs. This work at home... ..., then you are the person we are trying to find. Data entry agents come from all various backgrounds including, data entry,...Full timePart timeWork from home
- ...and currently lead a team of three. Before becoming a State Farm agent, I worked as a physician recruiter and later as an Office... ...life balance. We're proud to have earned more than 1,200 five-star reviews by treating every customer like family. Our culture is laid-back...Work at officeFlexible hours
- Data Entry Operator | Junior (Remote) Huntsville, AL, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...Remote job
- Data Entry Clerk Huntsville, Alabama, United States Job Openings Data Entry Clerk A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting data from paper documents into digital...
- Temporary Entry-Level Inventory Data Clerk FMS Aerospace is currently seeking a temporary entry-level Inventory Data Clerk for our Huntsville office location. This role combines physical inventory handling with basic computer data entry. Ideal candidate will be responsible...Temporary workWork experience placementWork at officeFlexible hours
- ...and finance in and out-processing. Peripheral task requirements include processing and management of voucher documents. Extension reviews include expenses as indicated on the voucher, supporting documents and travel orders to ensure service members are paid correctly....Work at officeWeekend work
Strategic Resources Inc
Huntsville, AL5 days ago - Production Control ClerkThe Production Control Clerk will compile and record a variety of production data such as use of material, quality control and other aspects of production.ResponsibilitiesCompiles and records production data for industrial establishments to compare...Work experience placement
Science and Engineering Services
Huntsville, AL5 days ago - Job Description Job Description Benefits: Bonus based on performance Flexible schedule Opportunity for advancement Training & development Job Description: We are currently seeking a dynamic and enthusiastic Telemarketer to join our team. As a Telemarketer...Flexible hours
- ...organizations including Tesla, American Medical Response, Mercedes Benz, and Mustang Cat. Similar Career Fields Include: Cargo and Freight Agent, Cargo Inspector, Vehicle Operator. About Our Organization: The U.S. Army is wealth of possibilities for your future –...Contract workPart timeImmediate startRelocation packageFlexible hours
U.S. Army Reserve
Huntsville, AL3 days ago - Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFull timeFlexible hours
- ...run for both vendors and employee reimbursement Reconcile vendor statements and maintain accounts payable files Expense report review, validation and posting Monitors vendor ledgers Accounts Receivable Billing Cost Plus (CPFF, CPAF, CPIF), Time and...Full timeContract workLocal area
Scientific Research
Huntsville, AL1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
- tsa agent Huntsville, AL
- executive protection agent Huntsville, AL
- freight agent no experience Huntsville, AL
- airport agent Huntsville, AL
- telemarketer - state farm agent team member Huntsville, AL
- import export agent Huntsville, AL
- agent Huntsville, AL
- state farm agent Huntsville, AL
- cruise agent Huntsville, AL
- signing agent Huntsville, AL