AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$114.1k - $268.18k
...areas of inspiration and expand your capabilities, then consider a career in Advisory. KPMG is currently seeking a Manager, SAP AI & Agent Governance to join our Advisory Technology Organization. Responsibilities: Lead the design and delivery of AI-enabled and...SuggestedFull timeH1bLocal areaKPMG
Los Angeles, CA3 days ago- Job Description Job Title: Document Reviewer Job Type: Contract Location: Remote Job Summary: In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world...SuggestedRemote jobContract work
- ...WGA Advisors is seeking a Principal AI Agent Architect to lead senior architectural and advisory work across the AI Workforce Solutions portfolio. This remote role involves guiding enterprise agentic AI strategy, conducting C-suite diagnostic engagements, and translating...SuggestedRemote job
$125 per hour
...leaders make and execute the decisions that matter most — across AI and digital transformation strategy, ESG, growth, culture, operations... .... The Opportunity WGA Advisors is engaging a Principal AI Agent Architect to lead the most senior architectural and advisory work...SuggestedHourly payTemporary workRemote work- A leading AI research company is seeking an Agents Product Manager to shape how developers build applications using AI models. The role involves understanding user needs, defining product roadmaps, and collaborating with engineering teams to enhance product offerings....SuggestedRelocation package
- ...Artificial Intelligence Summer Analyst for 2026 to develop autonomous agents and enhance existing systems. This on-site role requires a rising senior or recent graduate with hands-on experience in AI coding tools and full-stack development. The selected candidate will own...Summer work
- Affirm is reinventing credit with a remote-first culture and strong focus on agent enablement tools and AI-powered capabilities. We are seeking a Senior Product Manager to shape the experience of how millions get help and how internal teams deliver it. You will understand...Remote work
- ...Job Description System1 is building the next generation of AI Agent–powered consumer products — and we’re looking for a Lead Product... ...intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and...Full timeWork at officeLocal area
System1
Los Angeles, CA8 days ago $165k - $270k
...About the Role This is a founding-level engineering role at a seed-stage, YC-backed insurtech startup building AI agents that automate complex claims operations for insurance carriers and TPAs. You'll take true end-to-end ownership of the core platform — from eval pipelines...Full timeRemote workClera
Los Angeles, CA16 days ago- WelbeHealth is seeking an AI Enablement Lead, Strategic Initiatives to scale our distributed AI program. You will design, build, test, and maintain high‑impact AI agents that optimize workflows and partner with a network of super users to extend capabilities across the...
- SME Careers is seeking an Attorney SME for an hourly, remote contractor role. You will review AI-generated legal responses, craft expert content, and provide precise written feedback to improve accuracy and clarity of legal reasoning. The position supports multiple practice...Remote jobHourly payFor contractors
- This position involves bidding, engaging and reviewing single family residential appraisals covering all states. The qualified candidate will become familiar, and then competent, with RIMS which is an internet based appraisal management system. The position involves both...Full timeWork at office
- A fast-growing AI company in Los Angeles is seeking a Senior AI Systems Engineer to develop agent execution systems that interact with real infrastructure. The candidate should have strong backend experience, particularly in TypeScript and Python, and experience building...
- NIGHT AUDIT AGENT | SILVER LAKE POOL & INN POSITION DESCRIPTION: We're looking for a skilled Night Audit Agent that's prepared to oversee... ...thrive. There’s also an opportunity to expand your career trajectory as we are a fast‑growing company with hotels and restaurants in...Local areaFlexible hoursShift workNight shiftAfternoon shift
$24.84 - $26.15 per hour
...retirement benefits. JOB SUMMARY AND PURPOSE Under the direction of a Quality Assurance Coordinator, the Quality Assurance Reviewer I performs Quality Assurance review and audit of VOALA Program documentation and database practices related to grant, contract and...Hourly payContract workWork experience placementWork at officeLocal areaVolunteers of America
Los Angeles, CA4 days ago$170k - $220k
Senior AI Systems Engineer - Agentic Platforms $170k-$220k + Equity A fast-growing, venture-backed AI company is hiring engineers to... ...workflows across infrastructure . The team is building autonomous agents that operate in production environments, interacting with real...$30 per hour
...property and assets. Position: Overnight Executive Protection Agent Benefits: Full-time, non-exempt hourly compensation ($3... ...within the security industry, we’ll provide mentorship and trajectory. We are growing... come grow with us! Compensation is...Hourly payFull timeWork at officeNight shiftWeekend work- ...Senior Software Engineers to support an AI training project by creating reinforcement... ...golden reference solutions. Evaluate AI agents' ability to reason through complex codebases... .... Experience with rigorous code reviews and software engineering best practices ....Remote jobFor contractors
- ...Job Description System1 is building the next generation of AI Agent–powered consumer products, and we’re looking for a Lead AI Product... ..., simulation-based evals, automated scoring, human-in-the-loop review). Understanding of cost-aware execution for agents (dynamic...Full timeWork at officeLocal area
System1
Los Angeles, CA8 days ago $34.57 - $46.11 per hour
...every stage of your career. Try new things, learn new skills and discover what you excel at—all from Day One. Job Description Reviews appraisals and reports including desktop valuations from a wide variety of bank channels. Reviewer Appraiser positions review all types...Temporary workWork at officeLocal area3 days per weekU.S. Bank
Los Angeles, CA5 days ago- A national provider of Utilization Review services is seeking board-certified Occupational Medicine physicians to join their telework panel. This independent consulting role offers flexibility, allowing you to choose case types and workloads while maintaining your active...Remote workFlexible hours
- YO AI Labs is seeking an experienced AI/ML Engineer to build and deploy secure, scalable AI solutions for mission-critical initiatives. You will work with LLMs, RAG, prompt engineering, multi-agent systems, and cloud AI platforms to develop production-grade machine learning...Remote job
$35 - $40 per hour
Document Control Specialist Los Angeles, CA By combining first-hand knowledge of what cannabis users want with the most advanced technology and cutting-edge design, Puffco hasn't just grown a company: we have started a movement that will mainstream cannabis enjoyment...Contract workPart timeWork experience placementCasual workFlexible hoursPuffco
Los Angeles, CA2 days ago$90k - $120k
...organized and accessible records. Generate reports on document status, approvals, and outstanding actions. Conduct periodic reviews to ensure document integrity and compliance with project requirements. Assist in preparing documentation for regulatory submissions...Temporary workFor contractorsWork at officeLocal areaFlexible hoursLuster National
Los Angeles, CA2 days ago- kozmetickesluzby.vecnakraska.sk - Jobboard is looking for an App Reviewer to join their remote team based in Los Angeles, CA. The position involves participating in paid national and local focus groups, clinical trials, and market research studies. The ideal candidate...Remote jobLocal areaFlexible hours
$22 - $32 per hour
...organize, and manage high volumes of regulatory hard-copy records Review and separate records by proceeding number and applicable... ...Disclosure By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from KellyMitchell and...Full timeContract workWork at officeLocal areaRelocationKellyMitchell Group
Los Angeles, CA5 days ago$28 per hour
...Location : Glendale, CA (not remote) Hours : Part-time flexible (8-12 hours per week) Rate : $28 Per Hour As the On-Site Quality Control Reviewer (not remote), you are responsible for reviewing the work completed on-site to ensure it meets the quality standards set forth by...Hourly payExtra incomePart timeWork at officeFlexible hours$28 - $38 per hour
Job Title Job Description Benefits 401(k) Bonus based on performance Health insurance Paid time off Who Is Jlm Strategic Talent Partners & What Is Our Role We partner with National & International prime contractors to provide them with qualified talent they...Hourly payFor contractorsJLM Strategic Talent Partners
Glendale, CA4 days ago$33 - $58 per hour
Document Specialist Cooley is seeking a Document Specialist to join the Document Processing team. Position summary: The Document Processing Specialist is responsible for providing document production and processing support.The Document Processing Specialist will...Hourly payTemporary workWork at officeFlexible hoursCooley
Los Angeles, CA2 days ago- Document Control Clerk Leading Construction company in Los Angeles is actively seeking an experienced Document Control clerk to help with a major infrastructure project. Job Description The qualified individual will be responsible for: Implement document...Work at office
JLM Staffing
Los Angeles, CA1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
