AI Developer Trace Task Auditor
$70 - $90 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: AI Developer Trace Task Auditor
Type: Contract
Compensation: $70–$90/hour
Location: Remote
Role Responsibilities
- Evaluate the quality and correctness of AI-assisted software-development traces to enhance model training and evaluation.
- Assess end-to-end coding sessions produced with AI-assisted developer tools for correctness, workflow soundness, and reasoning.
- Provide clear, rubric-based written feedback to improve AI model outputs .
- Review complex coding trajectories for alignment with best practices and correctness.
- Collaborate with AI research teams to ensure consistency and relevance in training data.
- Work independently and asynchronously to meet deadlines while enhancing AI model performance .
Qualifications
Must-Have
- 3+ years professional software development.
- Hands-on experience with AI-assisted coding tools and agentic/spec-driven workflows ( Cursor , GitHub Copilot , Claude Code , or similar).
- Strong code-reading and debugging skills across full-stack or backend systems.
- Ability to evaluate multi-step coding trajectories for correctness and best practice.
Preferred
- Experience with Kiro or Amazon CodeCatalyst .
- Prior work evaluating or grading AI-generated code .
- Contributions to developer tooling .
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the AI Developer Trace Task Auditor in San Francisco, CA vacancy
- ...Evaluate the quality, correctness, and production-readiness of Kubernetes tasks used to train and evaluate a frontier AI lab's models. You'll assess cluster-operations scenarios, manifest correctness, and failure-mode troubleshooting , and provide clear, rubric-based...SuggestedTemporary work
Mercor
San Francisco, CA23 days ago - ...Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity , and provide clear, rubric...SuggestedTemporary work
Mercor
San Francisco, CA23 days ago $130 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...: Contract Compensation: $130 per task Location: Remote Role Responsibilities... ...Ability to use debuggers , sanitizers , tracing, and profilers . ~ Precise written...SuggestedContract workSummer workRemote workMercor
San Francisco, CA3 days ago- ...About Echelon Echelon is building the AI platform for Business Operations. Our goal... ...specialized agents and agent swarms, including task decomposition, delegation, shared state,... ...and prompting strategies with production traces and measured outcomes rather than...SuggestedWork at office
Echelon
San Francisco, CA5 hours ago $70 - $90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Larry Summers , and Jack Dorsey . Position: SWE-Bench Task Auditor Type: Contract Compensation: $70–$90/hour Location...SuggestedContract workSummer workRemote workMercor
San Francisco, CA4 days ago$70 - $90 per hour
...Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate a frontier AI lab's models. You'll assess experiment design, model-selection reasoning, and evaluation methodology , and provide clear, rubric-based...Contract work$70 - $90 per hour
...soundness of AWS serverless and infrastructure-as-code tasks used to train and evaluate a frontier AI lab's models. You'll assess multi-service serverless... ...• AWS certifications (Solutions Architect / Developer) • Experience with observability and cost-optimization...Contract workLocal area$200k - $350k
Get AI-powered advice on this job and more exclusive features.... ...engineering talent - highly analytical developers with strong educational and... ...and voice , executing complex tasks with precision Translating... ...observability tools (e.g., Trace View) to audit and improve agent...Full timeFlexible hours$171k - $240k
...control spend effortlessly. Brex’s AI-native automation and world-... ...manual expense and accounting tasks for customers so they can... ...the Brex platform. Our teams develop AI agents that don’t just surface... ...human-in-the-loop workflows, tracing, evaluation, or feedback mechanisms...Full timeWork at officeRemote workWork from home$80.4k - $316.3k
...core and unleashing the power of AI to create value at speed for... ...), instrument logs, metrics, traces, and SLOs, and lead incident response... ...development and operational tasks. Continuously identify... ...set engineering standards and develop the engineers around you. Facilitate...Full timeWork experience placementLive inWork at officeLocal area$100k - $170k
...the ones who actually run it. We call them AI operators: the employees inside companies... ...Support as a Product” backlog. You will own tasks with shipped/doing/todo statuses like a... ...distributed systems. You’re not stopped by a stack trace. Bidirectional communication: can...Immediate start$229k - $267.1k
...day-to-day project management tasks, including communicating project... ...project execution. Design, develop, and operate backend microservices... ...metrics, logs, and request tracing (e.g., Amazon CloudWatch, AWS... ...summarized by artificial intelligence (AI). You will have the...Full timeFor contractorsRemote workReddit
San Francisco, CA1 day ago$200k - $300k
...Description Senior/Staff Applied AI Engineer, Agent Harness... ...requests scoped access for each task, escalates to a human for approval... ...actually work for users. You will develop the core agent harness,... ...analyze production failures and trace them to the right layer (model,...Full timeContract workH1bWork at officeVisa sponsorshipMonday to FridayTransparent Search Group
San Francisco, CA10 days ago$31 - $49 per hour
...your internship or Co‑op, you will be tasked with one or more mission‑critical... ...address their monitoring needs along with AI native initiatives. The primary goal... .... You will: Design and develop monitoring, logging, tracing, and alerting solutions to improve product...Hourly payFull timeInternshipWorldwideC0035 LiveRamp, Inc.
San Francisco, CA1 day ago$40 per hour
Overview AI Trainer - Advanced SQL Developers About Prolific Prolific is not just another player in the AI space - we are building the biggest pool... ...an assessment to test your skills and suitability for AI tasks. If successful, you’ll be invited to join Prolific as a...Remote jobSelf employmentWork from homeFlexible hours$150k - $250k
...investment management company. Falcon is JLL's developer platform: the CI/CD, developer portal,... ....After a team's first deploy, give them tracing, evals, and cost attribution by default,... ...TeamReporting to the Head of Product, AI Platform. You'll lead a dedicated team of...Full timeLocal areaRemote workJones Lang LaSalle
San Francisco, CA6 hours ago$148k - $185k
...innovations and resolving pain points with AI automation solutionsDeliver scalable,... ...modeling, and operational cost.Experience developing AI features using multimodal LLMs, RAG pipelines... ...performance with robust evals,agent tracing tools, fine-tuning LLMs, optimizing RAG...Hourly payWork experience placementWork at officeLocal area3 days per weekLyft
San Francisco, CA4 days ago- ...services, data flows, and machine learning and AI capabilities to make member communication... ...maintainability of member communications.Trace and resolve issues across APIs, services,... ...depth in your primary technologies, and develops the judgment to make sound tradeoffs.You practice...Full timeInternshipWork at officeLocal areaRemote workWorldwide3 days per week
Hinge Health
San Francisco, CA2 days ago $68k - $85k
...determine the effectiveness of compliance and store operational tasks. This individual will visit store locations daily and perform regularly... ...with the Director, Internal Audit and operational functions to develop remediation activities.Develop recommendations for process...Work at officeFlexible hoursMonro, Inc.
San Francisco, CA2 days ago$200k - $220k
...Safer World. TRM Labs provides AI-powered intelligence solutions... ...enable investigators to trace illicit activity, build cases,... ...Infrastructure (RDBMS), you will develop, operate, and scale the... ...automation of routine database tasks; Postgres extension development...WorldwideDay shiftTRM Labs
San Francisco, CA20 hours ago- ...Title: Senior Agentic AI Engineer (Hybrid - San Francisco) Position Type: Full-Time... ...LangGraph, Google ADK, or OpenAI Agents SDK. Develop MCP servers and tool-registration APIs to... ...datasets, replay runs, and full-stack tracing. Implement knowledge-graph-backed...Full time
Oscar
San Francisco, CA2 days ago - ...AI Engineer Location: San Francisco (Hybrid - in-office 3+ days/week in downtown SF... ...personalization and automation of complex tasks Build prompt pipelines, retrieval systems... ...collaborative, deeply technical team to develop leading‑edge AI solutions How to apply...Work at office3 days per week
Billables Incorporated
San Francisco, CA4 days ago - ...physical world with tools that can't see it. Cyvl is the Physical AI company changing that. We build purpose-built sensors, computer... ...real world. Ownership that compounds. You own outcomes, not tasks. You work directly with customers and across our team, and you'll...Work at officeRemote work
RiverPark Ventures
San Francisco, CA6 hours ago $150k - $210k
...AI Engineer Location: San Francisco, CA, in-office Compensation: 150,000 - $210,000 Company: Cargomatic, Inc. Disruptive... ...AI systems capable of autonomous decision‑making and task execution Develop AI‑powered automation tools to streamline workflows across operations...Work at officeLocal areaImmediate startFlexible hoursArgo AI
San Francisco, CA4 days ago$150k - $250k
...Join to apply for the AI Engineer role at Arnata Base pay range $150,000.00/yr - $250,000.00/yr Why Arnata... ...Do As An AI Engineer Build AI Agents: Design and develop AI systems capable of handling complex, real‑world tasks over multiple channels, including email, phone,...Full timeArnata
San Francisco, CA2 days ago- ...Smart Bricks is a frontier AI lab building autonomous reasoning systems that allow capital to discover,... ...valuation, forecasting, scoring, and classification tasks - validated against real-world outcomes Developing and optimising LangChain-based agentic orchestration...Live in
Smart Bricks
San Francisco, CA2 days ago $175k - $275k
...the company Our client is a seed-stage AI company building the data and infrastructure... .... Its initial focus is design, where it develops reinforcement learning environments and evaluation... ...the systems that help models improve on tasks where there is no single objectively...Full timeRaydar
San Francisco, CA3 days ago- ...Francisco, California. The Role: As a AI Engineer , you will be a core... ...with filesystems, and complete complex user tasks. This role spans frontend interfaces, secure... ...and accessibility frameworks Experience developing browser extensions or automation tools with...Work at officeRelocation package
Zyphra
San Francisco, CAa month ago $120k - $200k
Our client, an AI-driven legal tech company, is hiring a Founding AI Engineer to join the... ...agents that execute complex, multi-step tasks using frameworks such as LangChain and Deep... ...management and tool integration. ~Develop RAG, memory and retrieval systems that allow...Full timeAlldus International Consulting Ltd
San Francisco, CAa month ago$120k - $180k
...About the Role This is a founding engineering role at a seed-stage AI startup building the intelligence layer for the construction... ...~ Willingness to handle the full spectrum of startup engineering tasks (no task is too small). ~ Based in San Francisco or willing to...Full timeWork at officeRelocationClera
San Francisco, CAa month ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Developer Trace Task Auditor. Be the first to apply!
Related searches
- senior ai engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- ai ml engineer San Francisco, CA
- ai developer San Francisco, CA
- ai prompt engineer San Francisco, CA
- ai engineer San Francisco, CA
- ai research engineer San Francisco, CA
- ai engineer remote San Francisco, CA
- inventory auditor San Francisco, CA
- lease auditor San Francisco, CA