App
Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Developer Trace Task Auditor

$70 - $90 per hour

Mercor

San Francisco, CA

Job Description

Job Description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: AI Developer Trace Task Auditor
Type: Contract
Compensation: $70–$90/hour
Location: Remote

Role Responsibilities

  • Evaluate the quality and correctness of AI-assisted software-development traces to enhance model training and evaluation.
  • Assess end-to-end coding sessions produced with AI-assisted developer tools for correctness, workflow soundness, and reasoning.
  • Provide clear, rubric-based written feedback to improve AI model outputs .
  • Review complex coding trajectories for alignment with best practices and correctness.
  • Collaborate with AI research teams to ensure consistency and relevance in training data.
  • Work independently and asynchronously to meet deadlines while enhancing AI model performance .

Qualifications

Must-Have

  • 3+ years professional software development.
  • Hands-on experience with AI-assisted coding tools and agentic/spec-driven workflows ( Cursor , GitHub Copilot , Claude Code , or similar).
  • Strong code-reading and debugging skills across full-stack or backend systems.
  • Ability to evaluate multi-step coding trajectories for correctness and best practice.

Preferred

  • Experience with Kiro or Amazon CodeCatalyst .
  • Prior work evaluating or grading AI-generated code .
  • Contributions to developer tooling .

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Apply
Vacancy posted 10 days ago
Similar jobs that could be interesting for youBased on the AI Developer Trace Task Auditor in San Francisco, CA vacancy
  • AI-Assisted Developer Workflows (Trace) AI Task Auditor - Freelance AI Trainer Project

    $60 per hour

     ...Job Application for AI-Assisted Developer Workflows (Trace) AI Task Auditor - Freelance AI Trainer Project at Meridial Back to jobs AI-Assisted Developer Workflows (Trace) AI Task Auditor - Freelance AI Trainer Project World Wide - Remote roject Overview... 
    Suggested
    Hourly pay
    Contract work
    Part time
    For contractors
    Freelance
    Remote work

    Invisible Technologies

    San Francisco, CA
    14 days ago
  • Software Engineer, Scheduled Tasks

    $196k - $312k

     ...from idea to production with speed, security, and exceptional developer experience.Now we're scaling our products for both agents and people...  ...millions of developers worldwide.About the Role:The Scheduled Tasks team builds the platform primitives that let applications and... 
    Suggested
    Work from home
    Worldwide
    Flexible hours

    Vercel

    San Francisco, CA
    2 days ago
  • Senior Agentic AI Engineer — Healthcare AI

    $110.7k - $372.9k

     ...Summary Senior Agentic AI Engineer — Healthcare AI Three...  ...horizon reliability — multi-step task completion, recovery from...  ...grounding & context engineering • Develop end-to-end Retrieval-Augmented...  ...• Implement observability and tracing for prompts, tool calls,... 
    Suggested
    Local area
    Visa sponsorship

    Deloitte

    San Francisco, CA
    1 day ago
  • AI Engineer, Product

    $171k - $240k

     ...control spend effortlessly. Brex’s AI-native automation and world-...  ...manual expense and accounting tasks for customers so they can...  ...the Brex platform. Our teams develop AI agents that don’t just surface...  ...surrounding harness (evaluation, tracing, feedback loops, human-in-the-... 
    Suggested
    Work at office
    Remote work
    Work from home

    Brex

    San Francisco, CA
    4 days ago
  • Senior Staff AI Engineer
     ...world.The role:SoFi’s SeniorStaff AI Engineer is a hands-on AI...  ...Distributed Agent Memory & State: Develop robust, persistent...  ...observability solutions and advanced tracing/testing frameworks that guarantee...  ...-trained models for specific tasks with minimal computational overhead... 
    Suggested
    Remote work

    SoFi

    San Francisco, CA
    3 days ago
  • South Asian Software Engineers: Coding Tasks for AI Evaluation

    $40 per hour

     ...What You'll Do Review proposed coding tasks and programming environments for realism Provide technical feedback on the structure and difficulty of evaluation harnesses Walk us through your approach to verifying complex code challenges Suggest improvements to make the... 
    Contract work
    Temporary work
    For contractors
    Remote work

    Terac

    San Francisco, CA
    2 days ago
  • AI Support Engineer
     ...We are an applied AI lab building end-to-end software agents....  ...that can reason on real-world tasks, apply to join us. About the...  ...across codebases, infrastructure, developer tooling, CI/CD systems, APIs,...  ...reproduce bugs, analyze logs, trace failures, form root-cause... 

    Cognition (Software Development Applications)

    San Francisco, CA
    4 days ago
  • Senior AI Systems Engineer — C++ / Applied AI

    $159.2k - $301.6k

     ...Opportunity We are looking for a Senior AI Systems Engineer with deep C++...  ..., backward compatibility, and developer experience in a mature...  ..., including automated task execution, output validation,...  ...handling. Create privacy-conscious tracing, observability, and diagnostics... 
    Full time
    Temporary work
    Local area
    Remote work
    Worldwide

    Adobe Systems

    San Francisco, CA
    9 hours ago
  • AI Product Engineer - ClickStack

    $500 per month

     ...Join Us In Building The AI Layer For Observability! ClickStack...  ...ClickHouse — logs, metrics, traces, and session replays unified so...  ...platform, with a focus on developer experience. If you've been building...  ...terms: p99 latency, cost per task, whether the system still... 
    Local area
    Remote work
    Home office
    Flexible hours

    ClickHouse

    San Francisco, CA
    1 day ago
  • TypeScript Developer - Fully Remote | Upto $130/task Task based

    $130 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...: Contract Compensation: $130 per task Location: Remote Role Responsibilities...  ...tooling. Preferred ~ Experience with developer tooling, compilers, test frameworks,... 
    Remote job
    Contract work
    Summer work

    Mercor

    San Francisco, CA
    6 days ago
  • Applied AI Engineer
     ...Pantomath is hiring an Applied AI Engineer, Agents to build AI...  ...with hands-on engineering to develop new capabilities, evaluate what...  ...improve performance on enterprise tasks. Design experiments and...  ...agents catch data incidents, trace them to their root cause across... 

    Pantomath

    San Francisco, CA
    10 days ago
  • AI Developer - Coding Sessions - AI Trainer

    $500 per unit

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...and Jack Dorsey . Position: Software Developers: Share Your AI Coding Sessions Type: Contract Compensation: $500 per task Location: Remote Commitment: Flexible... 
    Contract work
    Summer work
    Immediate start
    Remote work
    Flexible hours

    Mercor

    San Francisco, CA
    10 days ago
  • AI/ML Engineer

    $120k - $200k

     ...Description About the Role Join an AI team building systems that...  ...across large volumes of traces. Use user behavior signals...  ...do not explicitly report. Develop evaluation methods to measure...  ...appropriate models and methods for each task. Review production customer... 
    Full time

    Clera

    San Francisco, CA
    11 hours ago
  • Senior Staff AI Engineer
     ...Description Job Description Senior Staff AI Engineer, Agentic AI Location: Bay Area...  ...observable, measurable, and reliable through tracing and evaluation systems Evaluate agent performance against real scientific tasks Drive agent reliability to the level... 
    Full time
    Shift work

    NxT Level

    San Francisco, CA
    15 days ago
  • Senior Software Engineer, Release Infra

    $192k - $240k

     ...visibility, and control spend effortlessly. Brex’s AI-native automation and world-class service eliminate manual expense and accounting tasks for customers so they can focus on what...  ...observability tooling (metrics, logs, tracing) and integrating it into incident response... 
    Work at office
    Remote work
    Work from home

    Brex

    San Francisco, CA
    2 days ago
  • Staff Product Manager, AI Developer Platform

    $150k - $250k

     ...investment management company. Falcon is JLL's developer platform: the CI/CD, developer portal,...  ....After a team's first deploy, give them tracing, evals, and cost attribution by default,...  ...TeamReporting to the Head of Product, AI Platform. You'll lead a dedicated team of... 
    Full time
    Local area
    Remote work

    Jones Lang LaSalle

    San Francisco, CA
    4 days ago
  • AI Engineer II
     ...services, data flows, and machine learning and AI capabilities to make member communication...  ...maintainability of member communications.Trace and resolve issues across APIs, services,...  ...depth in your primary technologies, and develops the judgment to make sound tradeoffs.You practice... 
    Full time
    Internship
    Work at office
    Local area
    Remote work
    Worldwide
    3 days per week

    Hinge Health

    San Francisco, CA
    11 hours ago
  • Agentic AI Engineer, Senior - Anthropic/Claude

    $122k - $240.5k

    Position Summary Agentic AI is moving from experimentation to production, and...  ...workstreamsAbility to manage and prioritize multiple tasks in a fast-paced and dynamic...  ...offers comprehensive solutions for designing, developing, and operating advanced Data and AI... 
    Work at office
    Local area
    Visa sponsorship
    Shift work

    Deloitte

    San Francisco, CA
    9 hours ago
  • AI Agentic Engineer
     ...management (CLM). What you'll do As an AI Agentic Engineer on the Platform AI...  ...service requests, incidents, and routine tasks) using frameworks like Crew AI or Vertex...  ...languages (e.g., Python, Java)Experience in developing, designing or integrating front-end AI... 
    Permanent employment
    Full time
    Contract work
    Work at office
    Local area
    Remote work
    2 days per week

    DocuSign

    San Francisco, CA
    4 days ago
  • AI Developer

    $115k - $159k

     ...Lawyer.This is your opportunity to grow professionally in a dynamic, global environment, surrounded by forward-thinking peers.The AI Developer is a member of the Digital & Technology team responsible for developing, implementing, and supporting AI-powered applications,... 
    Full time

    Goodwin Procter

    San Francisco, CA
    3 days ago
  • Senior AI Software Engineer, Risk - Insurance Claims Management

    $148k - $185k

     ...innovations and resolving pain points with AI automation solutionsDeliver scalable,...  ...modeling, and operational cost.Experience developing AI features using multimodal LLMs, RAG pipelines...  ...performance with robust evals,agent tracing tools, fine-tuning LLMs, optimizing RAG... 
    Hourly pay
    Work experience placement
    Work at office
    Local area
    3 days per week

    Lyft

    San Francisco, CA
    2 days ago
  • Senior AI Software Engineer

    $149k - $240k

    Who We AreHP IQ is HP’s new AI innovation lab. Combining startup agility with HP’s global...  ...across HP’s portfolio. Together, we’re developing intuitive, adaptive solutions that spark...  ...breakthrough solutions that make complex tasks feel effortless, teamwork more natural,... 
    Full time
    Temporary work
    Local area
    Flexible hours

    HP IQ

    San Francisco, CA
    4 days ago
  • AI Software Engineer - HP IQ

    $127k - $175k

    AI Software Engineer - HP IQDescription -Who We AreHP IQ is HP’s new AI innovation lab...  ...across HP’s portfolio. Together, we’re developing intuitive, adaptive solutions that spark...  ...breakthrough solutions that make complex tasks feel effortless, teamwork more natural, and... 
    Full time
    Temporary work
    Local area
    Relocation
    Flexible hours
    Shift work

    Juniper Networks

    San Francisco, CA
    1 day ago
  • Agentic AI Engineer, Senior

    $122k - $240.5k

    Position Summary Join our AI & Engineering team in transforming technology platforms...  ...Plan and assign resources to associated tasks and deliverablesMonitoring progress of...  ...comprehensive solutions for designing, developing, and operating advanced Data and AI platforms... 
    Local area
    Visa sponsorship

    Deloitte

    San Francisco, CA
    1 day ago
  • Senior Auditor

    $80k - $150k

     ...Senior Auditor Employment Type: Full Time, Mid-level Department...  ...payments and kickbacks, tracing money flows, identifying assets...  ...assessing collectability, and developing payment terms and schedules....  ...use artificial intelligence (AI) tools to support parts of the... 
    Full time
    Flexible hours

    Contact Government Services, LLC

    San Francisco, CA
    5 days ago
  • Staff Software Engineer, RL Environments

    $252k - $315k

    About Scale AIAt Scale, our mission is to develop reliable AI systems for the world's most important...  ...real applications, designing task suites that expose specific capability...  ...CI/CD.Strong observability instincts: tracing, structured logging, and metrics for systems... 
    Full time
    Remote work

    Scale AI

    San Francisco, CA
    4 days ago
  • AI and Data Science Engineer III

    $137.5k - $193.6k

     ...strategies, advanced analytics, Generative AI, transformative technologies, and creative...  ..., and predictive modeling techniques to develop and refine solutions.Building proofs of concept...  ...to manage and prioritize multiple tasks in a fast-paced and dynamic environmentStrong... 
    Local area

    Deloitte

    San Francisco, CA
    4 days ago
  • AI & Data Science Engineer II

    $91.1k - $179.5k

    Position Summary Join our AI & Engineering team in transforming technology platforms...  ...to manage and prioritize multiple tasks in a fast-paced and dynamic environmentStrong...  ...comprehensive solutions for designing, developing, and operating advanced Data and AI... 
    Local area
    Visa sponsorship

    Deloitte

    San Francisco, CA
    2 days ago
  • Senior/Staff Applied AI Engineer, Agent Harness

    $200k - $300k

     ...Description Senior/Staff Applied AI Engineer, Agent Harness...  ...requests scoped access for each task, escalates to a human for approval...  ...actually work for users. You will develop the core agent harness,...  ...analyze production failures and trace them to the right layer (model,... 
    Full time
    Contract work
    H1b
    Work at office
    Visa sponsorship
    Monday to Friday

    Transparent Search Group

    San Francisco, CA
    25 days ago
  • Program Manager, AI and Developer Community Content and Events

    $148k - $179k

    At Datadog we’re growing our investment in AI and Developer focused events. As we scale this program we’re looking to hire a Program Manager on our Community team to support our content and advocacy team efforts across these events.The Datadog Technical Advocacy team are... 
    Work at office
    Local area

    Datadog

    San Francisco, CA
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Developer Trace Task Auditor. Be the first to apply!

Related searches