Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Quality Lead, Agentic AI Workflow Evaluation [Remote]

$75 - $85 per hour

Innodata Inc.

San Jose, CA
  • Remote job

Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.

Scope of the Role: 

We are standing up a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. Reviewers work through ambiguous, multi-step scenarios inside isolated test environments, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. The Quality Lead is the person accountable for whether that output is any good.  

This is a senior individual contributor role. You will not manage the reviewers — that sits with the Engagement Manager — but you set the standard they are held to. You own the audit sample, run calibration, keep the rubric usable as real cases stress it, and train reviewers into the work. You are also the deputy: when the Engagement Manager is out, the engagement runs on you.  

The quality approach here is not fully defined. We expect you to build it in partnership with the customer's quality leads, or at minimum to take what they have, run it honestly, and come back with specific recommendations for where it falls short.  

What You’ll Own:

  • Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported  
  • Build that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows  
  • Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes  
  • Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases  
  • Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer  
  • Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready  
  • Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working  
  • Report quality trends to the Engagement Manager and, alongside them, to the customer  
  • Deputize for the Engagement Manager on delivery operations during absences  
  • Maintain information security, privacy, and facility access practices required by the customer's onsite environment  

You’ll Thrive in This Role If You Have:

  • Bachelor's degree or equivalent practical experience  
  • 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain  
  • Direct experience owning a quality function: you designed the audit, not just executed someone else’s  
  • Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review  
  • Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes  
  • Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better  
  • Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it  
  • Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces) 
  • Experience training or onboarding reviewers into rubric-based work  

The expected hourly salary range for this position is $75-85 p/hour, based on experience, skills, and qualifications.

 

Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at  

If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at  View email address on job-boards.greenhouse.io and consider reporting it to the FTC at  ReportFraud.ftc.gov .

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Quality Lead, Agentic AI Workflow Evaluation [Remote] in San Jose, CA vacancy
  • $75 - $85 per hour

     ...Artificial Intelligence (AI) are inextricably...  ...by providing the data, evaluation frameworks, and human...  ...delivering the highest quality data and outstanding outcomes...  ...complex, real-world agentic AI workflows for a frontier AI...  ...reviewers and one QA lead working inside isolated... 
    Quality
    Remote job
    Hourly pay
    Fixed term contract

    Innodata Inc.

    San Jose, CA
    1 day ago
  • $101.14k - $182.62k

     ...AI Lead At Capgemini Engineering, the world leader in engineering...  ...data modeling, data quality, data pipelines, data platforms...  ...Conversational AI, AI agents, Agentic Workflows and GenAI application...  ...lifecycle, model integration, evaluation and production deployment.... 
    Quality
    Permanent employment
    Full time
    Contract work
    Local area

    Capgemini

    Santa Clara, CA
    2 days ago
  • $262.5k - $299.6k

     ...Applied Researcher 5 (AI Foundations: Evaluation and Agentic Workflows) At Capital One, we are creating trustworthy...  .... For years, Capital One has been leading the industry in using machine learning...  ...record of coming up with high quality ideas or improving upon existing ideas... 
    Quality
    Full time
    Part time
    Work at office
    Local area
    Flexible hours

    Capital One

    San Jose, CA
    4 days ago
  • $229.9k - $262.4k

     ...AI Engineer 5 (AI Foundations: Evaluations and Agentic Workflows) At Capital One, we are creating responsible and reliable AI...  ...teams to deliver our industry leading capabilities with breakthrough...  ...build systems, take pride in the quality of your work, and also share our... 
    Quality
    Full time
    Part time
    Work at office
    Local area

    Capital One

    San Jose, CA
    4 days ago
  • $132.23k - $176.31k

     ...trusted network for the AI‑powered world,...  ...The Role The Senior Lead Marketing Data Platform...  ...ready data structures, and agentic operational frameworks...  ..., schema control, data quality, and lifecycle management...  ...technology ecosystem. Evaluate emerging technologies and... 
    Quality
    Full time
    Temporary work

    Lumen

    San Jose, CA
    3 days ago
  • $224k - $356.5k

     ...seeking a highly technical, partner-facing Agentic AI Lead to drive the co-design of Generative...  ..., multi-agent, and long-horizon workflows on NVIDIA's stack (NeMo, Nemotron, NIM...  ...and emerging agent frameworks). Drive evaluations of NVIDIA libraries and ship the code... 
    Full time

    Nvidia

    Santa Clara, CA
    13 hours ago
  • $201.3k - $352.3k

     ...ServiceNow is the AI control tower for business...  ...any data, and any workflow— helping 85% of the...  ...-to-end, industry-leading work experiences...  ...Manager, Agentic & GenAI Benchmarking and Evaluations to establish and lead...  ...) to measure agent quality and eliminate regressions... 
    Quality
    Full time
    Work experience placement
    Work at office
    Immediate start
    Remote work
    Flexible hours
    Shift work

    ServiceNow

    Santa Clara, CA
    12 days ago
  • $237.6k - $401.7k

     ...Machine Learning and AI, Software and ServicesLocation...  ...assistant can be and lead the platform that...  ....As part of the Siri Agentic Evaluation Engineering organization...  ...a run, and the data workflows that carry results from...  ...to drive coverage and quality decisions with senior... 
    Quality
    Relocation

    Apple

    Cupertino, CA
    3 days ago
  • $145k - $180k

     ...Legence (Nasdaq: LGN) is a leading provider of engineering...  ...I&M Segment Applied AI Lead implements AI...  ...maintenance and service workflows, planning and dispatch,...  ...and energy performance, quality, safety, and technician...  ...configure, and support the agentic AI tools the company... 
    Quality
    Local area
    Immediate start
    Remote work
    Flexible hours

    Legence Holdings

    San Jose, CA
    1 day ago
  • $240k - $333k

     ...Tooling. Build and lead a team of PMs...  ...PM and UX research workflows, from evidence-backed...  ...synthetic users, AI-moderated research...  ...DeepMind on design evaluations so internal usage...  ...Google builds in the agentic era.Minimum...  ...products with measurable quality, including... 
    Quality

    Google

    San Jose, CA
    2 days ago
  • $50 - $55 per hour

     ...and Artificial Intelligence (AI) are inextricably linked....  ...intelligence by providing the data, evaluation frameworks, and human...  ...legacy delivering the highest quality data and outstanding...  ...evaluate complex, real-world agentic AI workflows, assessing whether AI agents... 
    Quality
    Remote job
    Hourly pay
    Fixed term contract

    Innodata Inc.

    San Jose, CA
    1 day ago
  • $20 - $94 per hour

     ...learning ignited modern AI - the next era of...  ...foundation models, and agentic AI systems. Intelligent...  ...train, fine-tune, and evaluate foundation models using...  ...Build and curate high-quality datasets and evaluation...  ..., and model evaluation workflows. Strong problem-solving... 
    Quality
    Hourly pay
    Summer work
    Internship

    NVIDIA

    Santa Clara, CA
    1 day ago
  • $267.8k - $401.7k

     ...Engineering (ASE) powers AI and machine learning...  ...leader to lead AI Safety Engineering & Evaluation within the Human-Centered...  ...models, assessment workflows, and measurement capabilities...  ...complex production-quality systems.8+ years of...  ...-based evaluation, agentic systems, model... 
    Quality
    Relocation

    Apple

    Cupertino, CA
    3 days ago
  • $100k

    Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance...  ...clear decisions, manual workflows into scalable automations...  ...and business logic into agentic solutions.The role sits...  ...workflows, improve data quality, and deliver executive-ready... 
    Quality
    Permanent employment

    Tenstorrent

    Santa Clara, CA
    3 days ago
  • $187.5k - $253.5k

     ...Role We're hiring an AI Web Platform Lead to modernize how our...  ...answer engines, to CMS workflows that move ideas from...  ...rollout Actively evaluate CMS platforms and adjacent...  ...authoring, agentic pipelines — and strong...  ...accelerates vs. degrades quality ~ Working knowledge... 
    Quality
    Full time

    Intuit

    Mountain View, CA
    4 days ago
  • $206.2k - $356.4k

     ...the ground up. As part of the Agentic Search Evaluation team, you will help advance...  ...for building scalable, high-quality solutions to evaluate...  ...product improvements. You will lead a team focused on evaluating...  ...that power personalized Siri AI and enable answers to users'... 
    Quality
    Relocation

    Apple

    Cupertino, CA
    3 days ago
  •  ...technical point of contact, leading and mentoring a...  ...to onboard new AI agents and standardize model evaluation processes. ML/GenAI...  ...optimize performance. Agentic Evaluation Infrastructure & Quality: Configure isolated/...  ...and maintain CI/CD workflows for safe, automated... 
    Quality
    Full time

    Shrive Technologies LLC

    San Jose, CA
    5 days ago
  • $197.3k - $225.1k

    AI Engineer 4 (Gen AI Platform Services: Agentic AI, Agent Guardrails, Agent Evaluation, Agent Memory) At Capital One, we are...  ...deliver our industry leading capabilities with...  ...agents and multi-agent workflows, similarity search,...  ...take pride in the quality of your work, and... 
    Quality
    Full time
    Part time
    Local area

    Capital One

    San Jose, CA
    4 days ago
  • $229.9k - $262.4k

    Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating...  ...deliver our industry leading capabilities with...  ...systems, take pride in the quality of your work, and...  ...systems and agentic workflows ~ Passion for staying... 
    Quality
    Full time
    Part time
    Local area

    Capital One

    San Jose, CA
    4 days ago
  •  ...Position Overview: The Agentic AI Engineer will design, develop...  ...focuses on building autonomous AI workflows, integrating large language...  .... Understanding of AI evaluation, observability, guardrails,...  ..., reliability, and response quality. Develop security controls... 
    Quality
    H1b
    Local area

    Robotic Technologies Automation

    San Jose, CA
    1 day ago
  •  ...builds the world's largest AI chip, 56 times larger...  ...to deliver industry-leading training and inference...  ...intelligence via additional agentic computation.Cerebras...  ...performance, and operational quality of the systems Cerebras...  ...alerting, and on-call workflows. Run incident response... 
    Quality

    Cerebras Systems

    Sunnyvale, CA
    13 hours ago
  • $207k - $300k

     ...Software Engineer, Eval Infra TL, Quality Platform, Gemini Cloud Operator, Agentic AI Google's software engineers...  ...interactions, implementing real-time evaluations to detect successes and failures,...  ...contribute to making Gemini the leading model for cloud operational tasks... 
    Quality

    Google

    Sunnyvale, CA
    3 days ago
  • $175.5k - $311.7k

     ...better than we found it.We are looking for an Evaluation Science Lead to define how our multilingual Globalization AI solutions are measured and validated across Apple...  ...individual contributor on the Globalization Quality and Operations team, you will design statistically... 
    Quality
    Relocation
    Afternoon shift

    Apple

    Cupertino, CA
    3 days ago
  • $220k - $230k

     ...Title of position: Innovation Lead - Office of the CTO Position...  ...(EXTR) is a global leader in AI-powered cloud networking, committed...  ...& Development Identify and evaluate emerging opportunities in ML/...  ...exist Professional Qualities Ambitious and driven, thriving... 
    Quality
    Full time
    H1b
    Work at office
    Local area
    Remote work
    Work from home
    Worldwide
    Work visa
    Flexible hours

    Extreme Networks

    San Jose, CA
    a month ago
  • $31.83 - $63.65 per hour

     ...Job Description Leads complex, data center hardware support services...  ..., and automate repeatable workflows. Mentors and develops team...  ...forward plans, and ensures high-quality documentation of resolutions within...  ...to life-saving care. And with AI embedded across our products... 
    Quality
    Hourly pay
    Temporary work
    Local area
    Flexible hours
    Shift work

    Oracle Corporation

    Santa Clara, CA
    2 days ago
  • $120k - $150k

     ...enterprise applications and AI systems. Founded by the...  ...mission-critical workflows and AI agents at massive...  ...complex deterministic and agentic workflows with...  ...platform through high-quality technical education, community...  ...build, troubleshoot, and evaluate new technologies ~... 
    Quality
    Work at office

    Orkes

    Santa Clara, CA
    1 day ago
  • $224k - $356.5k

     ...Automated Synthetic Chemistry Science Lead who can coordinate science...  ...analytical characterization, AI modeling, laboratory...  ...without compromising scientific quality.What you'll be doing:Coordinate...  ...development, AI enabled science workflows, or equivalent experience.Deep... 
    Quality
    Full time

    Nvidia

    Santa Clara, CA
    13 hours ago
  • $224k - $356.5k

    NVIDIA seeks an AI enabled drug discovery science leader who can...  ...into executable plans.Lead NVIDIA workstreams spanning data...  ...risk and dependency tracking, quality thresholds, and paths for addressing...  ...lineage, compute planning, evaluation metrics, and closed-loop experimentation... 
    Quality
    Full time

    Nvidia

    Santa Clara, CA
    4 days ago
  •  ...discovery to powering AI and the technologies people...  ..., or bringing leading edge products to market...  ...and Validation teams to evaluate and optimize the power...  ...delivering timely, high-quality products. A broad and thorough...  ...tools and improve the workflow. Deliver high-... 
    Quality
    Full time

    AMD

    Santa Clara, CA
    12 days ago
  • $175.5k - $263.8k

     ...changes across teams, and build AI tools and automation that...  ...build AI-assisted tools and agentic workflows that make triage faster and...  ...experience in software development, quality, or reliability...  ...Understanding of how ML models are evaluated and deployed, and how they... 
    Quality
    Relocation

    Apple

    Cupertino, CA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Quality Lead, Agentic AI Workflow Evaluation [Remote]. Be the first to apply!