Quality Lead, Agentic AI Workflow Evaluation [Remote]
$75 - $85 per hourInnodata Inc.
- Remote job
Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.
Scope of the Role:
We are standing up a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. Reviewers work through ambiguous, multi-step scenarios inside isolated test environments, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. The Quality Lead is the person accountable for whether that output is any good.
This is a senior individual contributor role. You will not manage the reviewers — that sits with the Engagement Manager — but you set the standard they are held to. You own the audit sample, run calibration, keep the rubric usable as real cases stress it, and train reviewers into the work. You are also the deputy: when the Engagement Manager is out, the engagement runs on you.
The quality approach here is not fully defined. We expect you to build it in partnership with the customer's quality leads, or at minimum to take what they have, run it honestly, and come back with specific recommendations for where it falls short.
What You’ll Own:
- Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported
- Build that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows
- Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes
- Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases
- Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer
- Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready
- Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working
- Report quality trends to the Engagement Manager and, alongside them, to the customer
- Deputize for the Engagement Manager on delivery operations during absences
- Maintain information security, privacy, and facility access practices required by the customer's onsite environment
You’ll Thrive in This Role If You Have:
- Bachelor's degree or equivalent practical experience
- 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain
- Direct experience owning a quality function: you designed the audit, not just executed someone else’s
- Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review
- Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes
- Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better
- Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it
- Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces)
- Experience training or onboarding reviewers into rubric-based work
The expected hourly salary range for this position is $75-85 p/hour, based on experience, skills, and qualifications.
Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at
If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at View email address on job-boards.greenhouse.io and consider reporting it to the FTC at ReportFraud.ftc.gov .
$75 - $85 per hour
...Artificial Intelligence (AI) are inextricably... ...by providing the data, evaluation frameworks, and human... ...delivering the highest quality data and outstanding outcomes... ...complex, real-world agentic AI workflows for a frontier AI... ...reviewers and one QA lead working inside isolated...QualityRemote jobHourly payFixed term contract$101.14k - $182.62k
...AI Lead At Capgemini Engineering, the world leader in engineering... ...data modeling, data quality, data pipelines, data platforms... ...Conversational AI, AI agents, Agentic Workflows and GenAI application... ...lifecycle, model integration, evaluation and production deployment....QualityPermanent employmentFull timeContract workLocal area$262.5k - $299.6k
...Applied Researcher 5 (AI Foundations: Evaluation and Agentic Workflows) At Capital One, we are creating trustworthy... .... For years, Capital One has been leading the industry in using machine learning... ...record of coming up with high quality ideas or improving upon existing ideas...QualityFull timePart timeWork at officeLocal areaFlexible hours$229.9k - $262.4k
...AI Engineer 5 (AI Foundations: Evaluations and Agentic Workflows) At Capital One, we are creating responsible and reliable AI... ...teams to deliver our industry leading capabilities with breakthrough... ...build systems, take pride in the quality of your work, and also share our...QualityFull timePart timeWork at officeLocal area$132.23k - $176.31k
...trusted network for the AI‑powered world,... ...The Role The Senior Lead Marketing Data Platform... ...ready data structures, and agentic operational frameworks... ..., schema control, data quality, and lifecycle management... ...technology ecosystem. Evaluate emerging technologies and...QualityFull timeTemporary work$224k - $356.5k
...seeking a highly technical, partner-facing Agentic AI Lead to drive the co-design of Generative... ..., multi-agent, and long-horizon workflows on NVIDIA's stack (NeMo, Nemotron, NIM... ...and emerging agent frameworks). Drive evaluations of NVIDIA libraries and ship the code...Full time$201.3k - $352.3k
...ServiceNow is the AI control tower for business... ...any data, and any workflow— helping 85% of the... ...-to-end, industry-leading work experiences... ...Manager, Agentic & GenAI Benchmarking and Evaluations to establish and lead... ...) to measure agent quality and eliminate regressions...QualityFull timeWork experience placementWork at officeImmediate startRemote workFlexible hoursShift work$237.6k - $401.7k
...Machine Learning and AI, Software and ServicesLocation... ...assistant can be and lead the platform that... ....As part of the Siri Agentic Evaluation Engineering organization... ...a run, and the data workflows that carry results from... ...to drive coverage and quality decisions with senior...QualityRelocation$145k - $180k
...Legence (Nasdaq: LGN) is a leading provider of engineering... ...I&M Segment Applied AI Lead implements AI... ...maintenance and service workflows, planning and dispatch,... ...and energy performance, quality, safety, and technician... ...configure, and support the agentic AI tools the company...QualityLocal areaImmediate startRemote workFlexible hours$240k - $333k
...Tooling. Build and lead a team of PMs... ...PM and UX research workflows, from evidence-backed... ...synthetic users, AI-moderated research... ...DeepMind on design evaluations so internal usage... ...Google builds in the agentic era.Minimum... ...products with measurable quality, including...Quality$50 - $55 per hour
...and Artificial Intelligence (AI) are inextricably linked.... ...intelligence by providing the data, evaluation frameworks, and human... ...legacy delivering the highest quality data and outstanding... ...evaluate complex, real-world agentic AI workflows, assessing whether AI agents...QualityRemote jobHourly payFixed term contract$20 - $94 per hour
...learning ignited modern AI - the next era of... ...foundation models, and agentic AI systems. Intelligent... ...train, fine-tune, and evaluate foundation models using... ...Build and curate high-quality datasets and evaluation... ..., and model evaluation workflows. Strong problem-solving...QualityHourly paySummer workInternship$267.8k - $401.7k
...Engineering (ASE) powers AI and machine learning... ...leader to lead AI Safety Engineering & Evaluation within the Human-Centered... ...models, assessment workflows, and measurement capabilities... ...complex production-quality systems.8+ years of... ...-based evaluation, agentic systems, model...QualityRelocation$100k
Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance... ...clear decisions, manual workflows into scalable automations... ...and business logic into agentic solutions.The role sits... ...workflows, improve data quality, and deliver executive-ready...QualityPermanent employment$187.5k - $253.5k
...Role We're hiring an AI Web Platform Lead to modernize how our... ...answer engines, to CMS workflows that move ideas from... ...rollout Actively evaluate CMS platforms and adjacent... ...authoring, agentic pipelines — and strong... ...accelerates vs. degrades quality ~ Working knowledge...QualityFull time$206.2k - $356.4k
...the ground up. As part of the Agentic Search Evaluation team, you will help advance... ...for building scalable, high-quality solutions to evaluate... ...product improvements. You will lead a team focused on evaluating... ...that power personalized Siri AI and enable answers to users'...QualityRelocation- ...technical point of contact, leading and mentoring a... ...to onboard new AI agents and standardize model evaluation processes. ML/GenAI... ...optimize performance. Agentic Evaluation Infrastructure & Quality: Configure isolated/... ...and maintain CI/CD workflows for safe, automated...QualityFull time
$197.3k - $225.1k
AI Engineer 4 (Gen AI Platform Services: Agentic AI, Agent Guardrails, Agent Evaluation, Agent Memory) At Capital One, we are... ...deliver our industry leading capabilities with... ...agents and multi-agent workflows, similarity search,... ...take pride in the quality of your work, and...QualityFull timePart timeLocal area$229.9k - $262.4k
Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating... ...deliver our industry leading capabilities with... ...systems, take pride in the quality of your work, and... ...systems and agentic workflows ~ Passion for staying...QualityFull timePart timeLocal area- ...Position Overview: The Agentic AI Engineer will design, develop... ...focuses on building autonomous AI workflows, integrating large language... .... Understanding of AI evaluation, observability, guardrails,... ..., reliability, and response quality. Develop security controls...QualityH1bLocal area
- ...builds the world's largest AI chip, 56 times larger... ...to deliver industry-leading training and inference... ...intelligence via additional agentic computation.Cerebras... ...performance, and operational quality of the systems Cerebras... ...alerting, and on-call workflows. Run incident response...Quality
$207k - $300k
...Software Engineer, Eval Infra TL, Quality Platform, Gemini Cloud Operator, Agentic AI Google's software engineers... ...interactions, implementing real-time evaluations to detect successes and failures,... ...contribute to making Gemini the leading model for cloud operational tasks...Quality$175.5k - $311.7k
...better than we found it.We are looking for an Evaluation Science Lead to define how our multilingual Globalization AI solutions are measured and validated across Apple... ...individual contributor on the Globalization Quality and Operations team, you will design statistically...QualityRelocationAfternoon shift$220k - $230k
...Title of position: Innovation Lead - Office of the CTO Position... ...(EXTR) is a global leader in AI-powered cloud networking, committed... ...& Development Identify and evaluate emerging opportunities in ML/... ...exist Professional Qualities Ambitious and driven, thriving...QualityFull timeH1bWork at officeLocal areaRemote workWork from homeWorldwideWork visaFlexible hours$31.83 - $63.65 per hour
...Job Description Leads complex, data center hardware support services... ..., and automate repeatable workflows. Mentors and develops team... ...forward plans, and ensures high-quality documentation of resolutions within... ...to life-saving care. And with AI embedded across our products...QualityHourly payTemporary workLocal areaFlexible hoursShift work$120k - $150k
...enterprise applications and AI systems. Founded by the... ...mission-critical workflows and AI agents at massive... ...complex deterministic and agentic workflows with... ...platform through high-quality technical education, community... ...build, troubleshoot, and evaluate new technologies ~...QualityWork at office$224k - $356.5k
...Automated Synthetic Chemistry Science Lead who can coordinate science... ...analytical characterization, AI modeling, laboratory... ...without compromising scientific quality.What you'll be doing:Coordinate... ...development, AI enabled science workflows, or equivalent experience.Deep...QualityFull time$224k - $356.5k
NVIDIA seeks an AI enabled drug discovery science leader who can... ...into executable plans.Lead NVIDIA workstreams spanning data... ...risk and dependency tracking, quality thresholds, and paths for addressing... ...lineage, compute planning, evaluation metrics, and closed-loop experimentation...QualityFull time- ...discovery to powering AI and the technologies people... ..., or bringing leading edge products to market... ...and Validation teams to evaluate and optimize the power... ...delivering timely, high-quality products. A broad and thorough... ...tools and improve the workflow. Deliver high-...QualityFull time
$175.5k - $263.8k
...changes across teams, and build AI tools and automation that... ...build AI-assisted tools and agentic workflows that make triage faster and... ...experience in software development, quality, or reliability... ...Understanding of how ML models are evaluated and deployed, and how they...QualityRelocation
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Quality Lead, Agentic AI Workflow Evaluation [Remote]. Be the first to apply!
- supplier quality San Jose, CA
- quality assurance quality control engineer San Jose, CA
- rn quality San Jose, CA
- quality reviewer San Jose, CA
- quality San Jose, CA
- quality manufacturing San Jose, CA
- quality inspection San Jose, CA
- quality equipment San Jose, CA
- quality improvement rn San Jose, CA
- quality intern San Jose, CA






