Quality Lead, Agentic AI Workflow Evaluation [Remote]
$75 - $85 per hourInnodata Inc.
- Remote job
Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.
Scope of the Role:
We are standing up a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. Reviewers work through ambiguous, multi-step scenarios inside isolated test environments, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. The Quality Lead is the person accountable for whether that output is any good.
This is a senior individual contributor role. You will not manage the reviewers — that sits with the Engagement Manager — but you set the standard they are held to. You own the audit sample, run calibration, keep the rubric usable as real cases stress it, and train reviewers into the work. You are also the deputy: when the Engagement Manager is out, the engagement runs on you.
The quality approach here is not fully defined. We expect you to build it in partnership with the customer's quality leads, or at minimum to take what they have, run it honestly, and come back with specific recommendations for where it falls short.
What You’ll Own:
- Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported
- Build that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows
- Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes
- Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases
- Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer
- Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready
- Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working
- Report quality trends to the Engagement Manager and, alongside them, to the customer
- Deputize for the Engagement Manager on delivery operations during absences
- Maintain information security, privacy, and facility access practices required by the customer's onsite environment
You’ll Thrive in This Role If You Have:
- Bachelor's degree or equivalent practical experience
- 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain
- Direct experience owning a quality function: you designed the audit, not just executed someone else’s
- Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review
- Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes
- Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better
- Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it
- Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces)
- Experience training or onboarding reviewers into rubric-based work
The expected hourly salary range for this position is $75-85 p/hour, based on experience, skills, and qualifications.
Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at
If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at View email address on job-boards.greenhouse.io and consider reporting it to the FTC at ReportFraud.ftc.gov .
$107.5k - $188.4k
...the product capabilities that help ensure AI-enabled healthcare experiences are... ...every day. As AI becomes part of healthcare workflows, quality cannot be treated as a one-time launch checkpoint. AI experiences need to be evaluated, monitored, and improved continuously so...QualityFull timeWork at officeRemote work$132.83k - $166.04k
...growth, we are adding a Lead, Software Development AI Practice to our team!... ...the outcomes of improved quality and a measurable increase... ...platforms, generative AI, and agentic workflows/tools. This role builds our... ...business impact. Evaluate emerging AI platforms, tools...QualityFull timeWork experience placementRemote workFlexible hours$96k - $192k
...temperature control, air quality and transportation, we... ...in 1902, we lead with purpose: enhancing... ...Carrier is seeking an Agentic AI Engineer Lead for the... ...AI agents and agentic workflows that accelerate engineering... ...governance, security, evaluation, and responsible AI practices...QualityMinimum wageFull timeTemporary workLocal areaMonday to FridayDay shiftEarly shift$132.23k - $176.31k
...trusted network for the AI‑powered world,... ...The Role The Senior Lead Marketing Data Platform... ...ready data structures, and agentic operational frameworks... ..., schema control, data quality, and lifecycle management... ...technology ecosystem. Evaluate emerging technologies and...QualityFull timeTemporary work$134.5k - $265.1k
...all increasingly powered by AI and agentic technologies. You will move... ...future-ready architectures, and leading implementation across... ...orchestration, and continuously evaluating emerging tools as the landscape... ...attention to detail and quality of work productAbility to build...QualityLocal areaVisa sponsorship$50 - $55 per hour
...and Artificial Intelligence (AI) are inextricably linked.... ...intelligence by providing the data, evaluation frameworks, and human... ...legacy delivering the highest quality data and outstanding... ...evaluate complex, real-world agentic AI workflows, assessing whether AI agents...QualityRemote jobHourly payFixed term contract- ...across a business—until generative AI. Today, AI is the number one... ...shift to the New using leading-edge technologies on some of the... ...you can imagine.Join our Global Agentic Center of Excellence and help... ...and production, and ensuring quality, value, and reliability of deployed...QualityWork experience placementLive inWork at officeLocal areaShift work
$105.79k - $141.05k
...trusted network for the AI‑powered world,... ...We are seeking a Senior Lead Technical Marketing Manager... ...seller confidence, solution quality, and conversion.... ...integrations, and operational workflows. Proven ability to... ...Background results will be evaluated on a case-by-case basis...QualityTemporary workRemote workWork from home$107.5k - $188.4k
The Lead Product Manager, Healthcare Analytics... ...on identifying, evaluating, and developing investment... ...of customer workflows, identifies meaningful... ...to embrace AI-enabled ways of working and the Agentic Product Development... ...adoption, product quality, learning velocity,...QualityFull timeWork at office$141.2k - $278.3k
Position Summary Lead Solution Architect - AI & Engineering/EaasJoin Deloitte’s AI & Engineering... ...priorities, risks, dependencies, and quality throughout the delivery lifecycleDriving... ...environmentsExperience with model evaluation, observability, guardrails, and security...QualityLocal areaFlexible hours$105.79k - $141.05k
...trusted network for the AI‑powered world,... ...The GTM Growth Strategy Lead develops and implements... ...audience selection, signal quality, recommendation logic,... ...Familiarity with CRM, seller workflows, revenue intelligence,... ...results will be evaluated on a case-by-case basis...QualityFull timeTemporary work$148.19k - $198.17k
...Salesforce Salesforce is the #1 AI CRM, where humans with... ...career at the company leading workforce transformation in the agentic era? You’re in the right... ...work, dev lifecycle and quality practices. AI Governance... ...recruiters assess and evaluate candidates’ resumes and qualifications...QualityFull time- ...Technical Product Manager - Agentic AI Framework to join our... ...ready for execution.Lead Program Increment (PI)... ...-driven multi-agent workflows adopted across the... ...orchestration engine - evaluating frameworks (LangGraph,... ...teams to construct high-quality, well designed technical...QualityFull time
- ...North America portfolio includes leading brands like AgriGold®,... ...GDMSeeds. Job Summary: The Quality Management Lead provides... ...winter grow-out programs and evaluate resulting performance data.... ...reports. Oversee the daily workflow and performance of the Quality...QualityTemporary workSummer workWork at officeFlexible hoursNight shift
- Job Description Senior Agentic Software Engineer \n... ...hands-on expertise in AI-assisted software development... ..., and deliver high-quality software solutions. \n... ...existing software engineering workflows. \n Leverage Claude... ...automation. \n Evaluate and refine AI-generated...QualityFull timeContract workImmediate startHome office
$134.5k - $265.1k
...Summary The Cloud Platform Lead designs, builds, and operates... ...teams, and applies to AI both to improve platform operations... ...interfaces, and event-driven workflows.Operate Kubernetes and container... ...attention to detail and quality of work productAbility to build...QualityWork at officeLocal areaVisa sponsorshipNight shift1 day per week$21.96 - $36.6 per hour
...products, and services that make quality care more accessible and... ...we want to hear from you. The Lead Pharmacy Technician supports... ...pharmacy operations by coordinating workflow, assisting with work... ..., equity, regular job market evaluations, and geographical markets. The...QualityWork at officeFlexible hoursShift work- Quality Lead-Receiving InspectionTrelleborg is a world leader in engineered polymer solutions... ...behavior. Plans trains and motivates, evaluates performance of Inspectors.Tasks and... ...resource functionsAssigns and prioritizes workflow for the Receiving Inspection department...QualityPermanent employmentWork at officeLocal areaFlexible hours
- Lead Respiratory TherapistThe Lead Respiratory Therapist monitors shift operations... ...a patient-oriented environment and workflow conducive to quality patient care delivery.Who We... ...unit team members such as performance evaluations, disciplinary actions, and interviews...QualityLocal areaShift work
- ...Team Lead HRforGrowth is engaged as the exclusive talent acquisition... .... HRforGrowth identifies evaluates and presents top-tier... ...understanding the impact of workflow on productions by meeting performance... ...~ Comply with all safety and quality policies and conduct audits...QualityHourly payPermanent employmentTemporary workLocal area
- ...Description The Inbound Team Lead contributes to meeting... ...exceptional customer service, quality, process improvement, and profitability... .... Performance will be evaluated based on meeting the... ...assign workloads and expedite workflow Ensure quality checks are...QualityDay shift
- ...Clinical AI Strategist (Pharmacy) LOCATION... ...Health Technology (PHT) Agentic PDLC product pod that... ...pharmacist workflows, clinical decision points... ...logic, domain-specific quality standards, evaluation rubrics, and AI evals... ...with Product Managers, Lead Builder, Builders, Product...QualityFull timeWork at office
- ...accelerate their adoption of Agentic AI using Salesforce Agentforce and... ..., design intelligent agent workflows, and build and validate Agentforce... ...Agentforce use cases.Lead discovery workshops to understand... ...solutions.Establish testing, evaluation, monitoring, and continuous-...Work experience placementLive inWork at officeLocal area
$94.4k - $266.3k
...’s voice and purpose. We help leading brands create and deliver the... ...processes, drives data model and workflow design, and ensures that the... ...on industry trends related to Agentic Commerce and leverage of GenAI... ...implementationsExperience with GenAI or Agentic AI integration within PIM...Live inWork at officeLocal area- ...Lead Cardiology Technician The Lead Cardiology Technician oversees... .... This position ensures high-quality patient care, adherence to... ...and ensure efficient workflow. Prepare and coordinate staff... ...Care and Conduct and will be evaluated according to these standards...Quality
- ...global leader in AI and cloud transformation... .... Our work spans agentic AI architecture,... ...As a Manager, you lead solution design... ...planning, execution, evaluation, and coordination... ...does not. Own AI quality and observability:... ...augmented development workflows hands-on, and set...QualityWork experience placementLive inWork at officeLocal area
- ...customer experiences.As AI agents increasingly... ...consumers discover, evaluate, and purchase products, the quality and structure of... ...WHAT'S IN IT FOR YOU?Lead the product data foundation that powers Agentic Commerce transformations... ..., enrichment workflow design, attribute schema...QualityLive inWork at officeLocal area
- ...we’re building the future of AI-powered Content Operations. Our... ...Finance team as the AI & Innovation Lead, G&A, reporting to the CFO.... ...of G&A operations, data, workflow automation, and AI. You bring... ...trails, access management, data quality, and human-in-the-loop review,...QualityFull timeContract workRemote workDay shift
$141.2k - $278.3k
...ERP, MES, PLM, OT), and AI agents on a unified... ...6. Work you’ll do The Lead Architect owns the technical... ...engines, the pre-built agentic actions that do the... ...raising the bar on technical quality and on how efficiently... ...of experience using or evaluating agentic systems in...QualityWork experience placementLocal area$29 - $32 per hour
...Job Title: Lead Lab Technician Job Description... ...ensure compliance with quality and safety standards, and... ...project metrics. Identify workflow inefficiencies and... ...Artificial Intelligence (AI): We may use... ...sourcing, screening, and evaluating candidates. AI helps assess...QualityContract workTemporary workShift workNight shiftWeekend workAfternoon shift
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Quality Lead, Agentic AI Workflow Evaluation [Remote]. Be the first to apply!






