Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Quality Lead, Agentic AI Workflow Evaluation [Remote]

$75 - $85 per hour

Innodata Inc.

Indiana
  • Remote job

Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.

Scope of the Role: 

We are standing up a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. Reviewers work through ambiguous, multi-step scenarios inside isolated test environments, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. The Quality Lead is the person accountable for whether that output is any good.  

This is a senior individual contributor role. You will not manage the reviewers — that sits with the Engagement Manager — but you set the standard they are held to. You own the audit sample, run calibration, keep the rubric usable as real cases stress it, and train reviewers into the work. You are also the deputy: when the Engagement Manager is out, the engagement runs on you.  

The quality approach here is not fully defined. We expect you to build it in partnership with the customer's quality leads, or at minimum to take what they have, run it honestly, and come back with specific recommendations for where it falls short.  

What You’ll Own:

  • Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported  
  • Build that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows  
  • Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes  
  • Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases  
  • Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer  
  • Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready  
  • Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working  
  • Report quality trends to the Engagement Manager and, alongside them, to the customer  
  • Deputize for the Engagement Manager on delivery operations during absences  
  • Maintain information security, privacy, and facility access practices required by the customer's onsite environment  

You’ll Thrive in This Role If You Have:

  • Bachelor's degree or equivalent practical experience  
  • 4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain  
  • Direct experience owning a quality function: you designed the audit, not just executed someone else’s  
  • Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review  
  • Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes  
  • Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better  
  • Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it  
  • Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces) 
  • Experience training or onboarding reviewers into rubric-based work  

The expected hourly salary range for this position is $75-85 p/hour, based on experience, skills, and qualifications.

 

Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at  

If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at  View email address on job-boards.greenhouse.io and consider reporting it to the FTC at  ReportFraud.ftc.gov .

Vacancy posted 6 days ago
Similar jobs that could be interesting for youBased on the Quality Lead, Agentic AI Workflow Evaluation [Remote] in Indiana vacancy
  • $107.5k - $188.4k

     ...the product capabilities that help ensure AI-enabled healthcare experiences are...  ...every day. As AI becomes part of healthcare workflows, quality cannot be treated as a one-time launch checkpoint. AI experiences need to be evaluated, monitored, and improved continuously so... 
    Quality
    Full time
    Work at office
    Remote work

    Wolters Kluwer

    Indianapolis, IN
    4 days ago
  • $132.83k - $166.04k

     ...growth, we are adding a Lead, Software Development AI Practice to our team!...  ...the outcomes of improved quality and a measurable increase...  ...platforms, generative AI, and agentic workflows/tools. This role builds our...  ...business impact. Evaluate emerging AI platforms, tools... 
    Quality
    Full time
    Work experience placement
    Remote work
    Flexible hours

    Navitus Health Solutions, LLC

    Indiana
    12 days ago
  • $96k - $192k

     ...temperature control, air quality and transportation, we...  ...in 1902, we lead with purpose: enhancing...  ...Carrier is seeking an Agentic AI Engineer Lead for the...  ...AI agents and agentic workflows that accelerate engineering...  ...governance, security, evaluation, and responsible AI practices... 
    Quality
    Minimum wage
    Full time
    Temporary work
    Local area
    Monday to Friday
    Day shift
    Early shift

    Carrier

    Indianapolis, IN
    14 hours ago
  • $132.23k - $176.31k

     ...trusted network for the AI‑powered world,...  ...The Role The Senior Lead Marketing Data Platform...  ...ready data structures, and agentic operational frameworks...  ..., schema control, data quality, and lifecycle management...  ...technology ecosystem. Evaluate emerging technologies and... 
    Quality
    Full time
    Temporary work

    Lumen

    Fort Wayne, IN
    3 days ago
  • $134.5k - $265.1k

     ...all increasingly powered by AI and agentic technologies. You will move...  ...future-ready architectures, and leading implementation across...  ...orchestration, and continuously evaluating emerging tools as the landscape...  ...attention to detail and quality of work productAbility to build... 
    Quality
    Local area
    Visa sponsorship

    Deloitte

    Indianapolis, IN
    4 days ago
  • $50 - $55 per hour

     ...and Artificial Intelligence (AI) are inextricably linked....  ...intelligence by providing the data, evaluation frameworks, and human...  ...legacy delivering the highest quality data and outstanding...  ...evaluate complex, real-world agentic AI workflows, assessing whether AI agents... 
    Quality
    Remote job
    Hourly pay
    Fixed term contract

    Innodata Inc.

    Indiana
    6 days ago
  •  ...across a business—until generative AI. Today, AI is the number one...  ...shift to the New using leading-edge technologies on some of the...  ...you can imagine.Join our Global Agentic Center of Excellence and help...  ...and production, and ensuring quality, value, and reliability of deployed... 
    Quality
    Work experience placement
    Live in
    Work at office
    Local area
    Shift work

    Accenture

    Carmel, IN
    2 days ago
  • $105.79k - $141.05k

     ...trusted network for the AI‑powered world,...  ...We are seeking a Senior Lead Technical Marketing Manager...  ...seller confidence, solution quality, and conversion....  ...integrations, and operational workflows. Proven ability to...  ...Background results will be evaluated on a case-by-case basis... 
    Quality
    Temporary work
    Remote work
    Work from home

    Lumen Inc

    Indianapolis, IN
    2 days ago
  • $107.5k - $188.4k

    The Lead Product Manager, Healthcare Analytics...  ...on identifying, evaluating, and developing investment...  ...of customer workflows, identifies meaningful...  ...to embrace AI-enabled ways of working and the Agentic Product Development...  ...adoption, product quality, learning velocity,... 
    Quality
    Full time
    Work at office

    Wolters Kluwer

    Indianapolis, IN
    1 day ago
  • $141.2k - $278.3k

    Position Summary Lead Solution Architect - AI & Engineering/EaasJoin Deloitte’s AI & Engineering...  ...priorities, risks, dependencies, and quality throughout the delivery lifecycleDriving...  ...environmentsExperience with model evaluation, observability, guardrails, and security... 
    Quality
    Local area
    Flexible hours

    Deloitte

    Indianapolis, IN
    2 days ago
  • $105.79k - $141.05k

     ...trusted network for the AI‑powered world,...  ...The GTM Growth Strategy Lead develops and implements...  ...audience selection, signal quality, recommendation logic,...  ...Familiarity with CRM, seller workflows, revenue intelligence,...  ...results will be evaluated on a case-by-case basis... 
    Quality
    Full time
    Temporary work

    Lumen

    Terre Haute, IN
    2 days ago
  • $148.19k - $198.17k

     ...Salesforce Salesforce is the #1 AI CRM, where humans with...  ...career at the company leading workforce transformation in the agentic era? You’re in the right...  ...work, dev lifecycle and quality practices. AI Governance...  ...recruiters assess and evaluate candidates’ resumes and qualifications... 
    Quality
    Full time

    Salesforce

    Indianapolis, IN
    3 days ago
  •  ...Technical Product Manager - Agentic AI Framework to join our...  ...ready for execution.Lead Program Increment (PI)...  ...-driven multi-agent workflows adopted across the...  ...orchestration engine - evaluating frameworks (LangGraph,...  ...teams to construct high-quality, well designed technical... 
    Quality
    Full time

    Cummins

    Columbus, IN
    2 days ago
  •  ...North America portfolio includes leading brands like AgriGold®,...  ...GDMSeeds. Job Summary: The Quality Management Lead provides...  ...winter grow-out programs and evaluate resulting performance data....  ...reports. Oversee the daily workflow and performance of the Quality... 
    Quality
    Temporary work
    Summer work
    Work at office
    Flexible hours
    Night shift

    GDM Seeds

    Westfield, IN
    17 hours ago
  • Job Description Senior Agentic Software Engineer \n...  ...hands-on expertise in AI-assisted software development...  ..., and deliver high-quality software solutions. \n...  ...existing software engineering workflows. \n Leverage Claude...  ...automation. \n Evaluate and refine AI-generated... 
    Quality
    Full time
    Contract work
    Immediate start
    Home office

    Energy Jobline ZR

    Indianapolis, IN
    3 days ago
  • $134.5k - $265.1k

     ...Summary The Cloud Platform Lead designs, builds, and operates...  ...teams, and applies to AI both to improve platform operations...  ...interfaces, and event-driven workflows.Operate Kubernetes and container...  ...attention to detail and quality of work productAbility to build... 
    Quality
    Work at office
    Local area
    Visa sponsorship
    Night shift
    1 day per week

    Deloitte

    Indianapolis, IN
    2 days ago
  • $21.96 - $36.6 per hour

     ...products, and services that make quality care more accessible and...  ...we want to hear from you. The Lead Pharmacy Technician supports...  ...pharmacy operations by coordinating workflow, assisting with work...  ..., equity, regular job market evaluations, and geographical markets. The... 
    Quality
    Work at office
    Flexible hours
    Shift work

    Jobtome

    Indianapolis, IN
    3 days ago
  • Quality Lead-Receiving InspectionTrelleborg is a world leader in engineered polymer solutions...  ...behavior. Plans trains and motivates, evaluates performance of Inspectors.Tasks and...  ...resource functionsAssigns and prioritizes workflow for the Receiving Inspection department... 
    Quality
    Permanent employment
    Work at office
    Local area
    Flexible hours

    Trelleborg

    Fort Wayne, IN
    3 days ago
  • Lead Respiratory TherapistThe Lead Respiratory Therapist monitors shift operations...  ...a patient-oriented environment and workflow conducive to quality patient care delivery.Who We...  ...unit team members such as performance evaluations, disciplinary actions, and interviews... 
    Quality
    Local area
    Shift work

    Franciscan Alliance

    Munster, IN
    2 days ago
  •  ...Team Lead HRforGrowth is engaged as the exclusive talent acquisition...  .... HRforGrowth identifies evaluates and presents top-tier...  ...understanding the impact of workflow on productions by meeting performance...  ...~ Comply with all safety and quality policies and conduct audits... 
    Quality
    Hourly pay
    Permanent employment
    Temporary work
    Local area

    Advatix

    South Bend, IN
    7 days ago
  •  ...Description The Inbound Team Lead contributes to meeting...  ...exceptional customer service, quality, process improvement, and profitability...  .... Performance will be evaluated based on meeting the...  ...assign workloads and expedite workflow Ensure quality checks are... 
    Quality
    Day shift

    Langham Logistics, Inc

    Whiteland, IN
    5 days ago
  •  ...Clinical AI Strategist (Pharmacy) LOCATION...  ...Health Technology (PHT) Agentic PDLC product pod that...  ...pharmacist workflows, clinical decision points...  ...logic, domain-specific quality standards, evaluation rubrics, and AI evals...  ...with Product Managers, Lead Builder, Builders, Product... 
    Quality
    Full time
    Work at office

    Wolters Kluwer

    Indianapolis, IN
    17 hours ago
  •  ...accelerate their adoption of Agentic AI using Salesforce Agentforce and...  ..., design intelligent agent workflows, and build and validate Agentforce...  ...Agentforce use cases.Lead discovery workshops to understand...  ...solutions.Establish testing, evaluation, monitoring, and continuous-... 
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Carmel, IN
    1 day ago
  • $94.4k - $266.3k

     ...’s voice and purpose. We help leading brands create and deliver the...  ...processes, drives data model and workflow design, and ensures that the...  ...on industry trends related to Agentic Commerce and leverage of GenAI...  ...implementationsExperience with GenAI or Agentic AI integration within PIM... 
    Live in
    Work at office
    Local area

    Accenture

    Carmel, IN
    1 day ago
  •  ...Lead Cardiology Technician The Lead Cardiology Technician oversees...  .... This position ensures high-quality patient care, adherence to...  ...and ensure efficient workflow. Prepare and coordinate staff...  ...Care and Conduct and will be evaluated according to these standards... 
    Quality

    Methodist Hospitals

    Merrillville, IN
    3 days ago
  •  ...global leader in AI and cloud transformation...  .... Our work spans agentic AI architecture,...  ...As a Manager, you lead solution design...  ...planning, execution, evaluation, and coordination...  ...does not. Own AI quality and observability:...  ...augmented development workflows hands-on, and set... 
    Quality
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Carmel, IN
    17 hours ago
  •  ...customer experiences.As AI agents increasingly...  ...consumers discover, evaluate, and purchase products, the quality and structure of...  ...WHAT'S IN IT FOR YOU?Lead the product data foundation that powers Agentic Commerce transformations...  ..., enrichment workflow design, attribute schema... 
    Quality
    Live in
    Work at office
    Local area

    Accenture

    Carmel, IN
    1 day ago
  •  ...we’re building the future of AI-powered Content Operations. Our...  ...Finance team as the AI & Innovation Lead, G&A, reporting to the CFO....  ...of G&A operations, data, workflow automation, and AI. You bring...  ...trails, access management, data quality, and human-in-the-loop review,... 
    Quality
    Full time
    Contract work
    Remote work
    Day shift

    Sanity

    Indiana
    23 days ago
  • $141.2k - $278.3k

     ...ERP, MES, PLM, OT), and AI agents on a unified...  ...6. Work you’ll do The Lead Architect owns the technical...  ...engines, the pre-built agentic actions that do the...  ...raising the bar on technical quality and on how efficiently...  ...of experience using or evaluating agentic systems in... 
    Quality
    Work experience placement
    Local area

    Deloitte

    Indianapolis, IN
    2 days ago
  • $29 - $32 per hour

     ...Job Title: Lead Lab Technician Job Description...  ...ensure compliance with quality and safety standards, and...  ...project metrics. Identify workflow inefficiencies and...  ...Artificial Intelligence (AI): We may use...  ...sourcing, screening, and evaluating candidates. AI helps assess... 
    Quality
    Contract work
    Temporary work
    Shift work
    Night shift
    Weekend work
    Afternoon shift

    Actalent

    Indianapolis, IN
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Quality Lead, Agentic AI Workflow Evaluation [Remote]. Be the first to apply!