Quality Analyst — AI Annotation Audit & Label Consistency Review
$20 - $50 per hourAuraOne Human Data
Quality Analyst --- AI Annotation Audit & Label Consistency Review is a remote review track for evaluating AI outputs across quality analyst specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next step so the modeling team can train on it. Why this role matters Quality Analyst specialist operations AI has to fit into an actual day at work. AuraOne uses experienced operators to grade outputs the way a senior peer would --- checking workflow, policy, and the unwritten rules that decide whether a task actually gets done. Responsibilities Review AI outputs against current quality analyst specialist operations workflows, playbooks, and firm policy for Quality Analyst --- AI Annotation Audit & Label Consistency Review assignments. Grade tone, escalation logic, and stakeholder fit on a structured rubric. Flag operational risk, missed escalations, and policy-adherence gaps with severity tags. Capture the right next step so the modeling team can train on it. Adjudicate disputed treatments against published playbooks or firm guidance. Maintain reviewer-quality scores in inter-rater calibration cycles. Qualifications Direct working experience in quality analyst specialist operations on real teams for Quality Analyst --- AI Annotation Audit & Label Consistency Review work. Comfort applying multi-page rubrics consistently across long batches. Clear written reasoning that names the policy or workflow being applied. Strong attention to detail and the ability to flag when a prompt itself is the problem. Reliable async availability for at least 10 hours per week. Example tasks Grade a model's response to a real quality analyst specialist operations ticket and rate workflow, tone, and escalation. Flag a missed escalation with the right severity tag and corrected next step. Adjudicate a disputed playbook call between two reviewers using firm guidance. Audit a 25-row batch for rubric consistency and report drift to the program lead. Nice to have Prior experience training, calibrating, or QA-ing operations teams. Familiarity with AI-assisted workflow tooling and its failure modes. Bilingual experience for cross-region operations. Skills Operational review Policy adherence Workflow judgment Stakeholder communication Quality Analyst specialist operations Work model Remote --- US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US. Compensation $20--$50 / hr #J-18808-Ljbffr AuraOne Human Data
- AuraOne Human Data is seeking a Quality Analyst for AI Annotation Audit & Label Consistency Review. This remote contractor role evaluates AI outputs across workflows, grades workflow correctness, policy adherence, and stakeholder fit, and documents next steps to train...SuggestedRemote jobFor contractors10 hours per week
$38 per hour
...our team as Data Quality Analysts. This role sits... ...bridging Data Labeling Associates (... ...both people and AI systems — auditing outputs, supporting... ...accuracy and consistency Act as the... ...experience in data annotation, content... ...experience auditing or reviewing the work of...SuggestedFull timeContract workRemote workVisa sponsorshipWelo Data
New York, NYmore than 2 months ago- AI Trainer Jobs seeks a Valuation Research Analyst to remotely review AI outputs across valuation workflows. You will grade calculations, narrative reasoning, and... ..., and the ability to apply multi-page rubrics consistently with clear reasoning. Availability of at least...SuggestedRemote job
$38 per hour
...Judgment and AI Excellence — NYC... ...seeking Data Quality Analysts for Project Perseus... ...'t just about labeling; you are... ...-to-Human Auditing: Audit the work... ...years in data annotation, QA, content quality... ...auditing or reviewing the work of... ...ability to stay consistent while...SuggestedHourly payFull timeWork at officeShift work- AuraOne is seeking Analysts/Insights Experts to remotely review AI outputs across specialist operations. You will grade workflow correctness, policy adherence, and stakeholder fit, flag operational risk, and document the next steps for training the modeling team. The role...SuggestedRemote jobFor contractorsFlexible hours
- ...speaking professionals for a short-term AI quality evaluation project. You will create... ...Responsibilities include creating and reviewing prompts, assessing multi-turn interactions... ...supported; prior AI evaluation or data annotation experience is preferred. #J-18808-Ljbffr...Remote jobTemporary work
- Eucalyptus is seeking a licensed clinician to perform clinical audits and improve patient safety across its international teams. The role... ...collaborate with teams in Australia and the Philippines, supported by AI-driven tools, while maintaining rigorous accuracy and timely...Remote job
$20 - $40 per hour
Financial Records Quality Analyst is a remote review track for evaluating AI outputs across finance and risk workflows. Reviewers... ...to grade AI outputs the way an audit reviewer would — with citations,... ...applying multi-page rubrics consistently across long batches. Clear...For contractorsWork experience placementRemote work10 hours per week- ...Dane Street, LLC is hiring a Peer / Record Review Quality Analyst responsible for managing medical file reviews and ensuring client standards are met. The ideal candidate must have strong communication skills, an educational background in healthcare or business, and...Remote workWork from home
Dane Street
New York, NY17 hours ago - ...private conversations into source-backed knowledge a firm can build on. \n \n We are now hiring an AI Quality Analyst, based in New York. \n \n You will review what our AI produces, identify where it falls short and build the examples our AI team uses to measure...Immediate start
Syntrace
New York, NY4 days ago - EXL Service seeks an IRF Quality Analyst to conduct random, targeted, and follow-up reviews ensuring audit conclusions align with IRF-PAI and supporting records. You will verify documentation across medical records, therapy, physician, nursing, and orders, with focus on...Remote job
$85k - $92k
...a real impact in healthcare quality? EXL Health is looking for an... ...experienced Outpatient Quality Analyst - QC Coder to join our remote... ...of our outpatient coding audits- ensuring every audit meets EXL... ...documentation, lead quality reviews, and provide expert feedback...Remote work- ...academic programs, or those with backgrounds in teaching or education. In this role, you will review, assess, and provide structured feedback on domain-specific documents, ensuring quality, accuracy, and relevance. This position suits professionals with strong analytical...
- ...We are seeking a Freelance Apparel Quality Analyst / Data Analyst to support product quality... ...organized, with the ability to mine customer reviews, return data, and product performance... ...based on trust, respect, and the consistent delivery of excellent results. For more...Hourly payContract workTemporary workFreelanceWork at officeImmediate start3 days per week
Solomon Page
New York, NYa month ago - AI Trainer Jobs is seeking a SQL Support Analyst for a remote contractor role focused on evaluating AI outputs within sql support analyst specialist operations workflows. Reviewers assess workflow correctness, policy adherence, and stakeholder fit, flag operational risk...Remote jobFor contractors
- ...evaluation, and project-based workstreams. We are seeking an experienced Data Analyst to review large business datasets, ensure privacy and data quality, and deliver clear recommendations that support AI training and evaluation in a fully remote setting. This independent...Remote jobHourly payFor contractorsFlexible hours
$80 per hour
A forward-thinking tech company is seeking QAs for autonomous AI agents to validate complex task structures and improve evaluation frameworks... ...analytical skills and attention to detail. Applicants will review scenarios, identify inconsistencies, and help define expected...Remote jobFlexible hours- Supporting quality and compliance activities across healthcare operations for the... ...remote Quality Assurance Compliance Analyst will conduct audits, analyze claims data, and collaborate... ...Key responsibilities Conduct quality reviews and audits of healthcare claims and operational...Remote work
$80 per hour
A forward-thinking AI company is seeking Quality Analysts for autonomous AI agents. This project-based opportunity is ideal for those with strong analytical... ...and detail-oriented skills. Responsibilities include reviewing evaluation tasks, identifying inconsistencies, and...Remote jobFlexible hours$60 - $120 per hour
AI Trainer Jobs is seeking a Management Analyst for a remote review role evaluating AI outputs across operations workflows. You will grade workflow correctness, policy adherence, and stakeholder fit, flag risks, and document next steps for the modeling team to train on....Remote jobHourly payFor contractors- AuraOne is seeking a Computer Systems Analyst to remotely review AI outputs across specialist operations. You will evaluate workflow correctness, policy adherence, and stakeholder fit, flag risks, and document next steps to train the modeling team. This contractor role...Remote jobFor contractors10 hours per week
- ...Job Purpose/Overview: The Quality Analyst is responsible for monitoring... ...interaction monitoring and review of documentation to ensure adherence... ...and external client audits as directed by Program Leadership... ...to promote scoring consistency, quality standards, and continuous...Work experience placementWork at officeRemote workNight shiftAfternoon shift
Inizio Medical
New York, NY1 day ago - ...To enhance personalized AI experiences, the full-time English AI Quality Analyst will design and run multi-turn prompts, compare AI responses for quality, and provide evidence-based rationales while working remotely. Key responsibilities Design and execute short, multi...Full timeRemote work
Virtual Vocations Inc
New York, NY1 day ago $50 - $60 per hour
Manual QA Analyst (Property & Casualty Insurance)OverviewWe are seeking... ...(10%), ensuring the quality, reliability, and performance... ...Use of Artificial Intelligence (AI): We may use Artificial Intelligence... ...that your application may be reviewed using AI tools.Job SummaryJob...Contract workTemporary workTEKsystems
New York, NY1 day ago- ...traceability matrices. Participate in sprint planning, daily standups, and test review meetings. Coordinate UAT with stakeholders and provide go/no-go recommendations for production releases. Use AI tools like ChatGPT or Copilot to generate test scenarios, identify edge...Remote work
Sprintsynergy
New York, NY17 hours ago - ...MANDATORY - MUST HAVE: software automation testing and quality assurance experience Java 17 experience. Jmeter expertise in test... ...Scrum environment Experience in Java programming/scripting Experience using AI to develop and execute automated test cases...
3B Staffing LLC
New York, NY22 hours ago $20 per hour
SupportFinity™ is seeking a Social Media Content Manager to help train AI models. This role involves evaluating chatbot outputs and ensuring the quality of AI generated content. Candidates should have expertise in writing, editing, and language. The position offers flexibility...Remote jobHourly payFull timePart time$48k - $52k
...thematic and technical experts to deliver top-quality research and insights. Simply speaking,... ...to be doing: Ensuring the quality and consistency of responses collected through our... ...interviews conducted in our Retail Labs Reviewing responses to open-ended questions to identify...Work experience placementWork at officeRemote workFlexible hours$38 per hour
...Description Join the Human Intelligence Behind the World’s Leading AI – California Are you sharp, curious, and obsessed with the details? Welo Data is looking for Data Quality Analysts who can go beyond simple labels to help define how AI should behave in the real world.This...Full timeWork at office- To support quality assurance initiatives, the full-time remote Quality and Training Analyst will ensure consistent application of policies and procedures across operational departments, conduct quality audits, and contribute to training content development. Key responsibilities...Full timeRemote workShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Quality Analyst — AI Annotation Audit & Label Consistency Review. Be the first to apply!
- quality improvement analyst New York, NY
- quality analyst New York, NY
- remote call center quality analyst New York, NY
- quality tech New York, NY
- quality inspection New York, NY
- quality lead New York, NY
- air quality New York, NY
- quality New York, NY
- quality environment health safety manager New York, NY
- quality advocate New York, NY