AI Quality Analyst

AI Quality Analyst

This is a remote position. AI Quality Analyst Reporting to Manager, Quality Engineering & AI Validation, focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. Builds benchmark scenarios, defines scoring rubrics, evaluates business usefulness, and identifies failure patterns that conventional pass or fail software testing would not catch. Key Responsibilities AI Output Evaluation · Design and execute structured evaluations for AI-enabled features and workflows. · Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk. · Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations. Benchmark & Rubric Development · Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards. · Develop rubrics that allow quality to be measured consistently across releases and changes. Workflow & Model Change Validation · Compare performance across prompt versions, workflow revisions, tools, and models. · Support release decisions with evidence on quality regression or improvement. Business & Domain Partnership · Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts. · Help define human-review thresholds and escalation patterns for higher-risk use cases. Production Feedback · Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time. Requirements Required Qualifications · 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles. · Strong writing, analysis, and pattern-recognition skills. · Experience evaluating outputs against nuanced criteria rather than only binary correctness. · Ability to work with structured rubrics, scenario libraries, and evidence-based reviews. · Comfort collaborating across Engineering and business teams. · Experience with finance, accounting, FP&A, transaction services, or business process design preferred. · Bachelor's degree preferred. You Are · Thoughtful, precise, and highly discerning. · Strong at spotting subtle output problems others miss. · Comfortable with ambiguity but disciplined in scoring and documentation. · Focused on trust, usefulness, and business reality. Benefits Salary plus performance-based bonus. Actual compensation packages are determined by evaluating a wide array of factors unique to each candidate, including but not limited to skill set, years and depth of experience, education, certifications, cost of labor, and internal equity.