AI Evaluation Engineer (QA)

Responsibilities

• Run human and automated evaluations across large question sets, including hundreds of thousands or millions of evaluations.
• Statistically measure factual grounding and before-and-after accuracy lift.
• Build metrics frameworks that quantify answer quality improvements.
• Design load and quality tests as the corpus scales.
• Define and maintain test plans, test cases, and quality gates.
• Automate regression and evaluation suites and integrate them into CI/CD.
• Report quality metrics to technical and non-technical stakeholders.
• Collaborate with engineering to reproduce, triage, and verify fixes.
• Continuously improve QA processes and test coverage.

Requirements

• Bachelor’s degree in a technical field or equivalent experience.
• 4+ years of experience in QA or test engineering with exposure to data or ML systems.
• Strong Python and data-science techniques for measuring factual grounding and answer quality.
• Experience with LLM evaluation frameworks and statistical analysis.
• Ability to build automated, large-scale evaluation harnesses and human-in-the-loop A/B tests.
• Experience working across outputs from multiple LLM providers.
• Experience with test automation frameworks and scripting.
• Strong analytical, problem-solving, communication, and collaboration skills.
• Prior consulting experience.
• Prior Life Sciences industry experience and connections are preferred.

Benefits

• Equal opportunity employer with a commitment to diversity, equity, and inclusion.
• Accommodations are available upon request throughout the recruitment process.