QA Analyst - LOIS

QA Analyst - LOIS

We are seeking a QA Analyst for our LOIS for Word project. In this role, you will define and own the test strategy, process, and QA playbook while collaborating with product and engineering teams to enhance the quality of AI outputs. You will evaluate accuracy, citations, and reasoning in AI-generated content, and design evaluation sets for non-deterministic outputs. Additionally, you will implement automated guardrails within CI/CD for critical flows.

Key Responsibilities

• Define and own the test strategy and QA playbook.
• Partner with product and engineering early to move QA upstream.
• Evaluate AI output quality, focusing on accuracy and reasoning.
• Design evaluation sets for non-deterministic outputs and regressions.
• Build automated guardrails integrated into CI/CD processes.

Required Skills & Qualifications

• 2+ years of QA experience with ownership of test strategy.
• Proven track record in building QA processes and automation.
• Strong analytical and problem-solving skills.
• Ability to reason about non-deterministic systems.
• Excellent written communication skills for bug reporting.