Sr. Manager, Test Automation & AI Evaluation

Description

Position at WebMD

WebMD Ignite, a division of WebMD and Internet Brands, is the growth partner for healthcare organizations. We guide people to better health at all stages of their journey, from discovery to recovery. Our combination of leading brands in the industry—including WebMD, Medscape, Krames, PulsePoint, Vitals, The Wellness Network, Mercury Healthcare, and Healthwise—offers comprehensive solutions that engage individuals with timely, relevant messaging that optimizes experiences and outcomes, driving loyalty and lifetime value for our clients. Learn more at WebMDIgnite.com.

WebMD Ignite, a division of WebMD and Internet Brands, is the growth partner for healthcare organizations. We guide people to better health at all stages of their journey, from discovery to recovery. Our combination of leading brands in the industry — including WebMD, Medscape, Krames, PulsePoint, Vitals, The Wellness Network, Mercury Healthcare, and Healthwise — offers comprehensive solutions that engage individuals with timely, relevant messaging that optimizes experiences and outcomes, driving loyalty and lifetime value for our clients. Learn more at WebMDIgnite.com.

About the Role:

The Sr. Manager, Test Automation & AI Evaluation is an automation-first quality leader responsible for quality engineering across the Ignite Growth Platform. As engineering velocity has accelerated with AI-assisted development, quality assurance has become the rate-limiting constraint; this role exists to remove that constraint. The Sr. Manager makes automation the default, builds the evaluation discipline required to test AI-powered and agentic features, and applies AI to the practice of QA itself. This role also provides direct leadership of the existing Educate quality assurance team and serves as the head of quality assurance across Ignite, owning quality standards, test strategy, and release-readiness governance organization-wide.

Key Responsibilities:

• Lead an automation-first quality strategy, eliminating manual testing waste and making automated testing the default across teams.

• Own modern test automation frameworks and end-to-end automated coverage, with deep, current expertise in Playwright and comparable frameworks.

• Build and own evaluation frameworks for LLM-powered features and agentic systems, covering correctness, hallucination, prompt drift, and behavioral regression.

• Establish and maintain evaluation methodology such as curated golden sets, LLM-as-judge approaches calibrated against subject-matter experts, and continuous learning loops that feed production failures back into the benchmark.

• Apply AI to QA itself, including self-healing automation, AI-assisted test generation, and intelligent defect triage.

• Own standardization of quality engineering practices across the organization, driving accountability for test coverage, defect management, release readiness, and overall delivery quality.

• Provide direct management, mentorship, and structure to the existing Ignite quality assurance function.

• Support the development of a scalable Quality Management System to ensure consistency, traceability, and process rigor across teams.

• Partner with engineering, product, and DevOps to define acceptance criteria and embed quality gates into the delivery lifecycle.

Qualifications:

• 7+ years of experience in quality engineering or test automation, including team leadership or management. 

• Deep, current, hands-on expertise in modern test automation frameworks such as Playwright, Cypress, or Selenium.

• Demonstrated experience building automated test suites and integrating them into CI/CD pipelines.

• Experience designing evaluation or testing approaches for AI/ML or LLM-powered systems, or a strong data-and-metrics orientation with the ability to build one.

• Strong understanding of test strategy, defect management, and release-readiness governance across multiple teams.

• Proven ability to lead, mentor, and grow a quality engineering team.

• Excellent communication skills and the ability to translate quality risk into business terms.

Preferred Skills:

• Experience testing generative AI or agentic products, including evaluation harnesses, prompt testing, and regression testing for AI flows.

• Experience in a healthcare, health IT, or other regulated delivery environment where quality standards are elevated.

• Familiarity with quality management systems and traceability requirements.

• Experience with AI-assisted testing tooling and self-healing automation platforms.

Compensation: $98,000 - $135,000

Bonus Eligible: This position is also eligible for a discretionary company bonus, based upon business results.

Benefits: Employees in this position are eligible to participate in the company sponsored benefit programs, including the following within the first 12 months of employment:

• Health Insurance (medical, dental, and vision coverage)
• Paid Time Off (including vacation, sick leave, and flexible holiday days)
• 401(k) Retirement Plan with employer matching
• Life and Disability Insurance
• Employee Assistance Program (EAP)
• Commuter and/or Transit Benefits (if applicable)
  

Eligibility for specific benefits may vary based on job classification, schedule (e.g., full-time vs. part-time), work location and length of employment.