Senior Automation Test Engineer

Senior Automation Test Engineer

Our client is a fast-growing healthcare technology company building AI voice agents that answer every call for physician practices — scheduling appointments, routing patients, and resolving routine requests around the clock. They are looking for a Senior Automation Test Engineer who can plug in and build rapidly, owning the test strategy and automation infrastructure for their voice AI platform and building the frameworks that make non-deterministic AI behavior measurable and regressions obvious.

Responsibilities

• Design, build, and maintain automated test infrastructure across unit, integration, end-to-end, and load/performance layers for a real-time voice AI platform
• Build evaluation and regression harnesses that score conversation outcomes and intent fulfillment for non-deterministic, LLM-driven agent behavior
• Generate synthetic conversation data and de-identified patient scenarios to expand test coverage for rare edge cases
• Develop test frameworks and tooling that enable the broader engineering team to write and run reliable tests efficiently
• Define and enforce quality standards and testing best practices across engineering teams
• Build simulation and regression frameworks for real-time voice and conversational AI scenarios, including heavy rules-based logic layered on top of nondeterministic AI behavior
• Own test environments and CI/CD pipeline integration for automated test execution
• Investigate and triage production quality issues, driving root cause analysis and prevention
• Develop testing approaches that satisfy HIPAA and healthcare compliance requirements
• Ramp quickly: within the first 30 days, understand the tech stack and business, and begin shaping a test suite strategy for the platform

Skills/Requirements

• 10+ years in software quality engineering with a strong focus on test automation
• Proven experience designing and owning test automation frameworks from the ground up
• Proficiency in Python, TypeScript, or equivalent, with strong scripting and tooling skills
• Experience with CI/CD integration (GitHub Actions, Jenkins, or equivalent)
• Hands-on AI evaluation experience — testing AI-powered applications, including prompt and model regression testing
• Demonstrated ability to test non-deterministic conversational flows, designing assertions and evaluation criteria that tolerate variation without masking real regressions
• Experience generating synthetic data at scale — realistic conversation transcripts, edge-case inputs, and de-identified patient scenarios
• Ability to move quickly and independently in a fast-scaling environment with minimal ramp-up time
• Nice to have: experience with real-time systems, telephony, or voice platforms; LLM evaluation tooling and prompt/model versioning; evaluating speech-to-text or text-to-speech quality; load/performance testing for high-throughput systems; HIPAA-regulated environment experience; contract testing or service virtualization