Automation Test Engineer

Job Description

Job Description

Calling your doctor shouldn't be harder than seeing your doctor. But at most practices, patients wait on hold, front-office staff drown in ring queues, and revenue walks out the door with every missed call.

Transform9 builds AI agents that answer every call for physician practices — scheduling appointments, routing callers, and resolving routine requests in seconds, around the clock. Our agents integrate directly with each practice's EHR, practice management, and phone systems, and adapt to how their front office actually works.

Our agents have handled millions of patient calls for thousands of providers, cut average hold times by 90%, and decreased call abandonment by 80%. We're headquartered in Birmingham, AL, and growing fast.

About the role

At Transform9, a broken call flow means a patient can't reach their doctor. Quality isn't just a process concern — it's a patient safety concern. As a senior automation testing engineer, you'll own the test strategy and automation infrastructure for a real-time voice AI platform, ensuring that every agent behavior, integration point, and call flow performs exactly as expected before it ever reaches a live patient.

This is testing where the system under test doesn't give you the same answer twice. Our agents are LLM-driven, so correctness is a range of acceptable behaviors rather than a single expected string. You'll build the frameworks that make that range measurable — and make regressions obvious.

What you'll do

- Design, build, and maintain the automated test infrastructure for Transform9's voice AI platform — unit, integration, end-to-end, and load/performance testing

- Build evaluation and regression harnesses for non-deterministic agent behavior — scoring conversation outcomes and intent fulfillment rather than exact output matching

- Generate synthetic conversation data and de-identified patient scenarios to expand coverage of edge cases that occur rarely in production

- Develop test frameworks and tooling that enable the broader engineering team to write and run reliable tests efficiently

- Define and enforce quality standards and testing best practices across engineering teams

- Build simulation and regression frameworks for real-time voice and conversational AI scenarios

- Partner with platform, provisioning, and integrations teams to ensure test coverage across all system boundaries

- Own test environments and CI/CD pipeline integration for automated test execution

- Investigate and triage production quality issues; drive root cause analysis and prevention

- Develop testing approaches that satisfy HIPAA and healthcare compliance requirements

Location

Remote (US). If you're in the Birmingham, AL area, this role is hybrid from our office (3 days in office / 2 remote).

Requirements

- 10+ years in software quality engineering, with a strong focus on test automation

- Proven experience designing and owning test automation frameworks from the ground up

- Proficiency in Python, TypeScript, or equivalent; strong scripting and tooling skills

- Experience with CI/CD integration (GitHub Actions, Jenkins, or equivalent)

- Deep understanding of testing strategies across unit, integration, and end-to-end layers

- Experience testing AI-powered applications — LLM- or model-backed features, including prompt and model regression testing

- Experience testing non-deterministic conversational flows, where the same input can yield different valid responses — designing assertions and evaluation criteria that tolerate variation without masking real regressions

- Experience generating synthetic data for test coverage at scale — realistic conversation transcripts, edge-case inputs, and de-identified patient scenarios

- Strong analytical and debugging skills — able to diagnose failures in complex distributed systems

- Experience testing APIs, backend services, and event-driven architectures

- Track record of raising quality standards across a team, not just writing tests

Nice to have

- Experience testing real-time systems, telephony, or voice platforms

- Familiarity with LLM evaluation tooling and prompt/model versioning workflows

- Experience evaluating speech-to-text or text-to-speech quality

- Load/performance testing experience for high-throughput systems

- HIPAA-regulated or compliance-heavy environment experience

- Experience with contract testing or service virtualization

Benefits

• Top-tier health coverage: medical, dental & vision
• 401(k)
• Paid time off + holidays, and paid parental leave

If you want to fix the front door of healthcare, we'd love to hear from you.

Transform9 is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.

Note: the Transform9 team will only email you from a transform9.com email address.