Quality Assurance Automation Engineer

21X | Remote | Full-Time

21X is building the next generation of conversational AI infrastructure.

Our platform powers intelligent AI agents that communicate with customers across voice, SMS, RCS, email, and web. We help enterprises automate complex customer interactions, execute transactions, and deliver exceptional customer experiences at scale.

We’re growing across mortgage, aviation, sports, and other industries, with a strong focus on enterprise reliability, performance, and scalability.

We’re looking for an exceptional Senior QA Automation Engineer to build and own our quality assurance function.

This isn’t a traditional QA position.

We’re looking for a highly technical, automation-first engineer who can design, implement, and lead our entire testing ecosystem.

You’ll work directly with engineering and product leadership to create a scalable QA infrastructure that enables rapid development without compromising quality.

Our ambition is to automate nearly everything that can reasonably be automated.

You’ll establish the frameworks, infrastructure, processes, and standards that make that possible.

As the company grows, you’ll also lead and manage manual QA engineers responsible for exploratory testing, complex scenarios, and validating AI behavior.

1. Build Our QA Automation Infrastructure

• Architect and implement an automated testing framework from scratch.
• Develop comprehensive end-to-end, integration, API, and regression testing.
• Integrate automated testing into our CI/CD pipelines.
• Establish automated testing across development, staging, and production environments.
• Create reusable test frameworks that scale with our engineering organization.
• Implement intelligent test execution, reporting, and failure analysis.

2. Automate Testing Across Our AI Platform

Our technology presents unique testing challenges. You’ll develop automation to validate:

• Voice AI: Call flows, agent responses, interruptions, transfers, latency, and conversation outcomes.
• SMS & RCS: Message delivery, conversational workflows, rich messaging, and carrier integrations.
• AI Agents: Tool execution, prompt adherence, knowledge retrieval, conversational accuracy, and agent behavior.
• CRM Integrations: Salesforce and other enterprise integrations, data synchronization, and API reliability.
• Enterprise Workflows: Multi-step automation, appointment scheduling, transaction execution, and complex business logic.

You’ll also build automated regression suites for AI agent behavior to identify changes in conversational performance as models, prompts, and underlying infrastructure evolve.

3. Implement AI-Powered QA

We want someone who thinks beyond traditional test automation.

• Leverage LLMs and AI-assisted testing to generate and maintain test cases.
• Build automated conversation simulations across multiple channels.
• Develop evaluation frameworks for conversational AI quality.
• Implement automated scoring for agent accuracy, compliance, and task completion.
• Create systems that detect regressions in AI performance.
• Explore autonomous testing agents that can identify defects and validate new functionality.

4. Lead Manual QA Engineers

• Establish QA standards, test plans, and release validation processes.
• Lead and mentor manual QA engineers as the team grows.
• Delegate exploratory testing, edge-case validation, and complex user scenarios.
• Create clear defect reporting, severity classifications, and escalation procedures.
• Ensure manual testing complements automation rather than duplicating it.

5. Own Release Quality

• Establish measurable quality gates for production releases.
• Implement automated smoke tests and regression testing.
• Build dashboards for test coverage, defect trends, and release readiness.
• Partner with engineering leadership to reduce production defects.
• Continuously improve test reliability, execution speed, and automation coverage.
  

Required Experience

• 5+ years in software QA engineering, with extensive test automation experience.
• Proven experience building QA automation frameworks from scratch.
• Strong programming skills in TypeScript, JavaScript, or Python.
• Hands-on experience with Playwright, Cypress, Selenium, or similar frameworks.
• Strong API testing experience, including REST and GraphQL.
• Experience implementing CI/CD testing pipelines.
• Experience testing distributed systems and complex integrations.
• Experience leading or mentoring QA engineers.
• Strong understanding of software architecture, debugging, and development workflows.
• Ability to operate independently in a fast-moving startup environment.

Highly Preferred

• Experience testing conversational AI, LLM applications, or AI agents.
• Familiarity with LLM evaluation frameworks such as LangSmith, Braintrust, or custom evaluation pipelines.
• Experience with voice technologies, telephony systems, SIP, or messaging APIs.
• Familiarity with AWS and cloud-native environments.
• Experience implementing load, performance, and reliability testing.
• Experience with automated testing of nondeterministic systems.
  

You’re an engineer first and a QA professional second.

You write production-quality code, understand complex systems, and believe that repetitive manual testing should be automated wherever practical.

You don’t wait for someone to tell you what to test. You understand the architecture, identify risk, and build systems that catch problems before customers experience them.

You’re equally comfortable writing automation code, designing testing infrastructure, and leading engineers.

You thrive in environments where you can build something from scratch and take complete ownership of the outcome.

I’d measure this hire against five outcomes:

1. Defect prevention : Meaningful reduction in customer-reported defects.
2. Automation coverage: 90%+ of critical, repeatable regression scenarios automated over time.
3. Release confidence: Every production deployment passes defined automated quality gates.
4. Testing efficiency: Significant reduction in the time required to validate a release.
5. AI reliability: Automated detection of regressions in conversation outcomes, tool execution, latency, and agent behavior.