QA Automation Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a QA Automation Engineer based in the United States. This role focuses on building reliable automated quality coverage for an evolving AI-powered platform. You will translate complex product requirements and real customer workflows into meaningful, risk-based testing strategies. The position combines browser automation, API and integration testing, AI response evaluation, security validation, and release readiness. You will use Playwright and modern development practices to create automation that engineers can trust and maintain at scale. A key part of the role involves testing nondeterministic AI behavior, including conversational workflows, retrieval, tool execution, and model interactions. You will collaborate closely with Product and Engineering to identify risks, investigate failures, and provide evidence-based release recommendations. The environment emphasizes auditability, security, government compliance, continuous learning, and thoughtful use of AI-assisted testing technologies. \n Accountabilities: - Translate product requirements, customer use cases, and complex feature interactions into clear, testable outcomes and acceptance criteria in collaboration with Product and Engineering. - Design, implement, maintain, and expand automated test coverage using Playwright, including critical user journeys, feature functionality, regressions, API integrations, and end-to-end workflows. - Develop and maintain a risk-based testing strategy that prioritizes customer impact, feature dependencies, product roadmap priorities, and changing delivery requirements.

- Test conversational AI workflows, including streaming responses, conversation history, file uploads, document retrieval and citations, model selection, tool execution, interruptions, timeouts, and partial failures. - Evaluate AI response quality by creating representative datasets and scoring criteria covering accuracy, grounding, instruction following, and appropriate responses to unsafe requests while accounting for natural variation in model behavior. - Integrate automated tests into CI/CD pipelines, investigate flaky tests, maintain isolated test data and fixtures, and provide actionable diagnostics when failures occur. - Communicate release readiness by documenting defects with reproducible evidence, customer impact, severity, coverage gaps, and residual risks ahead of UAT and production releases. - Maintain decision history and testing documentation, capturing expected behavior, approved changes, and the reasoning behind testing decisions so intended changes can be distinguished from regressions. - Extend automation across backend APIs, authentication, billing and token behavior, model routing, AI workflow execution, file parsing, MCP and tool execution, agentic harnesses, passthrough APIs, and provider-facing API compatibility. - Design AI workflow tests efficiently to minimize unnecessary token consumption, external provider calls, latency, and execution costs without reducing meaningful test coverage. - Build repeatable security-sensitive validation covering user isolation, permission boundaries, input validation, sanitization, rate limits, replay prevention, safe error handling, and layered security controls. - Use AI-assisted testing tools responsibly for test design, generation, triage, and maintenance while critically reviewing generated tests, assertions, and suggested repairs for reliability, reviewability, and deterministic validation. - Scale and maintain automation infrastructure, fixtures, test data, and supporting tooling as the product surface and engineering organization grow. Requirements: - At least 4 years of QA automation engineering experience , with demonstrated ownership of automated testing programs or significant automation initiatives.