Staff Test & Release Engineer

Responsibilities

• Design and build end-to-end test infrastructure that provisions homogeneous and heterogeneous inference deployments, runs tests, and tears them down.
• Translate user stories and acceptance criteria into automated end-to-end tests, including failure scenarios.
• Run tests on GPU and Corsair nodes, schedule fairly on scarce shared hardware, and own flaky-test and failure-routing processes.
• Own release strategy for the orchestration platform, Kubernetes tooling, and benchmarking tools, including versioning, cadence, promotion, rollback, release notes, and go/no-go criteria.
• Lead the transition from platform-wide releases to independently released components and define compatibility and upgrade testing.
• Define release gates covering functional tests, upgrade and compatibility tests, model accuracy evaluations, and performance baselines.
• Partner with QE, product, platform, infrastructure, and architecture teams on shared testing and release strategies.
• Enable feature engineers to write and maintain tests for their own work.

Requirements

• 6+ years of experience in test infrastructure, software engineering in test, release engineering, or a closely related role, including ownership of end-to-end testing or releases for a distributed system.
• Demonstrated experience building test or release automation that improved release reliability or speed.
• Experience operating test and release infrastructure across on-premises clusters and a public cloud, preferably Azure.
• Strong Kubernetes experience, including multi-service workload deployment and debugging, Helm, operators, custom resources, and automated cluster setup and teardown.
• Experience designing CI/CD pipelines with artifact versioning, environment promotion, and rollback.
• Proficiency in Python and pytest or a comparable framework, plus fluency in a second programming language sufficient to read and test qualified code.
• Experience testing shared, limited GPU or accelerator hardware fleets, including scheduling, contention, and hardware faults.
• Ability to make and explain end-to-end testing tradeoffs and build consensus across teams.
• Experience using AI and agentic tools for test writing, failure triage, and log and code search while verifying their output.
• Preferred: experience testing or releasing LLM serving systems such as vLLM, SGLang, or TensorRT-LLM, including accuracy evaluations and performance benchmarks.
• Preferred: familiarity with speculative decoding, disaggregated serving, heterogeneous inference, hardware-in-the-loop testing, operator and controller testing tools, CRD migration testing, customer-managed Kubernetes deployments, canary or progressive rollouts, Rust, or startup environments.