Test Lead

Job Title: Chaos Test Lead

Location: Denver, CO 80214 (Hybrid – candidates must be willing to relocate)

Contract Duration: 6 Months

Visa Requirement: Visa-independent candidates only

Job Description

We are looking for an experienced Chaos Test Lead with strong expertise in performance testing, chaos engineering, and observability to evaluate system resilience, identify potential failure points, and improve the reliability of distributed applications. The ideal candidate will have hands-on experience designing and executing chaos experiments, performance testing, and cloud-based environments.

Key Responsibilities

• Plan, design, and execute chaos testing strategies for services, applications, and infrastructure.
• Define steady-state metrics, Service Level Objectives (SLOs), experiment hypotheses, blast radius, and rollback plans.
• Execute fault-injection experiments in pre-production and production environments with appropriate safeguards.
• Develop and automate chaos experiments using tools such as Gremlin, Harness, Chaos Mesh, Litmus, or AWS Fault Injection Service.
• Integrate chaos testing into CI/CD pipelines and combine fault injection with load and performance testing.
• Monitor system behavior using observability tools such as Grafana, Prometheus, Splunk, Datadog, and Dynatrace.
• Analyze test results, identify root causes, and provide recommendations to improve system resilience.
• Conduct game days and failure drills with development, QA, and operations teams.
• Review system architecture to identify resilience gaps, single points of failure, retry issues, and timeout handling weaknesses.
• Validate monitoring alerts, runbooks, and incident response procedures.
• Track resilience metrics, including Mean Time to Recovery (MTTR), recovery time, and failure detection time.

Required Skills and Qualifications

• 8–10+ years of experience in performance engineering, QA automation, DevOps, or a related field.
• Hands-on experience with at least one chaos engineering tool, such as Gremlin, Harness, Litmus, Chaos Mesh, Chaos Monkey, or AWS FIS.
• Strong experience in performance and load testing using tools such as JMeter or LoadRunner.
• Solid understanding of distributed systems, microservices, and failure scenarios, including latency, network partitions, cascading failures, and retry storms.
• Experience with Kubernetes, containers, and cloud platforms such as AWS, Azure, or GCP.
• Proficiency in scripting or programming using Python, Java, or Bash.
• Hands-on experience with observability, monitoring, and distributed tracing tools.
• Familiarity with CI/CD tools such as Jenkins, GitLab CI, or GitHub Actions.
• Strong analytical, troubleshooting, documentation, and communication skills.
• Telecom BSS/OSS domain knowledge is required.

Preferred Skills

• Experience working across multiple cloud environments, including AWS, Azure, and GCP.
• Experience automating resilience testing and integrating chaos experiments into continuous delivery workflows.
• Knowledge of SRE practices, reliability engineering, incident management, and SLO-based monitoring.

Top Responsibilities

• Chaos testing planning and experiment design.
• Chaos test execution and result analysis.
• Providing recommendations to improve application and infrastructure resilience.

Interview Process

Virtual and face-to-face interviews.

Note: Candidates must be visa-independent and willing to work in the Denver, CO hybrid location. No certifications are required.