Staff Site Reliability Engineer

Staff Site Reliability Engineer (SRE)

This is a hybrid position requiring Tuesdays and Thursdays on-site. In CA, the on-site requirement is Tuesdays in San Francisco and Thursdays in San Jose, CA; in NYC, Tuesdays and Thursdays are on-site in our NYC office.

The Splunk Agent Resilience team is defining the future of AI resilience. Together our team provides scalable, cost-effective evaluation and guardrails that ensure AI agents behave as intended, improving reliability and reducing risks. This unified approach empowers our customers to expertly deploy and manage AI-powered applications with enhanced observability and control.

As a Staff Site Reliability Engineer (SRE), you will provide technical leadership for the reliability, scalability, and operational architecture of Splunk Agent Observability's platform. You will define the long-term reliability strategy, lead major infrastructure initiatives, and drive engineering excellence across deployment automation, production operations, and platform resiliency. In addition to owning complex production systems, you will influence engineering direction across teams and establish best practices for operating large-scale cloud and on-prem deployments.

Your Impact

• Define and drive the technical roadmap for platform reliability, scalability, and operational excellence.
• Lead the architecture and evolution of deployment platforms supporting cloud and air-gapped customer environments.
• Establish reliability engineering standards, including service level objectives (SLOs), operational readiness, capacity planning, and resiliency reviews.
• Lead major reliability and scalability initiatives across Kubernetes, deployment infrastructure, databases, and networking.
• Drive automation to eliminate operational toil and improve engineering productivity.
• Design and build internal platforms, frameworks, and tooling that enable reliable operations at scale.
• Lead complex production incident response and drive systemic improvements through root cause analysis and long-term remediation.
• Partner with engineering leadership to influence platform architecture, deployment strategy, and production readiness.
• Mentor engineers and raise the engineering bar through technical leadership, design reviews, and operational guidelines.
• Collaborate with customers and internal teams to build secure, scalable, and highly reliable deployment architectures for both cloud and on-prem environments.

Minimum Qualifications

• 8+ years' experience with a Bachelor's degree or 6+ yrs with Masters or 3+ years with a PhD, or equivalent related experience; Experience to include at least 6 years in Site Reliability Engineering, Platform / Cloud / Infrastructure Engineering, or related fields.
• 5+ years' operating large-scale Kubernetes platforms in production.
• Experience designing highly available, scalable, and resilient distributed systems.
• Experience with AWS, GCP, or other public cloud platforms.
• Strong experience designing CI/CD platforms and deployment automation at scale.

Preferred Qualifications

• Expertise in observability, monitoring, alerting, capacity planning, and performance engineering.
• Strong programming skills in Python and/or Go.
• Deep experience with Infrastructure as Code (Terraform or similar).
• Experience with MLOps preferred.
• Strong understanding of networking, distributed systems, storage, databases, and cloud architecture.
• Experience operating both SaaS and enterprise/on-prem deployments.
• Demonstrated technical leadership across multiple engineering teams.
• Experience leading incident management, postmortems, and long-term reliability initiatives.