Principal Site Reliability Engineer

Principal Site Reliability Engineer (IC4)

As a Principal Site Reliability Engineer (IC4), you will be responsible for designing, building, and operating highly available, scalable, secure, and resilient cloud services. You will combine software engineering with infrastructure expertise to improve service reliability, operational efficiency, and developer productivity across large-scale distributed systems.

You will lead complex reliability initiatives, drive automation-first operational practices, and develop software solutions that eliminate manual toil. You will partner closely with software engineering, cloud infrastructure, security, and product teams to architect resilient platforms that meet aggressive availability, scalability, and performance objectives.

Success in this role requires deep expertise in distributed systems, cloud infrastructure, coding, automation, observability, incident management, and operational excellence. You will leverage modern AI technologies, machine learning, and intelligent automation to streamline operations, accelerate incident response, improve troubleshooting, and enable autonomous system management.

You are expected to be a technical leader who influences architecture, establishes engineering best practices, mentors other engineers, and drives continuous improvements across multiple services and organizations.

Reliability Engineering & Service Ownership

• Design, build, and operate highly available, scalable, and fault-tolerant cloud services that meet defined Service Level Objectives (SLOs) and Service Level Agreements (SLAs).
• Lead architecture reviews to improve resiliency, scalability, observability, and operational readiness.
• Forecast infrastructure growth, capacity requirements, and resource utilization while proactively mitigating operational risks.
• Continuously improve platform reliability through engineering solutions rather than manual operational processes.
• Define reliability standards, operational best practices, and service readiness criteria across multiple engineering teams.

Software Engineering, Automation & AI

• Design and develop production-quality software, automation frameworks, and internal platforms using Python, Java, Go, or similar programming languages.
• Build scalable automation to eliminate repetitive operational work, reduce manual intervention, and improve engineering productivity.
• Develop APIs, microservices, and tooling that simplify infrastructure management, deployment, monitoring, and operational workflows.
• Leverage Generative AI, Large Language Models (LLMs), AI agents, and intelligent automation to:
• Automate routine operational tasks and runbooks.
• Accelerate incident triage and root cause analysis.
• Improve log analysis and anomaly detection.
• Generate operational insights and recommendations.
• Automate knowledge management and operational documentation.
• Enhance developer productivity and self-service capabilities.
• Identify opportunities to incorporate AI-driven operational intelligence into existing systems to improve efficiency, reliability, and scalability.

Infrastructure Engineering

• Design and optimize cloud infrastructure supporting distributed services across multiple regions and availability domains.
• Improve system resiliency through redundancy, automation, and infrastructure-as-code.
• Build and maintain deployment pipelines, provisioning frameworks, and configuration management solutions.
• Drive infrastructure standardization and platform modernization initiatives.

Observability & Operational Excellence

• Design comprehensive monitoring, logging, tracing, and alerting strategies.
• Build meaningful dashboards, health reporting, and service performance metrics.
• Improve alert quality, reduce operational noise, and enhance system visibility.
• Define and measure Service Level Indicators (SLIs), SLOs, and error budgets.
• Continuously optimize operational processes using data-driven insights.

Incident Management & Reliability

• Lead critical production incident response and act as a senior escalation point during major service events.
• Drive root cause analysis, corrective actions, and post-incident reviews with a focus on long-term engineering improvements.
• Develop automated remediation solutions to reduce Mean Time to Detect (MTTD) and Mean Time to Recover (MTTR).
• Continuously improve operational readiness through disaster recovery testing, game days, and failure injection exercises.

Performance & Scalability

• Identify performance bottlenecks across applications, infrastructure, storage, databases, and networking.
• Drive optimization initiatives to improve system throughput, latency, efficiency, and cost.
• Perform capacity planning and predictive scaling using historical trends and telemetry data.
• Optimize resource utilization while maintaining service reliability and customer experience.

Technical Leadership

• Serve as the technical leader for reliability engineering initiatives spanning multiple services and organizations.
• Establish engineering standards, architectural patterns, and operational best practices.
• Review system designs and influence technical decisions across engineering organizations.
• Mentor engineers on software engineering, distributed systems, cloud technologies, operational excellence, and automation.
• Drive engineering excellence through code reviews, design reviews, technical guidance, and knowledge sharing.

Innovation & Continuous Improvement

• Evaluate emerging cloud technologies, AI capabilities, automation frameworks, and observability platforms.
• Drive adoption of modern engineering practices including GitOps, Infrastructure as Code, continuous delivery, and policy-as-code.
• Identify opportunities to simplify architecture, eliminate operational complexity, and improve platform reliability.
• Champion engineering initiatives that increase service scalability, operational efficiency, and customer satisfaction.

Core Competencies

Planning & Execution

• Lead complex, cross-functional technical initiatives from design through production deployment.
• Balance reliability, scalability, performance, cost, and delivery priorities across multiple concurrent projects.
• Drive execution with minimal direction while effectively managing technical risk and dependencies.

Collaboration & Influence

• Partner with software engineering, cloud infrastructure, security, networking, and product organizations to deliver reliable services.
• Influence technical direction across organizations through strong communication and technical leadership.
• Build consensus among stakeholders with differing priorities and technical perspectives.

Problem Solving

• Solve highly ambiguous and complex technical problems involving distributed systems at cloud scale.
• Apply systematic debugging and data-driven analysis to identify root causes and implement long-term engineering solutions.
• Make sound technical decisions using engineering judgment and operational experience.

Continuous Learning

• Stay current with advances in cloud computing, distributed systems, AI, software engineering, cybersecurity, and Site Reliability Engineering.
• Evaluate emerging technologies and drive adoption where they provide measurable operational value.
• Foster a culture of continuous learning, experimentation, and technical excellence.

Talent Development

• Mentor engineers and contribute to their technical growth through coaching and knowledge sharing.
• Participate in hiring, interviewing, and technical assessments to build high-performing engineering teams.
• Lead by example through engineering excellence, ownership, and customer-focused decision making.

Preferred Technical Skills

• Strong software engineering experience with Python and/or Java (Go is a plus).
• Experience building production automation, APIs, services, and developer tooling.
• Cloud platforms (OCI, AWS, Azure, or GCP).
• Kubernetes, Docker, container orchestration, and service mesh technologies.
• Infrastructure as Code (Terraform, Ansible, Helm, Pulumi).
• CI/CD platforms and DevOps practices.
• Observability platforms such as Prometheus, Grafana, OpenTelemetry, ELK, Splunk, or Datadog.
• Distributed systems, networking, Linux, storage, databases, and performance tuning.
• AI-assisted operations, LLM integrations, automation frameworks, and intelligent operational tooling.
• Experience operating mission-critical, internet-scale production systems with stringent availability and reliability requirements.

Qualifications

Disclaimer: Certain U.S. based or U.S. customer or client-facing roles may be required to comply with applicable requirements, such as immunization/occupational health mandates, and/or drug testing requirements.