AI Site Reliability Engineer

Job Description

Job Description

Job Title: AI Site Reliability Engineer Location: Bay Area, CA [ONSITE]
Job Description:
The engineer will improve the availability, performance, and operability of AI platform services and their
supporting infrastructure. This senior role combines incident response with reliability engineering across
Linux, Kubernetes, GPU systems, storage, and networks. The engineer will turn production evidence
into automation, safer changes, and sustained reductions in operational toil.
Key responsibilities-

• Define service-level indicators and objectives for inference, platform APIs, orchestration, and relevant
• batch services. Establish actionable alerts and use error budgets to guide reliability work.
• Build observability across service, workload, GPU, host, and dependency layers using metrics, logs,
• traces, events, and health signals. Correlate symptoms across systems.
• Participate in the agreed on-call rotation and lead or support incident triage. Restore service, maintain
• incident timelines, coordinate escalations, and communicate impact clearly.
• Diagnose service degradation involving Kubernetes, Linux, scheduling, storage, networking, model
• endpoints, or job orchestration. Distinguish application faults from infrastructure failures.
• Automate routine operations and remediation with Python, Bash, or Go. Introduce preventive checks
• for recurring failures and validate automation before production use.
• Conduct capacity and performance analysis for GPU memory, utilization, queues, throughput, and
• service latency. Work with inference and platform engineers on scaling and bottlenecks.
• Review production readiness and change plans, including observability coverage, rollback, recovery,
• and dependency health. Execute resilience and recovery tests appropriate to the service.
• Lead post-incident reviews, track corrective actions, and maintain runbooks and escalation paths.
• Use operational reviews to prioritize recurring risks and reliability improvements.

Required experience and skills-

• 7 years in SRE, production operations, or reliability-focused infrastructure engineering, with hands on responsibility for distributed services.
• Strong troubleshooting across Linux, Kubernetes, networking, storage dependencies, and distributed systems under production pressure.
• Experience defining SLIs and SLOs, building alerting and response workflows, and using service evidence to prioritize engineering work.
• Strong scripting and automation skills, including tested remediation tooling, configuration management, and integration with delivery pipelines.
• Experience with incident management, root cause analysis, post-incident follow-through, and clear stakeholder communication.
• Ability to balance urgent service restoration with longer-term reliability improvements and to document actionable operational procedures.

Preferred experience-

• Experience operating AI platforms, ML infrastructure, GPU-backed inference, distributed training, or large HPC-like service environments.
• Familiarity with Prometheus, Grafana, ELK/OpenSearch, Loki, PagerDuty, DCGM, and GPU failure diagnostics.
• Experience with error budgets, safe deployment strategies, disaster recovery, and capacity planning across mixed batch and online services.

Recommended certifications-

• Preferred: NVIDIA-Certified Professional AI Operations (NCP-AIO) and Certified Kubernetes
• Administrator (CKA). Observability specialization: Prometheus Certified Associate (PCA). Foundational:
• NVIDIA-Certified Associate AI Infrastructure and Operations (NCA-AIIO).
• Certifications are preferred credentials. Candidates should demonstrate the corresponding skills
• through practical delivery and troubleshooting experience.

Expected deliverables and success measures-

• Deliver service dashboards, SLO definitions, actionable alerting, incident runbooks, and tested operational automation. Success is demonstrated through service objective attainment, reduced recurring incidents and manual toil, completed corrective actions, and validated recovery capability.