Principal Software Engineer - AI Platform (Production Engineering / Reliability)

Principal Individual Contributor (IC) For Ai Platform

We're building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you'll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time.

We are seeking a Principal Individual Contributor (IC) to lead production engineering, observability, and operational excellence for our AI Platform. This role sits at the intersection of ML systems, distributed infrastructure, and production reliability, ensuring that our AI services are scalable, observable, and resilient in real-world environments.

As a senior technical leader, you will define and drive best-in-class production practices, build robust monitoring and alerting ecosystems, and partner across engineering, ML, and platform teams to ensure mission-critical AI systems meet high availability, performance, and reliability standards.

Key Responsibilities

• Own and evolve production operations strategy for AI/ML platforms and services
• Define SLOs, SLIs, and error budgets for AI systems (online & batch/inference pipelines)
• Lead root cause analysis (RCA) and drive systemic improvements post-incident
• Establish operational readiness standards for launching new AI capabilities
• Build frameworks for on-call excellence, incident response, and escalation

Observability, Monitoring & Alerting

• Design and implement end-to-end observability systems across AI workloads:
• Model performance monitoring
• Data pipeline health
• Infrastructure metrics
• Build and scale monitoring and alerting frameworks using modern tooling (e.g., Prometheus, Grafana, OpenTelemetry, Datadog, Azure Monitor, etc.)
• Define actionable, low-noise alerts tied to business and system impact
• Develop dashboards and telemetry standards for real-time visibility across services
• Drive adoption of golden signals (latency, errors, throughput, saturation) in AI systems

AI/ML Production Systems Excellence

• Ensure reliable deployment and operation of:
• Real-time inference services
• Model pipelines (training, validation, deployment)
• Data ingestion and feature pipelines
• Implement model observability (drift detection, data skew, performance degradation)
• Partner with ML engineers to improve production readiness of models
• Establish lifecycle standards for models in production environments

Automation & Platform Development

• Build internal platforms and tooling for:
• Automated incident detection and response
• Self-healing systems
• Deployment validation and canarying
• Drive Infrastructure as Code (IaC) and policy automation
• Improve system resilience through chaos testing and fault injection

Technical Leadership & Strategy

• Act as a trusted technical advisor across platform, ML, and product teams
• Set direction for operational excellence in AI systems at org scale
• Mentor senior engineers and influence cross-team architectural decisions
• Lead adoption of industry best practices in reliability engineering and observability

Required Qualifications

• 10+ years in software engineering, production engineering, or SRE roles
• Deep experience operating large-scale distributed systems in production
• Proven track record building monitoring, observability, and alerting systems
• Strong expertise in incident management and production support models
• Experience working with cloud platforms (Azure, AWS, GCP)

Preferred Qualifications

• Experience supporting AI/ML platforms or data-intensive systems
• Familiarity with model lifecycle management and MLOps practices
• Knowledge of:
• OpenTelemetry, Prometheus, Grafana, Datadog
• Kubernetes and containerized workloads
• Streaming systems (Kafka, Event Hub, etc.)
• Experience defining and implementing SLO-driven engineering
• Background in high-availability, low-latency systems

Key Competencies

• Systems thinking and ability to reason about complex, interdependent systems
• Strong bias for automation, scalability, and long-term solutions
• Exceptional debugging and incident management skills
• Ability to influence without authority across multiple teams
• Passion for operational excellence and reliability

The typical pay range for this role is: $144,200.00 - $288,400.00. This pay range represents the base hourly rate or base annual full-time salary for all positions in the job grade within which this position falls. The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors. This position is eligible for a CVS Health bonus, commission or short-term incentive program in addition to the base pay range listed above. This position also includes an award target in the company's equity award program.

Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.

Great benefits for great people. We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families. This full-time position is eligible for a comprehensive benefits package designed to support the physical, emotional, and financial well-being of colleagues and their families. The benefits for this position include medical, dental, and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, based on eligibility.