Senior Systems Engineer

Senior Systems EngineerSan Francisco, California Onsite or RemoteAt Evidently, we are raising the quality of care for every patient by empowering clinicians with the right information at the speed of thought. Our Clinical Data Intelligence platform extracts meaning from an ocean of clinical notes, outside records, and scanned documents — and delivers it to clinicians in the moment it matters. We're a small, mission-driven team building something genuinely hard and genuinely impactful.We're looking for a Senior Systems Engineer who owns reliability the way a founding engineer owns their product: with a sense of personal accountability, eyes everywhere, and no tolerance for fires that were preventable.This is a clear home-base role. Infrastructure and reliability are your domain and your primary responsibility. You are the person who notices before anyone else that something is trending wrong. You work alongside leadership as a partner in operational decision-making, not getting paged when things break, but instead making sure they don't break in the first place, and standing shoulder-to-shoulder with the team when they do. You bring a calm, systematic approach to incident response, help divide up the work intelligently under pressure, and run thorough postmortems that actually change behavior.At the same time, you have the seniority and breadth to patrol adjacent territory. Integration work, security and compliance posture, database performance, AI inference infrastructure: you can cover these areas when they touch systems, and you're not waiting to be asked. You see the whole field.We are an AI-first company, and that applies to how you work as much as what you build. We expect you to use AI tools aggressively, as an operator who directs them, not a passive user who accepts their output uncritically.What You'll DoOwn ReliabilityHold end-to-end accountability for the reliability, availability, and performance of Evidently's production systems: our SMART on FHIR platform, AI inference pipelines, data systems, and the integrations that connect them all.Define and own SLOs, SLIs, and error budgets for critical services; build the dashboards and alerting that make the health of the system visible to everyone, not just you.Lead incident response: triage fast, communicate clearly, coordinate the team, contain blast radius. Run blameless postmortems that actually result in durable fixes and systemic improvements, not just tickets.Proactively identify reliability risks before they become incidents, through capacity planning, load testing, chaos engineering, and systematic review of production signals.Build runbooks and on-call practices that make rotating on-call humane and effective, not a rotation people dread.Build and Operate InfrastructureDesign, implement, and maintain highly available, scalable, and secure cloud infrastructure on Google Cloud Platform: GKE, Cloud Run, AlloyDB, Cloud Storage, Pub/Sub, and adjacent services.Own Infrastructure as Code across all environments with a cloud-agnostic, open-minded, code-centric approach, applying standard software engineering discipline (version control, code review, testing, promotion gates).Build and maintain CI/CD pipelines (GitHub Actions) that make deployments fast, safe, and boring, with rollback, canary, and feature toggling patterns baked in.Manage observability infrastructure end-to-end: instrumentation, tracing, log pipelines, alerting, and dashboards structured around what actually matters operationally.Own data infrastructure health across GCP data systems, including AlloyDB, Cloud Storage, BigQuery, and Cloud Logging: performance tuning, query analysis, storage lifecycle management, connection management, backup/restore, and disaster recovery.Own AI and LLM inference infrastructure end-to-end: Vertex AI, Model Garden, and model serving platforms (vLLM, Cloud Run, GPU provisioning), latency SLOs, cost optimization, auto-scaling characteristics, and capacity planning for production AI workloads.Set the Standard on Security and ComplianceMaintain and improve our security posture across all infrastructure: network segmentation, IAM, secrets management, vulnerability management, audit logging.Own infrastructure-level compliance for HIPAA and SOC 2, not just checkbox compliance, but genuinely secure and auditable systems.Be a thought partner to engineering on security architecture decisions, particularly where AI workloads introduce new data handling surface area.Patrol Adjacent TerritoryCover integration infrastructure where it intersects with systems: EHR integrations, third-party APIs, data pipelines, and AI inference serving. You don't need to own these, but you understand them well enough to step in.Support backend engineers on runtime performance: Python async, FastAPI/Django serving characteristics, concurrency patterns, and where infrastructure choices constrain application-level performance.What You'll BringNon-negotiables8+ years of experience in SRE, infrastructure engineering, or DevOps, with a clear track record of owning reliability, not just contributing to it.You think in SLOs. You have defined them, defended them, and used error budgets to make real decisions about when to slow down feature work.Deep experience with incident management: you've led responses to serious production incidents, written the postmortems, and driven the follow-through.Expert-level command of GCP (GKE, Cloud Run, Cloud SQL/AlloyDB, Pub/Sub, IAM, VPC, Cloud Armor) or directly comparable experience on AWS or Azure with a credible GCP migration story.Very strong shell scripting skills (Bash) and fluency in Python for automation; comfortable reading application code well enough to understand its operational behavior.Depth That MattersObservability done right: you've built meaningful monitoring from the ground up, not just enabled a tool. You know the difference between dashboards that look good and dashboards that catch problems.Deep knowledge of GCP data infrastructure and database operational characteristics: query planning and tuning in PostgreSQL/AlloyDB/BigQuery, log management and retention in Cloud Logging, data lifecycle management in Cloud Storage, analytical workloads in BigQuery, and how to diagnose issues across these services as they scale.Understanding of container and Kubernetes internals well beyond YAML: scheduling, resource management, network policies, pod disruption budgets, autoscaling behavior.Security and compliance grounding: you've operated in regulated environments (HIPAA, SOC 2, or similar) and understand the difference between compliance theater and systems that are actually secure.Deep operational mastery of AI and LLM inference serving, including Vertex AI and Model Garden: GPU provisioning and orchestration, model deployment strategies, optimizing inference latency and throughput, GPU cost controls, and managing the unique reliability patterns of production LLM workloads at scale.MindsetYou have a strong operational instinct. You notice anomalies, you ask why things are trending the way they are, and you're not satisfied until you understand the root cause.You treat reliability as a product you own, not a set of tasks assigned to you.You're calm under pressure, structured in how you communicate during incidents, and rigorous in follow-through after them.You're senior enough to have strong opinions about how things should be done and humble enough to change them when presented with better evidence.You use AI tools fluently as a practitioner, writing infrastructure code, debugging incidents, drafting runbooks, with the judgment to know when the output is right and when it needs correction.BonusFamiliarity with healthcare data standards: FHIR, HL7, SMART on FHIR application architecture.Experience operating infrastructure for LLM or AI inference workloads at production scale.Background in building internal developer platforms or golden-path tooling that makes other engineers more effective.Strong hands-on experience or familiarity with Infrastructure as Code tooling such as Kubernetes, Terraform, or Pulumi.Experience with OpenTelemetry for instrumentation and distributed tracing across complex environments.Tech StackBash · Python · FastAPI · Django · PostgreSQL · GCP (GKE · Cloud Run · AlloyDB · BigQuery · Cloud SQL · Pub/Sub) · Kubernetes · Docker · OpenTelemetry · GitHub ActionsYou'll join a small, highly effective,