Senior Software Engineer Observability

Senior Software Engineer — Observability Cary, North Carolina, United States Software and Services Imagine what you could do here. At Apple, new ideas have a way of becoming extraordinary products, services, and customer experiences very quickly. Apple Pay and Wallet are at the heart of how millions of people around the world make payments, store credentials, and interact with commerce every day. Behind every tap, every transaction, and every boarding pass is a set of systems that must be fast, reliable, and secure at massive scale. We are looking for a Senior Software Engineer to join the Wallet, Payments, and Commerce (WPC) team and help define how we build, observe, and operate these critical systems. In this role, you will design and develop the observability platforms, reliability tooling, and AI-powered automation that empower WPC engineering teams to ship with confidence, resolve issues in real time, and deliver seamless experiences to hundreds of millions of users. Description Apple Pay processes billions of transactions across millions of merchants worldwide. The systems behind Apple Pay, Apple Wallet, Apple Cash, Tap to Pay, and our commerce platforms must meet the highest standards for availability, latency, security, and correctness — with zero tolerance for downtime or data loss. This is a software engineering role within one of Apple's most critical and high-scale service organizations. You will bring strong design and architecture skills to solve complex problems at the intersection of payments infrastructure, observability, DevOps tooling, and AI — and you will ship production-quality software that WPC teams depend on to keep Apple Pay running for customers around the world. Responsibilities Design and build a solution that will give teams deep visibility into the health and performance of our services and the supporting infrastructure - an Observability platform using OTEL standards for metrics and traces, easily configurable visualization layer and intelligent alerting - built to support ingestion, processing and serving high-volume telemetry data while meeting the unique demands of financial and payment systems where every millisecond and every failed transaction matters and meeting compliance/privacy/security requirements for data is critical. Build and maintain helper library that provide reliable and easy to use functionality for emitting telemetry, propagating context and enforcing standards, (OTEL-based), and enable integration with consuming applications and telemetry visualization layer (such as Grafana or Datadog) ensuring zero telemetry loss during transfer. Configure and implement Otel processors, filters, and metadata enrichment. Standardize and deploy approved exporters across the pipeline. Execute metadata enrichment standards across the SDK, Collector, and Gateway levels. Refactor and migrate legacy Java-based instrumentation into standard OpenTelemetry formats to ensure accurate alerting. Provide technical documentation and training material to facilitate knowledge sharing and team adoption. Build intelligent tooling — from automated incident triage and transaction-aware root cause analysis to AI-assisted debugging and runbook generation — that meaningfully reduces on-call burden, accelerates resolution of payment-critical incidents, and improves developer productivity across WPC. Minimum Qualifications 7+ years of experience in software engineering with a strong emphasis on building backend systems, platforms, and tooling. Proven experience building observability or reliability tooling such as monitoring systems, alerting platforms, log pipelines, tracing infrastructure, or diagnostic tools. Deep technical understanding of the Otel ecosystem, including configuring Collectors, SDKs, and Exporters. Proven experience integrating telemetry data with Grafana and Datadog. Strong proficiency in Kotlin, Go, Python, or Java, with Kotlin experience highly valued. Deep understanding of distributed systems concepts including data pipelines, event-driven architectures, API design, and service-to-service communication. Experience designing systems that handle high-throughput data ingestion and processing such as telemetry pipelines, streaming systems, or transaction processing platforms. Solid understanding of datastores and their trade-offs — relational databases, NoSQL, time-series databases, caching layers, and message brokers. Working knowledge of production infrastructure concepts including containers, Kubernetes, cloud platforms, networking fundamentals, and CI/CD systems. Experience with observability platforms and standards such as Prometheus, Grafana, Datadog, OpenTelemetry, Splunk, or similar. Hands‑on experience with LLMs and AI APIs — building integrations, agents, or automation pipelines that solve real operational problems. Experience building internal developer tools, platforms, or CLIs that are adopted and used daily by engineering teams. Familiarity with SRE practices including SLOs/SLIs, error budgets, incident management, and post‑incident review. Strong software design skills — you write clean, well‑tested, maintainable code and care deeply about system architecture. Excellent communication and collaboration skills with the ability to lead design reviews and drive technical alignment across teams. BS or MS in Computer Science, Software Engineering, or a related field, or equivalent professional experience. Preferred Qualifications Experience working on payment systems, financial services platforms, or other high‑compliance, high‑availability domains. Experience applying AI/ML to operational problems such as anomaly detection, transaction pattern analysis, log summarization, automated diagnostics, or intelligent alerting. Familiarity with security and compliance considerations in financial and payment systems (PCI‑DSS, tokenization, encryption at rest and in transit). Experience with Infrastructure‑as‑Code tools such as Terraform or Pulumi as a tooling builder. Experience building systems that require strong correctness guarantees — idempotency, exactly‑once processing, auditability, and data integrity at scale. Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant #J-18808-Ljbffr Apple