Principal, Software Engineer - Observability

Principal, Software Engineer – Observability Location: Sunnyvale, CA Salary: $143,000 - $286,000 per year Employment type: Regular / Permanent Position Summary As an observability principal engineer, you will be a key researcher and technical lead expert in the architecture and development of cloud‑native observability designs, managed services, and real‑time telemetry software systems. You will use your depth of engineering experience to create visionary software architectures and telemetry systems that achieve an observability product portfolio. Additionally, you will design, develop, and implement large‑scale distributed systems that process large volumes of data, focusing on scalability, latency, and fault‑tolerance in every system built. You must be able to effectively communicate and build collaboration at all levels of the business and engineering. The ideal candidate will be adept at architecting large‑scale distributed systems and proficient in coding Java. You will also have experience in socializing architectural designs and roadmaps to internal and external customers. In this role you will work within a cloud infrastructure ecosystem consisting of VMs, Kubernetes, and containers to create metric software designs and solutions that enable real‑time monitoring and alerting of system and application metrics. You will lead research initiatives for cloud‑native designs and implementations within public and private clouds and utilize time‑series databases, correlation and data fusion, and AI/learned behaviors for anomaly detection and forward‑looking system and application performance analysis. Minimum Qualifications BS or MS in Computer Science, Engineering, or equivalent, with 10+ years of software engineering, design, and architecture experience. Deep understanding of the Java language and experience developing Java applications, libraries, SDKs, or services. Strong architecture leadership with demonstrated enterprise‑level software implementations. Experience leading architectural research, evaluation, creation of software designs, and distributed software implementations in production. Experience with technical leadership, software roadmaps, research and development, new software initiatives, and customer and engineering coordination and engagement. Full‑stack cloud software development experience. Preferred Qualifications Knowledge and/or experience with agentic AI, Large Language Models (LLMs), Retrieval‑Augmented Generation (RAG), and Natural Language Processing (NLP). Fluency in Python, JavaScript, advanced shell scripting, and configuration management tools (Ansible, Chef, Puppet). Experience with Application Performance Monitoring (APM), Distributed Tracing, Kubernetes, containers, service meshes, microservices, Istio, micrometer, OpenTelemetry, and Go development. Additional Skills API/lib/SDK development, integration, and utilization. Cloud technologies and cloud‑native designs. Cloud infrastructures such as OpenStack, Azure, GCP, or AWS. Large‑scale distributed systems with scalability and fault tolerance. Time‑series databases (InfluxDB, Kairos, Cortex, Thanos, Prometheus) or equivalent. ETL processes. Real‑time telemetry pipelines using publish/subscribe models (Kafka or equivalent). Data warehousing, data lakes, processing, and analytics. Advanced SQL (AzureSQL, Postgres, or equivalent). Unix/Linux shell scripting or similar knowledge. Real‑time monitoring and alerting with metric agents, dashboards, and alerting rules. Excellent written and verbal communication skills for diverse audiences. Ability to document requirements, architectural designs, and analysis findings in business and technical terminology. Experience developing in an Agile iterative CI/CD environment. Promotion and support of company policies, procedures, mission, values, and standards of ethics and integrity. Commitment to a Safe Workplace Walmart and its subsidiaries are committed to maintaining a drug‑free workplace and have a zero‑tolerance policy regarding the use of illegal drugs and alcohol on the job. This policy applies to all employees and aims to create a safe and productive work environment. #J-18808-Ljbffr Walmart