Principal, Software Engineer - Observability

Position Summary As an observability principal engineer, you will be a key researcher and technical lead expert in the architecture and development of cloud native observability designs, managed services, and real-time telemetry software systems. You will use your depth of engineering and experience to create visionary software architectures and telemetry systems to achieve an observability software product portfolio. Additionally, you will design, develop, and implement large‑scale distributed systems that process large volumes of data focusing on scalability, latency, and fault‑tolerance in every system built. You must be able to effectively communicate and build collaboration at all areas and levels of the business and engineering. An ideal candidate will be adept at architecting large‑scale distributed systems and proficient in coding Java. You will also utilize multiple telemetry technologies such as data models, metric libraries, data logging, distributed tracing, datalakes, data correlation, rule‑based alerting engines, real‑time data streaming pipelines, TSDBs, and application performance management (APM). Working in a cloud infrastructure ecosystem consisting of VMs, Kubernetes, and containers, you will create metric software designs and solutions enabling real‑time monitoring and alerting of system and application metrics. You will lead research initiatives for cloud native designs and implementation within public and private clouds and will utilize TSDBs and correlation and data fusion of multiple data types and heterogeneous data streams coupled with artificial intelligence (AI) and learned behaviors for anomaly detection and forward projections of system and application expected behaviors. This role will involve collaboration with enterprise architects, product managers, data scientists, engineers, and business managers to bring telemetry R&D projects into production. You will use a combination of open source and COTS technologies to solve real‑time telemetry problems at an enterprise‑wide scale. In parallel, you will lead the design of new systems and the redesign of existing systems to meet business requirements, changing needs, and integration of state‑of‑the‑art technology. You will be an evangelist for the observability foundation socialization technology designs and implementations to engineering and business customers. What You'll Do Design and architect cloud‑native observability solutions, mandating both system‑wide telemetry pipelines and real‑time monitoring capabilities. Lead the technical vision and roadmap, socializing designs with internal and external stakeholders. Execute large‑scale distributed system projects, ensuring scalability, latency, and fault‑tolerance. Drive research into AI‑enhanced anomaly detection and predictive analytics. Mentor and guide engineering teams, fostering best practices and technical excellence. Minimum Qualifications BS/MS in Computer Science, Engineering, or equivalent, with 10+ years in software engineering, design, and architecture. Deep understanding of the Java language and associated frameworks, and previous development of Java applications, libraries, SDKs, or services. Strong architecture leadership with demonstrated enterprise‑level software implementations. Previous architectural leadership in research, evaluation, creation of software designs, and distributed software implementations in production. Experience with technical leadership, software roadmaps, research and development, new software initiatives, and customer and engineering coordination and engagement. Full‑stack cloud software development experience. Option 1: Bachelor’s degree in computer science, computer engineering, software engineering, or related area and 5 years’ experience in software engineering or related area. Option 2: 7 years’ experience in software engineering or related area. Experience with the Following API/lib/SDK development, integration, and utilization. Cloud technologies and cloud native designs. Cloud infrastructures and technologies such as OpenStack, Azure, GCP, or AWS. Large‑scale distributed systems experience including scalability and fault tolerance. TSDBs (InfluxDB, Kairos, Cortex, Thanos, Prometheus) or equivalent. Extract, transform, and load (ETL) processes. Real‑time telemetry pipelines and publish/subscribe models (Kafka or equivalent). Data warehousing, datalakes, processing, and data analytics. SQL (AzureSQL, Postgres, or equivalent) with a solid foundation in advanced SQL. Unix/Linux shell scripting or similar programming/scripting knowledge. Real‑time monitoring and alerting: metric agents, real‑time dashboards, alerting rules. Excellent written and verbal communication skills for diverse audiences based on engineering subject matter. Ability to document requirements, architectural designs, and analysis findings in both business and technical terminology. Software development in an Agile iterative CI/CD development environment. Promote and support company policies, procedures, mission, values, and standards of ethics and integrity. Preferred Qualifications Knowledge and/or use of agentic AI – Model Context Protocol (MCP) servers, large language models (LLMs), retrieval‑augmented generation (RAG), natural language processing (NLP). Fluency in Python, JavaScript, advanced shell scripting, configuration management (Ansible, Chef, Puppet). Experience with application performance monitoring (APM) and/or distributed tracing. Deployment of Kubernetes, containers, service meshes, and microservices. Microservices architectures, Istio, and Micrometer. OpenTelemetry standards and protocols. Go development. Master’s degree in computer science, computer engineering, computer information systems, software engineering, or related area with 3 years’ experience. Background in creating inclusive digital experiences, implementing Web Content Accessibility Guidelines (WCAG) 2.2 AA standards, assistive technologies, and integrating digital accessibility seamlessly. Knowledge of accessibility best practices and a commitment to Walmart’s accessibility standards and guidelines for supporting an inclusive culture. Benefits Walmart offers competitive pay, performance‑based bonuses, and a range of benefits including medical, vision, and dental coverage; 401(k), stock purchase, company‑paid life insurance; paid time off (PTO), parental leave, family care leave, bereavement, jury duty, voting, short‑term and long‑term disability, company discounts, Military Leave Pay, adoption and surrogacy expense reimbursement, and more. PTO and/or PPTO may be used for vacation, sick leave, holidays, or other purposes. Benefits are subject to change and may depend on job classification and length of employment. For more information, see Compensation The annual salary range for this position is $143,000.00 – $286,000.00. Additional compensation may include annual or quarterly performance bonuses and stock. Eligibility & EEO Statement Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. These benefits are subject to change and may be subject to a specific plan or program terms. Walmart and its subsidiaries are committed to maintaining a drug‑free workplace and have a no‑tolerance policy regarding the use of illegal drugs and alcohol on the job. Walmart, Inc. is an Equal Opportunity Employer‑By Choice. We believe we are best equipped to help our associates, customers, and the communities we serve live better when we really know them. That means understanding, respecting, and valuing diversity—unique styles, experiences, identities, abilities, ideas, and opinions—while being inclusive of all people. #J-18808-Ljbffr Walmart