Principal Software Engineer

Overview The HPC/AI (High Performance Computing and Artificial Intelligence) team is building the infrastructure that powers large-scale AI training and inference. As supercomputers grow across first-party (1P) and third-party (3P) GPU capacity, we need a consistent, trusted view of their health and operational state. We are seeking an AI Supercomputer Observability Architect to define and deliver the end-to-end observability platform for this fleet. You will drive near-real-time telemetry from GPUs, hosts, networks, storage, and other critical components; connect those signals to resource identity and topology; and turn an ocean of data into actionable, AI-assisted intelligence. The goal is to help customers and operators identify impact, find likely causes, enable datacenter technicians to diagnose and repair issues quickly, verify each repair with immediate telemetry feedback, restore service faster, and use GPU capacity more effectively. This is a horizontal architecture role. You will align infrastructure, platform, operations, and partner teams on common contracts and a converged 1P/3P experience, while accounting for differences in data availability, quality, and access. You will move the work from architecture through production validation, with a focus on scalability, reliability, security, and measurable customer outcomes. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees, we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Responsibilities

• Lead the architecture for near-real-time telemetry ingestion, processing, storage, and query across GPUs, hosts, InfiniBand, Ethernet, NVLink, storage, power, cooling, and related supercomputer components.
• Define shared signal contracts and measurable standards for freshness, completeness, correctness, availability, retention, access, and cost; validate the end-to-end pipeline under production-scale conditions.
• Connect telemetry to a common resource identity, topology, and health model so users can navigate from fleet and cluster health to racks, nodes, GPUs, switches, and links, and understand dependencies and blast radius.
• Drive convergence of 1P and 3P GPU capacity telemetry into a consistent operational model and experience, making provider-specific gaps and differences in signal quality explicit.
• Build the foundations for AI-assisted anomaly detection, cross-signal correlation, likely-cause analysis, impact assessment, and actionable recommendations; establish evaluation and feedback loops so insights are trustworthy.
• Partner with customers, datacenter technicians, service owners, and engineering DRIs to define persona-appropriate workflows and secure, role-based views of the same underlying state.
• Build practical tools and dashboards for datacenter technicians that surface near-real-time component health, likely fault location, repair context, and the next diagnostic step, so technicians can act quickly at the rack or node.
• Create a fast repair-validation loop: show technicians whether a repair changed the relevant signals, whether the component and affected capacity recovered, and what remains unhealthy so they can iterate until the problem is resolved.
• Lead technical design reviews, identify cross-team dependencies, establish reusable interfaces and patterns, and guide teams from proof of concept through incremental production rollout.
• Define success measures such as signal freshness, coverage, time to diagnose, time to repair, repair-validation latency, repeat repair attempts, recovery time, and the ability to connect infrastructure events to affected GPU capacity and customer impact.
• Mentor engineers and influence teams across organizational boundaries; contribute to operational playbooks and continuously improve the platform based on production learning.

Qualifications

Required Qualifications:

• Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
• OR equivalent experience.

Other Requirements:

• Ability to meet Microsoft, customer and/or government security screening requirements are required for this role.
• These requirements include, but are not limited to the following specialized security screenings:
• Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.

Preferred Qualifications:

• Master's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
• OR Bachelor's Degree in Computer Science or related technical field AND 15+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
• OR equivalent experience.
• Experience architecting large-scale telemetry, streaming-data, or observability systems, including ingestion, processing, storage, query, real-time analytics, time-series data, topology or graph models, and fleet-scale visualization.
• Experience leading cross-team technical design and delivery of production services, with attention to reliability, security, and operational readiness.
• Experience with GPU infrastructure, AI training or inference systems, high-performance computing, or datacenter operations.
• Knowledge of InfiniBand, Ethernet, NVLink, host and accelerator telemetry, and the dependencies among compute, networking, storage, and facility systems.
• Experience applying machine learning or AI to operational data, including anomaly detection, correlation, root-cause assistance, and evaluation of insight quality.
• Experience integrating telemetry from multiple providers or environments while maintaining common identity, schema, access-control, and data-quality standards.
• Ability to translate complex infrastructure signals into clear customer impact and technician-ready repair workflows, with measurable improvements in diagnosis, repair validation, recovery, and GPU utilization.

#azurecorejobs Software Engineering IC6 - The typical base pay range for this role across the U.S. is USD $165,600 - $296,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $220,800 - $331,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.