Senior Distributed Systems Engineer

About the Institute of Foundation Models

The Institute of Foundation Models (IFM) designs and operates ultra-scale GPU supercomputing systems to train next-generation foundation models. We believe performance, fault tolerance, and scalability are co-designed across model architecture, communication systems, runtime, and hardware topology.

This role sits at the core of that effort - driving communication performance, distributed reliability, and cross-layer optimization for large-scale training workloads.

The Mission

We are looking for a deeply technical engineer to co-design and optimize the communication stack for large-scale distributed training, including hybrid parallelism and Mixture-of-Experts (MoE) workloads.

This is not a network operations role. This is a systems-level engineering position focused on performance engineering, distributed debugging, and communication-runtime co-design.

• Design and optimize expert-parallel and hybrid-parallel communication patterns

• Drive high-performance hierarchical collectives for MoE workloads

• Co-design runtime orchestration with communication topology awareness

• Reduce tail latency and improve determinism across thousands of GPUs

• Architect fault-tolerant distributed execution under real-world cluster failures

Core Technical Scope

• Communication-compute overlap and topology-aware collective optimization

• Deep debugging of NCCL, RDMA, and custom communication layers

• Hybrid expert parallel strategies in modern large-scale MoE systems

• Elastic and resilient distributed job orchestration concepts

• Congestion analysis and routing optimization across InfiniBand/RoCE fabrics

• Microbenchmarking and performance modeling for communication-heavy workloads

Expected Technical Depth

• Hybrid expert parallel communication for Mixture-of-Experts training

• Scaling behavior under network pressure

• Distributed orchestration for elastic, large-scale training

• Fault detection and recovery in distributed GPU workloads

• Cross-layer bottlenecks: GPU NIC PCIe NVSwitch Fabric Scheduler

Required Background

• Experience optimizing distributed training at 1,000+ GPU scale (or equivalent depth)

• Hands-on expertise with RDMA, InfiniBand, RoCE, and GPUDirect RDMA

• Deep familiarity with NCCL and/or UCX internals

• Strong systems programming ability (C/C++, Rust, or Go)

• Strong familiarity with modern model training frameworks such as PyTorch

• Ability to troubleshoot and profile training performance issues related to communication bottlenecks

• Ability to translate research ideas into production-grade optimizations

• Experience debugging distributed hangs, desynchronization, and performance regressions

What We Mean by "Hardcore"

• You can explain why an communication degrades at scale and how to fix it

• You have improved real cluster throughput via communication redesign

• You can trace a distributed hang across ranks and identify the root cause

• You are comfortable working at the boundary between hardware and runtime

Application Requirements

• Include a link to your GitHub (required)

• Provide links to relevant distributed systems, HPC, or large-scale training projects

• Include a list of publications and/or public technical reports (if applicable)

• Describe the hardest distributed debugging problem you solved

• Include measurable performance improvements you have delivered

Academic Qualifications

Master's, or Bachelor's + 1 year of relevant experience.

$200,000 - $400,000 a year

Visa Sponsorship

This position is eligible for visa sponsorship.

Benefits Include

*Comprehensive medical, dental, and vision benefits

*Bonus

*401K Plan

*Generous paid time off, sick leave and holidays

*Paid Parental Leave

*Employee Assistance Program

*Life insurance and disability