Staff Software Engineer - ML Infrastructure

ML Infrastructure Software EngineerExpand human ambition in the physical world.Critical infrastructure is constrained by labor shortages, hazardous working conditions, and operational complexity. Watney builds and deploys autonomous robotic systems that increase the speed and capacity of buildout, starting with data centers.About the RoleAt Watney, ML Infrastructure software engineers build the high-performance foundations that allow our perception and intelligence models to scale. You will architect the high-performance computing foundation that powers our physical intelligence models.In this role, you'll own the infrastructure required for large-scale multimodal training, which includes cluster orchestration, optimizing JAX-based pipelines that must ingest and stream video data, and transforming experimental architectures into reliable, highly distributed production training runs.This is a high-leverage systems role at the intersection of deep learning, advanced hardware acceleration, and scalable cluster infrastructure.What You'll DoOwn Training & Inference Infrastructure: Design and maintain multi-tenant scheduling systems that automatically place training and inference jobs based on hardware topology, cost, and priority, while enforcing fair resource sharing and preemption policies.Scale Distributed Training: Partner with researchers to scale JAX and PyTorch-based training loops across heterogeneous GPU/TPU clusters with minimal friction, ensuring rock-solid checkpointing and metrics collection.Optimize Performance & Hardware Bounds: Profile and improve memory usage, device utilization, throughput, and distributed synchronization, specifically navigating edge hardware bottlenecks like on-chip video decoders and memory bandwidth.Enable Rapid Iteration: Build clean abstractions for launching, monitoring, debugging, and reproducing experiments so researchers can submit massive jobs without needing to manage underlying cluster state.Contribute to Core Training Code: Evolve our core JAX model code and training pipelines to natively support new architectures, multimodal video/telemetry data streams, and robust evaluation metrics.Manage Compute Resources: Ensure highly efficient allocation and utilization of massive cloud-based compute clusters while aggressively monitoring and controlling resource costs.You May Be a Good Fit If You:Bring experience building machine learning platforms and large-scale distributed trainingPossess deep professional experience with distributed training backbones (FSDP, DeepSpeed, Megatron, Ray Train) or large-scale inference serving layers (vLLM, Triton, Ray Serve).Exhibit fluency in Python alongside Rust or C/C++, with a strong mathematical background and practical knowledge of GPU kernel optimization or network topologies.Have experience navigating structural edge-case hardware bottlenecks, specifically regarding video decoding, multimodal alignment, or high-throughput real-time playback.We're committed to building a diverse, inclusive team. At Watney Robotics, we welcome people of all backgrounds and identities, and we make hiring decisions based on skills, experience, and potential. If you're passionate about robotics but don't meet every requirement, we still encourage you to apply!