Machine Learning Engineer - Training Systems

At Rhoda AI, we're building the full-stack foundation for the next generation of humanoid robots — from high-performance, software-defined hardware to the foundational models and video world models that control it. Our robots are designed to be generalists capable of operating in complex, real-world environments and handling scenarios unseen in training. We work at the intersection of large-scale learning, robotics, and systems, with a research team that includes researchers from Stanford, Berkeley, Harvard, and beyond. We're not building a feature; we're building a new computing platform for physical work — and with over $400M raised, we're investing aggressively in the R&D, hardware development, and manufacturing scale-up to make that a reality. We're hiring a Staff/Principal ML Systems Engineer to own training performance end-to-end and turn our training platform into a high-efficiency, high-reliability engine for research iteration. What You'll Do Own performance at scale Diagnose and improve end-to-end training performance for large models trained on multimodal robotic data (vision, proprioception, actions, language, video) Build a repeatable workflow for performance attribution: step-time breakdown (compute vs collectives/communication vs dataloader vs checkpointing), scaling curves and bottleneck identification at different GPU counts Drive measurable gains in: Distributed efficiency (overlap, bucket sizing, rank/topology mapping, parallelism strategy) Compute efficiency (kernel hotspots, fusion, attention performance, framework overhead) Memory efficiency (activation checkpointing, packing/bucketing, reduced padding waste) Make performance observable and durable Create "source of truth" metrics and dashboards for both: per-job performance ("why is this run slow?") and fleet-wide performance ("where are we losing GPU-hours this week?") Build automated performance regression detection: microbenchmark suite per model family, CI/perf gates or lightweight canary runs, "golden configs" and standard launch templates Partner deeply with researchers (no silos) Work closely with researchers and research engineers to translate model changes into scalable implementations Provide guidance on training strategy tradeoffs relevant to robotics world models (sequence lengths, rollout/eval cadence, variable-length multimodal data, etc.) Reduce the operational burden on researchers so they can focus on model quality and robotic behavior Collaborate on cluster efficiency (as part of the infra team) Partner with infra/SRE to reduce wasted GPU-hours from: Stragglers and degraded nodes Network health issues Checkpoint stalls and storage bottlenecks Scheduler placement issues for large distributed jobs What We're Looking For Significant experience delivering distributed training performance improvements in production research environments (large-scale GPU training strongly preferred) Strong hands-on experience with modern training stacks (e.g., PyTorch; familiarity with JAX a plus) Deep understanding of distributed training concepts and tradeoffs: sharded training (FSDP/ZeRO-style), tensor/pipeline parallelism, gradient accumulation, comm/compute overlap, and diagnosing and improving collective communication performance Strong debugging and measurement instincts: you can turn ambiguous "it's slow" into a clear bottleneck + experiment plan + validated fix Comfortable operating in a fast-moving startup environment with high ownership and minimal bureaucracy Nice to Have (But Not Required) Experience with GPU kernel-level performance work (CUDA/Triton), fused ops, compiler/graph capture Experience with multimodal/video training and variable-length sequence packing/bucketing Experience building observability systems for ML training (metrics/logs/traces + dashboards + alerting) Familiarity with large-cluster scheduling or topology-aware placement (Slurm/K8s/HPC environments) Why This Role Direct impact on model iteration speed — your work translates directly into faster research cycles and better robotic capability Work at the frontier of large-scale training for real-world robotics, not toy benchmarks Tight collaboration between systems, research, and infrastructure (no silos) High ownership in a small, ambitious team building foundational technology Meaningful leverage: improvements you make compound across every training run the research team executes #J-18808-Ljbffr Rhoda AI