Senior Site Reliability Engineer

Responsibilities

• Own the stability, performance optimization, and issue resolution of large-scale generative AI training and inference systems.
• Design and improve resource scheduling systems, including utilization, prioritization, fine-grained scheduling, and scheduling explainability.
• Plan, deploy, operate, and optimize high-speed RDMA networks and heterogeneous GPU, NPU, and XPU clusters.
• Own the architecture, selection, operation, performance, reliability, and stability of high-performance storage systems.
• Build capacity planning methods and deliver reliable, timely, and cost-effective resource expansion and configuration optimization.
• Develop unified compute pools for co-located training and inference workloads and establish operating methods for compute efficiency and cost control.
• Build monitoring, alerting, and automated remediation across the training and inference path and resolve complex production incidents.
• Partner with algorithm, product, and operations teams on MLOps solutions and establish engineering practices for large-scale generative AI systems.

Requirements

• Bachelor's degree or above in Computer Science, Software Engineering, or a related field.
• 3+ years of experience in large-scale machine learning, generative AI MLOps, or closely related infrastructure work.
• Proficiency in Python and/or Go, strong Linux development and operations skills, and expert shell scripting and automation capabilities.
• Expertise with Docker or Kata containerization and Kubernetes orchestration, including hands-on operation and optimization of large-scale heterogeneous GPU or NPU clusters.
• Knowledge and practical optimization experience with RDMA high-speed networking and distributed or object storage.
• Expertise designing and optimizing resource scheduling systems and improving scheduling policies and utilization.
• Experience maintaining and optimizing large-scale generative AI training or inference systems and resolving issues across the full training and inference path.
• Experience with co-located training and inference compute pools or offline inference efficiency, plus strong communication, collaboration, ownership, and production operations skills.
• Preferred qualifications include productionizing large-scale generative AI models; managing heterogeneous clusters and capacity planning in multi-cloud or hybrid-cloud environments; tuning distributed training and inference systems; optimizing RDMA or storage performance; delivering compute capacity and operating frameworks with measurable cost or utilization improvements; designing distributed systems; and running fault-injection or chaos exercises.