AI Infrastructure — Training Engineer (Large Model) [33251]

Responsibilities

• Distributed training framework optimization. Own the R&D and tuning of distributed training frameworks for large models (LLMs, multimodal), resolving scalability bottlenecks at the scale of 10k–100k GPU clusters.
• Kernel & performance tuning. Work close to the underlying hardware (NVIDIA GPU / NPU) on kernel acceleration, memory optimization, and communication optimization (tensor parallelism, pipeline parallelism, ZeRO, and related techniques).
• System resilience & scheduling. Build stable large-scale training clusters; design high-availability fault-tolerance mechanisms (checkpoint/resume, automatic recovery) and compute scheduling strategies to raise overall cluster throughput and resource utilization.
• Training pipeline engineering. Build an end-to-end MLOps platform spanning data preprocessing, distributed training, model fine-tuning (RLHF / DPO, etc.), and automated evaluation.

Qualifications

• Education. Bachelor's degree or above in Computer Science, Software Engineering, Electrical Engineering, or a related field.
• Programming. Very strong engineering implementation skills; proficient in C/C++ and Python, with a solid foundation in data structures and algorithms.
• Distributed & parallel computing. Hands-on mastery of mainstream distributed training frameworks such as PyTorch, Megatron-LM, DeepSpeed, DeepSpeed-Chat, or Horovod.
• Low-level systems & communication. Familiar with Linux internals, the network stack (RoCE/RDMA), GPU communication primitives (e.g., NCCL), and common storage systems.
• Tuning & debugging. Skilled with profiling and debugging tools such as Nsight, GDB, and PyTorch Profiler; able to quickly diagnose cluster deadlocks, performance bottlenecks, and out-of-memory (OOM) issues.