Senior Research Engineer, LLM Training & Post-Training

Salary: $165,000 - 310,000 per year Requirements:

• We need substantial experience training, fine-tuning, evaluating, and optimizing transformer-based language models with PyTorch.
• We expect hands-on experience with modern LLM training and post-training methods such as continued pretraining, supervised fine-tuning, reinforcement learning from human feedback, preference optimization, reward modeling, or related approaches.
• We look for a strong grasp of distributed training and multi-GPU systems, including ways to improve performance, scale, and efficiency.
• We require solid software engineering fundamentals, with the ability to build production-grade Python software and research tooling.
• We want experience designing experiments, measuring model performance, and troubleshooting complex training or optimization problems.
• We value excellent communication and collaboration skills, including effective partnership across research, product, infrastructure, and customer-facing work.
• We need someone comfortable operating in fast-paced, ambiguous environments where priorities shift over time.
• We require a masters degree, PhD, or equivalent industry experience in Machine Learning, AI, Computer Science, or a related field.
• Experience with DeepSpeed, FSDP, Megatron-LM, Hugging Face Transformers, TRL, PEFT, Lightning Fabric, or similar frameworks is strongly preferred.
• Familiarity with CUDA, Triton, vLLM, SGLang, TensorRT, or other AI systems and performance tuning tools is a plus.
• Background in GPU performance optimization, mixed precision, memory optimization, or distributed training optimization is advantageous.
• Open-source contributions, research publications, or experience supporting large-scale training or inference workloads are highly desirable.
• Startup experience or work on highly cross-functional engineering teams is considered an asset.

• We design, build, and refine training and post-training pipelines for large language models.
• We improve model quality through supervised fine-tuning, continued pretraining, preference optimization, reinforcement learning, evaluation, and experimentation.
• We develop and enhance PyTorch-based training infrastructure, tooling, and developer workflows.
• We optimize distributed training across multi-GPU environments by improving throughput, memory usage, scalability, and GPU utilization.
• We investigate difficult model training issues such as convergence problems, instability, communication overhead, and performance bottlenecks.
• We create evaluation methods, benchmark models, analyze failure cases, and steer model improvements through experimentation.
• We work directly with customers to understand real-world workloads and turn those insights into platform improvements.
• We partner closely with research, infrastructure, and platform engineering teams to deliver production-ready AI systems.
• We contribute to open-source projects through new features, tooling enhancements, documentation, and community involvement.

• AI
• CUDA
• Fabric
• Fine-tuning
• Support
• LLM
• Machine Learning
• Model Training
• PyTorch
• Python
• vLLM
• LESS
• Security

More:

We are Lightning AI, the company behind PyTorch Lightning. Founded in 2019, we build an end-to-end platform for developing, training, and deploying AI systems, and through our merger with Voltage Park, we combine developer-first software with large-scale, cost-efficient compute. We support solo researchers, startups, and large enterprises from offices in New York City, San Francisco, Seattle, and London, with a hybrid setup requiring at least two in-office days per week in San Francisco, Seattle, NYC, or London, while fully remote candidates outside those hubs may also be considered. Our team values urgency, ownership, open communication, strong team building, continuous growth, and long-term thinking. We offer a competitive rewards package that includes equity, bonus potential, comprehensive health coverage, retirement support, flexible time off, winter break, parental leave, learning support, wellness stipends, sabbatical leave, flexible schedules, and in-office meals. We are committed to building an inclusive workplace where diverse teams can thrive and contribute fully.

last updated 33 week of 2026