Software Development Manager, LLM Inference

Salary: $212,700 - 287,700 per year Requirements:

• 3 years of experience managing engineering teams
• 7 years of experience working directly in engineering teams
• 3 years of experience designing or architecting new and existing systems, including design patterns, reliability, and scaling
• Experience partnering with product or program management teams
• Strong background in LLM model architectures, model performance optimization, and inference techniques
• Experience delivering high-performance models using distributed inference libraries
• Ability to operate effectively in a fast-changing environment with shifting priorities
• Strong technical understanding of a vertically integrated system stack, including the PyTorch inference library, Neuron compiler, runtime, and collectives
• Experience communicating with users, technical teams, and senior leadership to gather requirements and explain technical designs, product features, and strategy
• Experience recruiting, hiring, mentoring, coaching, and managing software engineering teams to strengthen their effectiveness

• Lead a team of AI/ML engineers to onboard and optimize open-source and customer LLMs, including dense and MoE models, for inference on Trainium accelerators
• Drive improvements in model enablement speed and user experience
• Advance inference usability and quality through features, infrastructure optimization, tools, and automation
• Manage project plans and deliver against commitments
• Oversee the day-to-day activities of the engineering team
• Handle resource planning, staffing, mentoring, and team development
• Report development status, quality, operations, and model performance to management
• Work with senior management and technical leaders to define model enablement and performance optimization for the latest SOTA LLMs
• Build and deliver optimized models to customers
• Help the team solve complex technical challenges
• Adapt team priorities as new models and technologies emerge

• AI
• AWS
• Cloud
• Hardware
• LLM
• Machine Learning
• PyTorch
• Support

More:

We develop AWS Neuron, the complete software stack for Trainium, our custom cloud-scale machine learning accelerators. We are hiring an SDM for the LLM Inference Model Enablement team to lead expert AI/ML engineers focused on optimizing large language models for fast inference on Trainium hardware. You will help improve the onboarding experience for models and enhance inference usability and quality for Neuron-supported models. We offer comprehensive benefits, including health coverage, 401(k) matching, paid time off, parental leave, sign-on payments, and restricted stock units. The role is based in Cupertino, California, with a listed annual base salary range of 212,700 to 287,700 USD.

last updated 37 week of 2026