Sr. Software Dev Engineer, Inference Team

Salary: $168,400 - 193,500 per year Requirements:

• We require 5+ years of full software development lifecycle experience, including coding standards, code review, source control management, build processes, testing, and operations
• We require a bachelors degree in computer science or an equivalent background
• We require 5+ years of programming experience in a modern language such as Java, C++, or C#, along with object-oriented design experience
• We require a solid foundation in machine learning models, including architecture, training and inference lifecycles, plus hands-on work improving model performance
• We prefer a masters degree in computer science or a related discipline
• We prefer practical experience with PyTorch or JAX, especially developing and deploying LLMs in production on GPUs, AWS Neuron, TPUs, or similar AI accelerator hardware
• We prefer experience extending or contributing to inference frameworks such as vLLM or SGLang
• We prefer kernel development experience, such as CUDA or Triton

• We lead the customization and performance tuning of open-source inference frameworks like vLLM and SGLang on AWS Neuron
• We optimize core framework behavior, including scheduling and model execution, to improve large-scale LLM serving efficiency
• We apply advanced techniques in kernel development, parallel computation, distributed KV cache, speculative decoding, and systems engineering to deliver strong inference performance
• We evaluate emerging inference research and turn it into production-ready capabilities
• We help shape the technical roadmap for the team
• We provide design leadership and mentorship to raise engineering standards
• We collaborate closely with model development, compiler, runtime, and performance engineering teams to ensure end-to-end model performance

• AI
• AWS
• C#
• Cloud
• CUDA
• Hardware
• Java
• LLM
• Machine Learning
• PyTorch
• vLLM
• DevOps
• Support

More:

AWS Neuron is the complete software stack for AWS Inferentia and Trainium, AWS purpose-built accelerators for cloud-scale machine learning. This senior software engineering role sits on the Machine Learning Inference Applications team, where we focus on delivering high-performance model inference solutions for customer workloads running on Inferentia- and Trainium-powered instances. We are a team that supports new members, values knowledge-sharing and mentorship, and includes a broad mix of experience levels and tenures. Our senior members enjoy one-on-one mentoring and thoughtful code reviews, and we invest in career growth by assigning projects that help team members build expertise and take on more complex work over time. We also offer comprehensive benefits, including health coverage, 401(k) matching, paid time off, parental leave, sign-on payments, and restricted stock units, with compensation varying by experience, qualifications, and location.

last updated 37 week of 2026