Sr. Software Development Engineer, Inference

Salary: $168,100 - 261,500 per year Requirements:

• We require at least 5 years of full software development lifecycle experience, including coding standards, code reviews, source control, build processes, testing, and operations.
• We require a bachelors degree in computer science or an equivalent qualification.
• We require at least 5 years of programming experience in a modern language such as Java, C++, or C#, with object-oriented design experience.
• We require a strong grasp of machine learning models, their architecture, and the training and inference lifecycle, along with hands-on work improving model performance.
• We prefer a masters degree in computer science or a related discipline.
• We prefer practical experience with PyTorch or JAX, especially for developing and deploying LLMs in production on GPUs, AWS Neuron, TPUs, or other AI accelerator hardware.
• We prefer experience contributing to or extending inference frameworks such as vLLM or SGLang.
• We prefer kernel development experience, such as CUDA or Triton.

• We lead customization and optimization of open-source inference frameworks like vLLM and SGLang on AWS Neuron.
• We own core framework logic, including scheduling and model execution, to improve large-scale inference serving.
• We apply advanced techniques in kernel development, parallel computation, distributed KV cache, speculative decoding, and systems engineering to deliver high-performance LLM serving.
• We evaluate emerging inference research and turn it into production-ready features.
• We influence the technical roadmap through design leadership and mentorship.
• We collaborate closely with model development, compiler, runtime, and performance engineering teams to ensure end-to-end model performance.

• AI
• AWS
• C#
• Cloud
• CUDA
• Hardware
• Java
• LLM
• Machine Learning
• PyTorch
• vLLM
• DevOps
• Support

More:

AWS Neuron is the full software stack for AWS Inferentia and Trainium, our purpose-built accelerators for cloud-scale machine learning. We are hiring for our Machine Learning Inference Applications team, where we focus on delivering production-ready, high-performance model inference for customer workloads on Inferentia- and Trainium-powered instances. Our team values knowledge-sharing, mentorship, and career growth, with supportive senior engineers, one-on-one guidance, and thoughtful code reviews. We offer comprehensive benefits, including health insurance, 401(k) matching, paid time off, parental leave, sign-on payments, and RSUs, and the role includes competitive base salary ranges depending on location.

last updated 37 week of 2026