Senior Machine Learning Engineer, Inference

Responsibilities

• Optimize realtime machine learning inference using modern serving frameworks and model-acceleration techniques.
• Profile and improve high-performance inference systems running on NVIDIA GPUs.
• Build distributed, multi-GPU and multi-node serving infrastructure capable of handling thousands of concurrent connections.
• Use Kubernetes, Ray, and custom load balancing to scale and operate inference workloads reliably.
• Take models from research through containerization, serving optimization, and stable production deployment.
• Own ambiguous performance, latency, throughput, and reliability problems from investigation through implementation.

Requirements

• Deep understanding of inference-serving frameworks and techniques such as vLLM or TRT-LLM.
• Hands-on experience with quantization, distillation, caching strategies, continuous batching, paged attention, and speculative decoding.
• Proficiency in C++, CUDA, Rust, or highly optimized Python, including the ability to profile and optimize code for NVIDIA GPUs.
• Experience with Kubernetes, Ray, custom load balancing, multi-GPU or multi-node inference, and high-concurrency systems.
• Non-trivial systems programming projects, major inference-engine open-source contributions, or deep-dive technical write-ups.
• PhD in computer science, physics, or mathematics, or equivalent practical experience building backend or machine learning systems.
• Ability to work independently in ambiguous environments and take systems from research through reliable production operation.

Benefits

• Full-time position with a base salary of $250,000-$300,000 plus bonus, equity, and benefits.
• Relocation assistance is offered.
• In-person work in the Mountain View office.