Sr. Software Engineer, Training/Inference

Responsibilities

• Architect and implement distributed model-serving infrastructure for load balancing, auto-scaling, batch scheduling, and global KV caching.
• Optimize model inference latency, throughput, concurrency, reliability, and tail latency at production scale.
• Benchmark, fine-tune, and accelerate inference engines through GPU kernel work and code generation.
• Develop tools to trace, replay, and resolve issues across orchestration and GPU-kernel layers.
• Build CI/CD infrastructure for endpoint deployment, image publishing, and inference engine updates.
• Advance research on test-time compute scaling, RL rollout, and model-hardware co-design.

Requirements

• Bachelor’s degree in computer science, computer engineering, or a science discipline.
• 5+ years of low-level systems programming experience in C/C++ or Rust.
• Experience with large-scale, highly concurrent production serving.
• Experience with GPU inference engines such as vLLM, SGLang, Triton, or TensorRT-LLM.
• Strong background in batching, caching, load balancing, parallelism, GPU kernels, and code generation.
• Experience with quantization, speculative decoding, distillation, low-precision numerics, testing, benchmarking, reliability, and CI/CD for inference systems.

Benefits

• $200,000–$375,000 base salary, plus equity.
• Comprehensive medical, vision, and dental coverage.
• 401(k) retirement plan.
• Short- and long-term disability insurance and life insurance.
• Various discounts and perks.