AI Inference Solutions Developer

Position: AI Inference Solutions Developer Location: San Francisco Bay Area or Remote Duration: Long-term Job Summary: We are seeking a highly skilled AI Inference Solutions Developer to design, build, and optimize a scalable, hardware-agnostic LLM inference platform across heterogeneous accelerators, including GPUs and other AI processors. The role focuses on disaggregated inference, KV cache management, cache-aware routing, and high-performance multi-node inference using open-source engines such as vLLM or SGLang. The ideal candidate will have deep expertise in distributed systems, RDMA networking, Kubernetes, accelerator programming, and performance optimization. This is a hands-on engineering role involving architecture, development, production deployment, and performance validation. Must-Have Skills: LLM Inference Engines: Deep hands-on experience with vLLM or SGLang internals, including scheduling, KV cache management, prefix caching, and connector APIs. Disaggregated Inference: Understanding of prefill/decode separation, KV cache transfer, and multi-tier caching architectures. Multi-Node Accelerator Workloads: Production experience with NCCL or equivalent technologies, tensor parallelism, expert parallelism, and kernel/network-level debugging. Programming: Strong Python development skills, along with production-quality experience in Go, C++, or Rust. RDMA Networking: Hands-on experience with RoCEv2 or InfiniBand networking. Kubernetes: Strong systems-level knowledge of Kubernetes and multi-node inference deployments. Accelerator Programming: Experience with CUDA, ROCm, Triton, or vendor-specific accelerator SDKs, including understanding differences between accelerator architectures. Distributed Systems: Strong knowledge of stateful distributed systems, caching, scheduling, and scalable service architecture. Performance Engineering: Experience with profiling, benchmarking, bandwidth optimization, latency analysis, and performance troubleshooting. Cache-Aware Routing: Knowledge of prefix-aware routing, cache-aware request scheduling, and service-level objective (SLO) admission. Agentic Development: Regular hands-on use of AI coding tools such as Claude Code, with strong code review and debugging capabilities. Software Engineering Practices: Strong testing, CI/CD, observability, and clean, versioned API development. Technical Communication: Ability to analyze performance measurements, make architecture decisions, and document technical findings clearly. Responsibilities: Design, develop, and optimize disaggregated LLM inference serving using vLLM or similar open-source engines, including prefill/decode pools, expert parallelism, and KV cache transfers. Build multi-tier KV cache management solutions, including connector integrations, cache offloading, storage policies, and shared storage admission. Develop cache-aware routing capabilities with prefix-aware scoring, hash consistency, and SLO-based request admission using open-source routing frameworks. Design accelerator and engine adapter interfaces, build conformance tests, and deploy scalable multi-node inference workloads on Kubernetes. Optimize inference performance through benchmarking, profiling, and validation of throughput, cache hit rates, tenant isolation, and failover. Troubleshoot accelerator and network-level issues, contribute improvements to open-source projects, and support production deployment and infrastructure lifecycle operations. #J-18808-Ljbffr