Staff GenAI Inference Engineer: Optimize LLM Serving Latency
$190.9k - $232.8kNeon
A leading data and AI company is seeking a Staff Software Engineer for GenAI inference to lead the architecture and optimization of the inference engine. The role requires expertise in CUDA, GPU programming, and distributed systems design. Ideal candidates will have a strong software engineering background and a proven ability to collaborate with researchers and drive architectural decisions. Competitive compensation is offered, with a salary range of $190,900 to $232,800 USD. #J-18808-Ljbffr Neon
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Staff GenAI Inference Engineer: Optimize LLM Serving Latency in San Francisco, CA vacancy
- Databricks is seeking a Staff Software Engineer for GenAI inference to drive the architecture, development, and optimization of their inference engine. The role involves high-throughput, low-latency solutions and full GenAI inference stack management. Candidates must have...Suggested
$190k - $265k
...business. Founded by engineers — and customer-... ...model training, model serving, and Vector Search... ...Foundation Model Inference team is the... ...serve, scale, and optimize frontier models with... ...you will have:Build LLM infrastructure powering... ...reliability, latency, and efficiency of...SuggestedLocal areaWorldwideDataBricks
San Francisco, CA6 days ago- Crusoe in San Francisco is seeking a Staff Technical Program Manager to lead the Managed Inference platform team. You will ensure end-to-end program delivery for LLM workloads, driving innovation in... ...a strong focus on AI and model serving. Join us at Crusoe and make a...Suggested
$190k - $265k
...business. Founded by engineers — and customer-... ...model training, model serving, and Vector Search... ...Foundation Model Inference team is the... ...serve, scale, and optimize frontier models with... ...will have: Build LLM infrastructure powering... ...reliability, latency, and efficiency of...SuggestedLocal areaWorldwideDatabricks Inc.
San Francisco, CA1 day ago$250k - $320k
(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure About The Role We're looking for a software... ..., diagnosing hero-model and LoRA-serving faults that don't show up in generic... ...that improve the performance, latency, throughput, and efficiency of our...SuggestedPermanent employmentFull timeTogether Computer Inc
San Francisco, CA14 hours ago$188k - $275k
...Of The Team The Inference team is... ...performance model serving capabilities that... ...improve throughput, latency, reliability, and... ...for an Applied AI Engineer to help us understand... ...driving targeted optimizations for both platform-... ...Familiarity with LLM inference systems...Permanent employmentFull timeTemporary workCasual workWork at officeFlexible hoursCoreWeave
San Francisco, CA1 day ago$250k - $300k
...means owning the inference stack end to end:... ...bringing modern optimization techniques into real... ...deep into the serving code when the defaults... ...patterns, latency targets, and cost... ...directly with customer engineering teams to tailor... ...with modern LLM serving frameworks...Temporary workCrusoe
San Francisco, CA6 days ago- ...design, build, and optimize the AI infrastructure... ...infrastructure for inference, evaluation, and... ...of infrastructure engineering, production ML systems, LLM/LVM inference, evaluation... ...performance across latency, throughput, GPU... ...AI. Improve model-serving architecture,...Full timeWork at officeLocal areaFlexible hours3 days per week
- ...mission-critical inference for the world's most... ...the platform engineers turn to ship AI products... ...large-scale LLM inference on Baseten... ...calling, and multimodal serving. Implement... ...teams to make new optimizations broadly available... ...where reliability, latency, and scale are first...Flexible hours
Baseten
San Francisco, CA3 days ago $200k - $250k
...vertical indexes with LLM-optimized retrieval systems to power... ...Our team includes engineers, researchers, product... ...through one layer: our inference gateway. It allows users... ...to optimize on cost , latency and high availability.... ...sits between model serving and product: routing and...Immediate startRemote workWork from homeFlexible hoursMixpeek
San Francisco, CA2 days ago$230k - $390k
...SoftBank — to transform how they serve customers and grow their... ...and act in real time. The Inference team builds the systems... ...efficient at scale.As a Software Engineer on Inference, you’ll help... ...capacity and quota, and optimizing for latency, reliability, and cost.This...Full timeFlexible hoursSierra
San Francisco, CA5 days ago- Anthropic in San Francisco is seeking a Cloud Inference Engineer to scale and optimize Claude across AWS, GCP, and Azure. You will own validation, deployment, and performance of inference servers and load balancers on multiple CSPs, ensuring correctness and reliability...
$186k - $269k
Build GenAI PoCs (LLMs, RAG, agentic frameworks) to... ...software solutions.Serve as engineering Lead delivering complex... ...safety, compliance, and latency.Minimum qualifications... ...experience building LLM evaluation (evals) pipelines... ..., observability, and optimizing LLM-native metrics at...Google
San Bruno, CA4 days ago- ...SoftBank — to transform how they serve customers and grow their... ...and act in real time. The Inference team builds the systems... ...at scale. As a Software Engineer on Inference, you’ll help define... ...capacity and quota, and optimizing for latency, reliability, and cost....Full timeFlexible hours
Candidate
San Francisco, CA1 day ago - ...committed researchers, engineers, policy experts, and business... ...the role The Cloud Inference team scales and optimizes Claude to serve the massive audiences of... ...baselines (throughput and latency), and correctness checks... ...a strong interest in LLM serving; prior inference...
$190.9k - $232.8k
...1285About This RoleAs a staff software engineer for GenAI inference, you will lead the architecture... ..., development, and optimization of the inference engine... ...high throughput, low latency, and robust scaling. Your... ...and collaborate on model-serving stack optimized for large...Local areaWorldwideDataBricks
San Francisco, CA6 days ago$300k - $400k
...Software Engineer, Inference The mission of Thinking Machines is to build... ...of the systems that serve our models to real users. Our... ...model versions, and inference optimizations, ensuring safe, incremental... ...Experience operating large-scale, latency-sensitive production...Visa sponsorshipWork visaRelocation packageThinking Machines Lab
San Francisco, CA1 day ago$190.9k - $232.8k
About This Role As a staff software engineer for GenAI inference, you will lead the architecture, development, and optimization of the inference engine that... ...high throughput, low latency, and robust scaling. Your... ...and collaborate on model‑serving stack optimized for large...Local area- DigitalOcean is seeking a Senior Engineer 2 to drive AI Inference Optimization. You will lead benchmarking and optimizations at the inference engine and GPU... ...CUDA/Triton expertise to push throughput and reduce latency across multi-node GPU clusters. #J-18808-Ljbffr DigitalOceanRemote job
$180k - $280k
...for training, evaluating and serving large language models used... ...tasks. What you'll do Build LLM training infrastructure Develop model serving systems Optimize inference performance Build distributed... ...serving experience Strong systems engineering skills Excellent debugging...Full time$180k - $320k
...hiring a Forward Deployed Engineer focused on Computer Use... ...stakeholders Serve as a trusted technical... ...evaluating, or deploying LLM‑based or agentic systems... ...Python and the modern GenAI/LLM stack ~ Demonstrated... ...learning, preference optimization, benchmarking, or model...Local areaSnorkel AI
San Francisco, CA1 day ago$202k
...Senior Software Engineer About the Role... ...backend and model-serving infrastructure that... ...personalization, and emerging LLM-based search... ...with strict latency and reliability requirements... ..., and modern LLM inference. You will... ...and retrieval, optimizing GPU inference and...Full timeWork at officeRemote workUber Technologies Inc.
San Francisco, CA3 days ago- ...growing group of committed researchers, engineers, policy experts, and business... ...systems. About the role The Cloud Inference team scales and optimizes Claude to serve the massive audiences of... ...container orchestration Are curious about LLM serving; prior inference or ML experience...
$218.4k - $365.2k
...Context Protocol (MCP), LLM Model Gateway, and Context Engineering capabilities.This... ..., retry rates, inference utilization, and... ...platforms, model-serving systems, model orchestration... ...of quality, latency, throughput,... ...quantization, inference optimization, or other ways to...Full timeSalesforce
San Francisco, CA2 days ago$140k - $193k
...oriented Applied AI Engineer to help develop,... ...model integration, inference services,... ...acquisition, model serving, monitoring, lifecycle... ..., and performance optimization. Learn quickly... ...learning models, LLM applications, computer... ...edge deployment, low-latency inference,...Work experience placementCasual workWork at officeRelocation package2 days per weekCHAOS Industries
San Francisco, CA1 day ago- ...build, train, and serve AI models... ...them to a working inference call fastest, and... ...are a working engineer: you ship code,... ...the Senior or Staff level and will... ...ve worked with LLM APIs in production... ...what inference latency, throughput,... ...Depth in inference optimization, serving...Permanent employmentFlexible hours
Fireworks Inc
San Francisco, CA4 days ago - ..., high-efficiency serving platform. Backed by... ...support from AMD engineers the team is scaling... ...AI models are optimized and deployed at scale... ...Design and build a low-latency, chat-like... ...engineering, open source inference engine like vLLM, Sglang, or TRT-LLM Streaming...Work at officeFlexible hours
Sciforium
San Francisco, CA3 days ago - ...in AI cloud infrastructure serving tens of thousands of customers... ..., with a focus on ultra-low-latency and high-bandwidth... ...operational visibility. Mentor engineers and cross-functional teams on... ...-to-GPU and storage traffic optimization. Expertise with BGP-based...Work at officeLocal areaWork from homeFlexible hours
Lambda
San Francisco, CA1 day ago $266k
...Software Engineer, Model Runtime Hardware - San... ...runtime within the inference engine that executes... ...layers of the cluster serving software stack,... ...efficient execution while optimizing for throughput, latency, utilization, and... ...Design and implement the LLM inference runtime...Work at officeLocal areaRelocation packageFlexible hoursOpenAI, Inc.
San Francisco, CA1 day ago- ...mission-critical inference for the world's... ...the platform engineers turn to ship AI... ...Language Model (LLM) engineering.... ...disaggregated serving). DevEx Improvement... ..."dev machines" optimized for model... ...InferenceMAX and genai-bench to... ...configuration (latency vs. cost vs. quality...Flexible hours
BaseTen
San Francisco, CA1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Staff GenAI Inference Engineer: Optimize LLM Serving Latency. Be the first to apply!
Related searches
- project engineer assistant project manager San Francisco, CA
- staff security engineer San Francisco, CA
- staff engineer San Francisco, CA
- research assistant engineering San Francisco, CA
- senior staff systems engineer San Francisco, CA
- engineering aide San Francisco, CA
- senior staff engineer San Francisco, CA
- assistant engineering manager San Francisco, CA
- assistant chief engineer San Francisco, CA
- staff data engineer San Francisco, CA