Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

LLM Inference Engineer (Mid, Sr, Staff)

Full-time

Hippocratic%20AI

Role Mission

As HAI's LLM Inference Engineer, you will own the serving infrastructure that determines whether our breakthrough healthcare AI reaches patients efficiently and reliably. You'll optimize the systems that translate raw model capability into sub-100ms responses—making the difference between conversational experiences that feel natural and those that feel broken. This role exists because inference optimization at scale is where research meets reality: your work directly determines latency, cost, and availability for millions of patient conversations across healthcare systems.

What You Will Accomplish

Own your first major outcome: By day 90, you will have shipped a measurable improvement to our inference serving stack (reduced latency, improved throughput, or optimized cost per inference), validated the gains across our production deployment scenarios, and established the performance optimization roadmap that will guide infrastructure investment.

Drive lasting impact: At 12 months, you will have designed and deployed advanced serving architectures (disaggregated inference, optimized caching, speculative decoding) that meaningfully improve patient experience and operational efficiency, contributed novel optimization techniques that become part of our core infrastructure, and made our serving stack a durable competitive advantage in healthcare AI deployment.

The Team

You'll work alongside systems engineers, ML researchers, and infrastructure experts who are obsessed with making AI systems fast, reliable, and cost-effective. This is a team that values deep technical rigor, continuous benchmarking, and solving hard systems problems that have real impact on patient experience and business unit economics.

What You'll Do

  • Design and implement multi-node serving architectures for distributed LLM inference

  • Optimize multi-LoRA serving systems

  • Apply advanced quantization techniques (FP4/FP6) to reduce model footprint while preserving quality

  • Implement speculative decoding and other latency optimization strategies

  • Develop disaggregated serving solutions with optimized caching strategies for prefill and decoding phases

  • Continuously benchmark and improve system performance across various deployment scenarios and GPU types

Location Requirement

We believe the best ideas happen together. This role is based in our Menlo Park, California office, expected to be in office five days a week.

Compensation

Compensation is based on experience, expertise, and level of responsibility. We offer competitive packages that reflect the seniority and scope of the role, along with equity, health insurance, and other benefits.

What You Bring

Must-Have:

  • Experience optimizing LLM inference systems at scale

  • Proven expertise with distributed serving architectures for large language models

  • Hands-on experience implementing quantization techniques for transformer models

  • Strong understanding of modern inference optimization methods, including:

    • Speculative decoding techniques with draft models

    • Eagle speculative decoding approaches

  • Proficiency in Python and C++

  • Experience with CUDA programming and GPU optimization

Nice-to-Have:

  • Contributions to open-source inference frameworks such as vLLM, SGLang, or TensorRT-LLM

  • Experience with custom CUDA kernels

  • Track record of deploying inference systems in production environments

  • Deep understanding of performance optimization systems

Show us what you've built: Tell us about an LLM inference or training project that makes you proud! Whether you've optimized inference pipelines to achieve breakthrough performance, designed innovative training techniques, or built systems that scale to billions of parameters - we want to hear your story.
Open source contributor? Even better! If you've contributed to projects like vllm, sglang, lmdeploy or similar LLM optimization frameworks, we'd love to see your PRs. Your contributions to these communities demonstrate exactly the kind of collaborative innovation we value.
Join a team where your expertise won't just be appreciated—it will be celebrated and amplified. Help us shape the future of AI deployment at scale!

References
1. Polaris: A Safety-focused LLM Constellation Architecture for Healthcare,
2 . Polaris 2:
3 . Personalized Interactions:
4 . Human Touch in AI:
5 . Empathetic Intelligence:

Why Join Hippocratic AI

Reinvent healthcare with AI that puts safety first. We’re building the world’s first healthcare‑only, safety‑focused LLM — a breakthrough platform designed to transform patient outcomes at a global scale. This is category creation.

Work with the people shaping the future. Hippocratic AI was co‑founded by CEO Munjal Shah and a team of physicians, hospital leaders, AI pioneers, and researchers from institutions like El Camino Health, Johns Hopkins, Washington University in St. Louis, Stanford, Google, Meta, Microsoft, and NVIDIA.

Backed by the world’s leading healthcare and AI investors. We recently raised a $126M Series C at a $3.5B valuation, led by Avenir Growth, bringing total funding to $404M with participation from CapitalG, General Catalyst, a16z, Kleiner Perkins, Premji Invest, UHS, Cincinnati Children’s, WellSpan Health, John Doerr, Rick Klausner, and others.

Build alongside the best in healthcare and AI. Join experts who’ve spent their careers improving care, advancing science, and building world‑changing technologies — ensuring our platform is powerful, trusted, and truly transformative.

Equal Opportunity

Hippocratic AI is an equal opportunity employer. We do not discriminate on the basis of race, color, religion, national origin, sex, age, disability, sexual orientation, gender identity or expression, genetic information, military or veteran status, or any other characteristic protected by applicable law. We are committed to building a team that reflects the patients we serve. We actively encourage applications from candidates of all backgrounds. If you require accommodations during the hiring process, please contact View email address on jobs.ashbyhq.com .

Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @ hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.

Vacancy posted 9 days ago
Similar jobs that could be interesting for youBased on the LLM Inference Engineer (Mid, Sr, Staff) in Menlo Park, CA vacancy
  •  ...ABOUT THE ROLE We’re seeking an inference systems engineer to build the distributed serving and high-performance networking layer behind our...  ...YOU’LL DO Design, implement, and operate disaggregated LLM serving architectures, including independent prefill and decode... 
    Suggested
    Full time
    Work at office

    Hippocratic%20AI

    Menlo Park, CA
    9 days ago
  • $159k - $198.8k

     ...AI as a transformative force for business. Job Description:Staff Inference Engineer -Uniphore is building an enterprise AI platform where agents...  ...serving• Production experience with vLLM, SGLang, or TensorRT-LLM• Strong Python and C++ (C++17/20)ti ability to read and... 
    Senior
    Full time

    Uniphore

    Palo Alto, CA
    2 days ago
  • $220k - $357k

     ...from people using those products shapes what we build next. SR. MODEL ENGINEER, RL INFERENCE The RL infrastructure team is looking for an engineer to...  ...of large-scale distributed systems Experience in LLM inference Strong knowledge in quantization and numerics... 
    Senior
    Temporary work

    SpaceXSI

    Palo Alto, CA
    19 days ago
  •  ...supporting the architecture and deployment of production-grade LLM systems for healthcare and public health impact. Commitment 5-1...  ...week Remote Flexible schedule Qualifications 5+ years software engineering experience 2+ years hands-on experience building LLM applications... 
    Senior
    Remote work
    10 hours per week
    Flexible hours

    Health4TheWorld

    Palo Alto, CA
    3 days ago
  • $193.3k - $261.5k

    We are looking for a Senior Inference Engineer to own inference for real-time multimodalconversational...  ...models that fall outside standard LLM serving patterns — sustained low-latencyoutput...  ...with other employees, supervisors, and staff; adhere to standards of excellence... 
    Senior
    Internship
    Local area
    Flexible hours

    Amazon

    Sunnyvale, CA
    1 day ago
  • $220k - $357k

     ...the intelligence of flagship models into flash models through synthetic data generation Optimize mid-training data mixtures to boost the ceiling for RL Engineer long-context data recipes Develop robust and diverse evaluation for mid-training checkpoints BASIC... 
    Senior
    Temporary work

    SpaceXSI

    Palo Alto, CA
    20 days ago
  •  ...designed for the next generation of AI workloads. The Senior/Staff Inference Reliability Engineer will own the end-to-end reliability and production...  ...sits at the intersection of inference platform engineering, LLM performance, and infrastructure reliability. You will... 
    Senior
    Full time

    parasail

    San Mateo, CA
    9 days ago
  • About InworldInworld is a research lab and inference provider focused on realtime AI for...  ...every day and we serve over 10 trillion LLM tokens per month. Our models and infrastructure...  ...-source contributions to major inference engines, or deep-dive technical write-ups.Full-... 
    Senior
    Full time
    Work at office
    Relocation package

    Inworld AI

    Mountain View, CA
    3 days ago
  • $237k - $296k

     ...Role Summary We are seeking a high-caliber Sr. Staff Design Verification Engineer to join our ADAS and Inference Silicon team. You will be responsible for the end...  ...Methodologies (nice-to-have): Strategize and Implement LLM-augmented workflows for Front End Development.... 
    Senior
    Full time
    Contract work
    Temporary work
    Part time
    Local area
    Shift work
    Night shift

    Rivian

    Palo Alto, CA
    4 days ago
  •  ...DescriptionJOHNSON & JOHNSON / CALIBRA Job DescriptionTitle: Sr. Quality Engineer for Swing Shift 2. Supervisor: QA Manager 3. Location: Redwood...  ...13485 and EN/ISO14971. SummaryType: ContractFunction: ManufacturingExperience level: Mid-Senior LevelIndustry: Pharmaceuticals
    Senior
    Work experience placement
    Work at office
    Afternoon shift

    LanceSoft

    Redwood City, CA
    13 hours ago
  •  ...at the right time.We are looking for a Senior Software Quality Engineer who will Support Tempus’ software development lifecycle (SDLC) activities...  ...with FDA Class I and II AI/ML-enabled software, including LLM-based agents ASQ CSQE or comparable quality/regulatory... 
    Senior
    Full time
    Remote work

    Tempus

    Redwood City, CA
    4 days ago
  •  ...solvingExceptional in execution and impactThe Sales Engineer is the Technical Lead for the Proofpoint...  ...closely with customer/prospect security staff for technical discovery, as well as...  ...Bay AreaThis role will focus on Mid-Market accounts in the Bay AreaWhat you bring... 
    Senior
    Full time
    Local area
    Night shift

    Proofpoint

    Sunnyvale, CA
    1 day ago
  •  ...About the Role Role Mission As HAI's Forward Deployed Engineer, you will be the technical owner of AI deployments that directly transform...  ..., LangSmith, and modern AI frameworks to implement advanced LLM techniques (RAG, prompt engineering, LLM-as-judge, chain-of-... 
    Senior
    Full time
    Work at office

    Hippocratic%20AI

    Menlo Park, CA
    9 days ago
  • $167.2k - $316.6k

     ...and pursue their dreams.Ford is seeking a Sr. Staff IVI Product Strategy Lead with a strong...  ...(IVI) product roadmap across near-, mid-, and long-term priorities; guide internal...  ...QualificationsYou'll have...• Bachelor's degree in Engineering, Computer Science, Electrical... 
    Senior
    Immediate start
    Visa sponsorship
    Flexible hours

    Ford

    Palo Alto, CA
    4 days ago
  • $167k - $301k

     ...of the threats detected in their environments. This is a Sr. Engineering Manager role in the CTIO team. In this role you will build...  ...data pipelines that feed them to delivering efficient on-device inference, and you will partner closely with data science, cloud and... 
    Senior
    Permanent employment
    Work experience placement
    Work at office
    Local area
    Remote work
    Sunnyvale, CA
    a month ago
  •  ...iPad) and Google (TPU, OpenTitan), and top researchers and engineers from Anthropic, Google DeepMind, Stanford, and MIT....  ...engineers to focus on a variety of challenges relating to scaling, low-level optimization, and core infrastructure for LLM training and inference.... 
    Full time

    Ricursive%20Intelligence

    Palo Alto, CA
    9 days ago
  • $160.36k - $291.15k

     ...strengthen the processes that support safe development and deployment.About the Role We are looking for a Senior or Staff Safety Management Systems Engineer to design, implement, and continuously improve Nuro’s company-wide Safety Management System (SMS) for autonomous... 
    Senior
    Work at office

    Nuro

    Mountain View, CA
    2 days ago
  • $272k - $336k

     ...Waymo Systems Engineering RoleWaymo is an autonomous driving technology company with the mission to be the world's most trusted driver. Since its start as the Google Self-Driving Car Project in 2009, Waymo has focused on building the Waymo Driver—The World's Most Experienced... 
    Senior
    Odd job
    Full time
    Remote work

    Latent Logic

    Mountain View, CA
    1 day ago
  • $195.2k - $262.2k

     ...large in-house AI/ML infrastructure. Built by engineers, for engineers. From large-scale GPU orchestration to inference optimization, we own the hard problems across...  ...regressions during production rollouts. Improve LLM and VLM endpoint latency, throughput, memory... 
    Senior
    Temporary work
    Immediate start
    Remote work

    Nebius

    Palo Alto, CA
    6 days ago
  • $195.2k - $262.2k

     ...large in-house AI/ML infrastructure. Built by engineers, for engineers. From large-scale GPU orchestration to inference optimization, we own the hard problems across...  ...is looking for scientists who can turn frontier LLM and VLM inference bottlenecks into research questions... 
    Senior
    Temporary work
    Immediate start
    Remote work

    Nebius

    Palo Alto, CA
    6 days ago
  •  ...industry-leading training and inference speeds; over 10 times faster...  ...inference.About the RoleAs a Staff GPU Inference SDET, you will...  ...Development team. Working closely with engineering leads and cross-functional...  ...and stress-test distributed LLM serving frameworks, focusing... 

    Cerebras Systems

    Sunnyvale, CA
    3 days ago
  • $215k - $290k

     ...Senior Forward Deployed Engineer About the Role Join a customer-facing engineering team delivering production-ready voice AI solutions,...  ...stakeholders aligned. Design and build voice AI solutions using effective LLM prompting techniques. Develop full-stack integrations in Python... 
    Senior
    Visa sponsorship

    True Talent Solutions Partners

    Redwood City, CA
    3 days ago
  • $215k - $290k

     ...About the job Senior Forward Deployed Engineer Senior Forward Deployed Engineer Company: Confidential - reimagining the call...  ...Become a go-to expert on building voice AI solutions, especially LLM prompting. Build full-stack integrations (Python, JavaScript)... 
    Senior
    Full time
    H1b
    Visa sponsorship

    Transparent Search Group

    Redwood City, CA
    2 days ago
  • $119k - $140k

     ..., and so are the rewards. The Corporate Engineering team is building and scaling the next generation...  ...solutions at scale. From enterprise LLM integrations to agentic workflows and...  ...AI workloads, tracking metrics such as inference latency, API costs, token usage, and error... 
    Senior
    Work at office
    Flexible hours
    Shift work
    3 days per week

    Robinhood Financial

    Menlo Park, CA
    3 days ago
  • $291k - $409k

     ...stack, CAS gives our robotaxi a second, diverse line of defense in the most challenging dense urban environments.As a Staff or Senior Staff engineer, you will be the technical lead for CAS algorithms, with a focus on CAS perception. You will shape how CAS sees the world... 
    Senior
    Full time
    Temporary work
    Relocation package

    Zoox

    Foster, CA
    2 days ago
  • $225k - $250k

     ...Staff Forward Deployed EngineerRedwood City, CASuki builds AI for clinical documentation...  ...EMRs.We are hiring a Staff Forward Deployed Engineer to lead technical delivery for our most...  ..., and React.Experience shipping LLM features to production, including prompt... 
    Senior

    Suki

    Redwood City, CA
    1 day ago
  • $100k - $200k

     ...Job Description Job Description OPPO US Research Center is seeking a full-time meticulous and innovative AI/LLM Test Engineer to join our cutting-edge AI team. In this critical role, you will evaluate the performance, reliability, and safety of Large Language Models... 
    Full time
    Contract work
    For contractors
    Work experience placement

    OPPO US Research Center

    Palo Alto, CA
    a month ago
  • $81.53k - $181.3k

    We are hiring a highly technical, business-savvy Senior GTM Engineer to help build the next generation of Navan’s GTM systems. This hands...  ....At least 2 years of hands-on experience building and deploying LLM-powered agents, copilots, or automations used by business teams... 
    Senior

    Navan

    Palo Alto, CA
    2 days ago
  • $170k - $230k

     ...is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars.SR. 3GPP STANDARDIZATION ENGINEER (STARLINK MOBILE)The 3GPP RAN Standardization Engineer will serve as the company’s technical expert within the 3GPP Radio... 
    Senior
    Permanent employment
    Temporary work

    SpaceX

    Palo Alto, CA
    4 days ago
  •  ...Sr. Electrical Design Engineer Build something that matters. At Delta Star the work we do has a real impact. Every transformer we build helps deliver reliable electricity to homes, businesses, and communities across North America. As we continue to grow, we are looking... 
    Senior
    Work at office
    Work from home
    Relocation
    Relocation package

    Delta Star

    Redwood City, CA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to LLM Inference Engineer (Mid, Sr, Staff). Be the first to apply!