Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Staff Software Engineer, Model Infrastructure [Remote]

$236k - $340k
Full-time

harvey

Why Harvey

At Harvey, we’re transforming how legal and professional services operate. By combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise, we’re reshaping how critical knowledge work gets done for decades to come.

This is a rare chance to help build a generational company at a true inflection point. We have strong product-market fit and world-class investor support. We’re scaling fast and defining a new category in real time. The work is ambitious, the bar is high, and the opportunity for growth — personal, professional, and financial — is unmatched.

Our team moves fast, takes ownership, and is deeply committed to the mission — operating with intensity, staying close to our customers, and pushing each other for excellence. We live by three values: Decisiveness, Simplicity, and Job's Not Finished. We act quickly on clear judgment over perfect information, we believe simplicity is what scales, and we're never satisfied with where we are. If you want to do the best work of your career alongside people who share that drive, we'd love to build with you.

At Harvey, the future of professional services is being written today — and we’re just getting started.

Role Overview

As a Staff Software Engineer on the Model Infrastructure team, you'll lead the design and development of the systems that power every AI request at Harvey. You'll partner closely with AI Research, Product Engineering, Infrastructure, and external model providers to build a platform that is highly reliable, scalable, observable, and efficient.


What You’ll Do

  • Lead the design and implementation of Harvey's Model Infrastructure platform.

  • Build systems to ensure high availability, low latency, and operational excellence for AI inference.

  • Design and improve Harvey's Unified Model Controller (UMC) and Model Selector platform to automatically detect model degradations and intelligently route traffic based on reliability, latency, quality, compliance, and cost.

  • Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers.

  • Integrate new model providers and maintain provider APIs and SDKs, enabling Harvey to rapidly adopt emerging frontier models.

  • Improve observability through health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry.

  • Partner with Product Engineering to support model launches, experimentation, and proactive monitoring of production AI workloads.

  • Drive infrastructure efficiency through capacity planning, utilization optimization, and cost visibility.

  • Collaborate with AI Research to build the infrastructure foundation for future model evaluation, training, and deployment.

  • Lead cross-functional technical initiatives and mentor engineers across the organization.

What You’ll Build

  • Model Reliability & Operations

    • Model health monitoring

    • Automated failover and recovery

    • Capacity provisioning

    • Operational tooling and incident automation

  • Unified Model Controller (UMC)

    • Policy-based model routing

    • Intelligent Model Selector

    • Model health monitoring

    • Traffic management

    • Reliability and latency optimization

  • Provider Platform

    • Multi-provider architecture

    • API and SDK integrations

    • OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, and future providers

    • Rapid adoption of new frontier models

  • Observability & Cost Platform

    • Token usage analytics

    • Cost attribution

    • Latency and reliability dashboards

    • Capacity forecasting

    • Utilization optimization

  • AI Platform Foundation

    • Infrastructure supporting model evaluation

    • Model deployment and operations

    • Future model training platform

    • Agent infrastructure and CcaaS


What You Have

  • 10+ years of software engineering experience building large-scale distributed systems.

  • Experience designing and operating highly available production services.

  • Strong programming skills in Go, Java, Python, Rust, or C++.

  • Deep understanding of distributed systems, cloud infrastructure, networking, and observability.

  • Experience leading technical projects across multiple engineering teams.

  • Ability to balance long-term architecture with pragmatic execution.

  • Strong communication and collaboration skills.

  • Passion for building foundational platforms that enable other engineering teams.

Nice to Have

  • Experience with AI infrastructure, LLM serving, or machine learning platforms.

  • Experience with model routing, inference gateways, or policy-based serving systems.

  • Experience working with OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, or open-source LLMs.

  • Experience with Kubernetes, cloud infrastructure, and service mesh technologies.

  • Experience with large-scale observability and SRE best practices.

  • Experience with data infrastructure technologies such as Kafka, Spark, Flink, Airflow, or Iceberg.

  • Familiarity with GPU infrastructure or model training platforms.

Compensation

$236,000 - $340,000 USD

Depending on your location, an Applicant Privacy Notice may apply to you. You can find all of our Applicant Privacy Notices here .

#LI-AK1

Harvey is an equal opportunity employer and does not discriminate on the basis of race, gender, sexual orientation, gender identity/expression, national origin, disability, age, genetic information, veteran status, marital status, pregnancy or related condition, or any other basis protected by law.

We are committed to providing reasonable accommodations to applicants with disabilities, and requests can be made by emailing View email address on jobs.ashbyhq.com

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Staff Software Engineer, Model Infrastructure [Remote] in San Francisco, CA vacancy
  • $250k - $300k

     ...intelligence . As the only vertically integrated AI infrastructure company built from the ground up, we own and operate...  ...with us at Crusoe. About This Role: The Senior Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a... 
    Suggested
    Full time
    Temporary work

    Crusoe

    San Francisco, CA
    4 days ago
  • $190k - $265k

     ...and running the world's best data and AI infrastructure platform so our customers can use deep...  ...to improve their business. Founded by engineers — and customer-obsessed — we leap at every...  ...everything from data apps, AI agents, model training, model serving, and Vector... 
    Suggested
    Local area
    Worldwide

    DataBricks

    San Francisco, CA
    14 hours ago
  •  ...About Flow Flow Engineering is an AI-native requirements platform for modern...  ...About the role Flow is hiring a staff software engineer with an infrastructure focus to build and scale the core...  ...AI engineers to define APIs, data models, and integration points that are robust... 
    Suggested
    Flexible hours

    Flow Engineering

    San Francisco, CA
    2 hours ago
  • $175k - $230k

     ...robots-we're building the essential infrastructure for next generation biologics. Our technology...  ..., cloud-integrated hardware-software systems, dextrous manipulation, and imitation...  ...to be traceable. We're hiring a Staff Software Engineer to be the first dedicated owner of... 
    Suggested
    H1b
    Work at office
    Remote work
    Visa sponsorship
    Flexible hours
    Shift work
    Afternoon shift

    Multiply Labs

    San Francisco, CA
    3 days ago
  •  ...ambition in the physical world. Critical infrastructure is constrained by labor shortages, hazardous...  ...the Role At Watney, ML Infrastructure engineers turn data collected from a live fleet of robots into better models. The fleet produces large volumes of video and... 
    Suggested
    Full time

    watney

    San Francisco, CA
    4 days ago
  • $160k - $280k

     ...About the Role We're looking for a software engineer to build the Kubernetes-native control...  ...they need, whether that's a cluster, a model deployment, or a capacity change, and...  ...packing, right-sizing), and think about GPU infrastructure as software to be engineered. A... 
    Full time

    Together AI

    San Francisco, CA
    8 days ago
  • $300 per month

     .... As the only vertically integrated AI infrastructure company built from the ground up, we own...  ...Crusoe’s Data Center Infrastructure Engineering (DCIE) team is fundamental to our mission...  ...seeking a highly skilled and motivated Software Engineer to join Crusoe’s Data Center Infrastructure... 
    Full time
    Temporary work

    Crusoe

    San Francisco, CA
    4 days ago
  •  ...Staff Software Engineer Opportunity Lunar is a stealth technology company building a new type of software platform for health systems....  ...generation platform is a fundamental rebuild of the information infrastructure that runs health systems. All of it – from doctors,... 

    Lunar GMBH

    San Francisco, CA
    2 days ago
  •  ...California or New York, New York. About the role As a Staff Software Engineer working on Platform, you will play a foundational role in...  ...local dev environments, build systems, CI/CD, test infrastructure, and time-to-first-commit for new hires Build and maintain... 
    Work at office
    Local area
    Work from home
    Worldwide
    Monday to Friday

    Luminai, Inc

    San Francisco, CA
    3 days ago
  • $250k - $295k

     ...Staff Software Engineer Unify is building the first AI-native outbound platform where agents and sellers work side by side. We help revenue...  ...inherit. Rapidly prototype with third-party APIs, databases, model-serving platforms, and infra vendors to accelerate customer... 

    Unify

    San Francisco, CA
    4 days ago
  • $260k - $350k

     ...re just getting started. Role Overview As the Engineering Manager for Model Infrastructure, you'll lead the team responsible for the platform powering...  ...You'll Do Lead and grow a high-performing team of software engineers responsible for Harvey's Model... 
    Full time

    Harvey

    San Francisco, CA
    8 days ago
  • $220k - $265k

     ...Nscale is the GPU cloud engineered for AI. We provide cost-effective, high-performance infrastructure for AI start-ups and large...  ...the Role We're hiring a Staff Cloud Native Software Engineer to build, operate...  ...reconciliation loops, and the object model. Strong networking... 
    Flexible hours

    Nscale

    San Francisco, CA
    8 days ago
  • $163k - $204k

     ...process. About the Role: We're looking for seasoned full-stack software engineers to join the teams behind Gusto's AI product platforms — the...  ...upon — to serve small business owners every day. As a Staff Software Engineer, you'll operate across the full lifecycle of... 
    Full time
    Work at office
    Local area
    Remote work
    2 days per week
    3 days per week

    Gusto

    San Francisco, CA
    3 days ago
  •  ...distributed computing and make it accessible to software developers of all skill levels. We’re...  ...About the Role Anyscale is seeking a Staff Software Engineer to lead the technical vision for our Infrastructure team. As a Staff Engineer, you will be responsible... 
    Full time
    Flexible hours

    Anyscale

    San Francisco, CA
    8 days ago
  •  ...Stripe is a financial infrastructure platform for businesses...  .... Pair with the engineer manager to mentor and...  ...10+ years of industry software engineering experience...  ...demonstrated impact at a staff or equivalent scope (leading...  ..., BIN sponsorship models, or issuing bank... 

    Stripe

    San Francisco, CA
    8 days ago
  • $192k - $264k

     ...at the center of everything we do. Staff Software Engineer - Analytics Compute Platform Team...  ...Help unify and evolve our core data model, including how non-event data (profile...  ...OLAP databases, query engines, or data infrastructure Experience with SQL, query planning... 
    Full time
    Work at office
    Home office
    Flexible hours

    amplitude

    San Francisco, CA
    4 days ago
  •  ...Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million...  ...from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack... 
    Full time
    Work at office
    Flexible hours

    Sciforium

    San Francisco, CA
    4 days ago
  • $199k - $270k

    Staff Software Engineer - AI Research InfrastructureP-1215At Databricks, we are obsessed with enabling...  ...enables companies to develop AI models and agents using their own data, with...  ...Staff Software Engineer, AI Research Infrastructure, you will be developing and running the... 
    Local area
    Worldwide

    DataBricks

    San Francisco, CA
    1 day ago
  • $208.45k - $364.8k

     ...process here.Pinterest is seeking a Staff Software Engineer to help evolve the serving platforms...  ...efficiency goals.Partner with product, ML, infrastructure, and Core/Ads engineering teams to...  ...page to learn more about our working model.In-Office Requirement Statement:We... 
    Work at office
    Local area
    Relocation
    Relocation package

    Pinterest

    San Francisco, CA
    3 days ago
  • $245.4k - $429.45k

     ...AI in our recruiting process here.Sr. Staff Software Engineer, Product ML InfrastructureMillions of...  ...with AI.Pinterest’s Product ML Infrastructure (PMLI) team enables fast, safe, and efficient...  ...will set technical direction across model training and serving, with a focus on... 
    Work at office
    Local area
    Remote work
    Relocation
    Relocation package

    Pinterest

    San Francisco, CA
    2 days ago
  • $232k

     ...development, and evolution of the build systems and CI infrastructure that every Android engineer at Uber depends on.Every change to Uber's Android apps...  ...run only the tests a change can actually affect.As the Staff Android Engineer on the team, you will be the technical... 
    Full time
    Work at office
    Local area
    Remote work

    Uber

    San Francisco, CA
    4 days ago
  • $198k - $299k

     ...data continuously. Our in-house OLAP engine, Nova, processes trillions of events in...  ...worldwide.That makes Nova the critical infrastructure in the loop, and as non-stop agents...  ...grows dramatically.We're looking for a Staff Software Engineer who wants to go deep on the engine... 
    Immediate start
    Worldwide
    Home office

    Amplitude

    San Francisco, CA
    2 days ago
  •  ...About the Role The Cloud Infrastructure team owns the foundational cloud primitives and deployment models that power Perplexity's products, from multi-tenant public...  ...infrastructure as code (Terraform) and strong software engineering skills in at least one of Python, Go, or... 
    Full time

    Perplexity

    San Francisco, CA
    8 days ago
  • $300 per month

     ...an ambitious team run by scientists and engineers from leading institutions across...  ...chemistry, and AI. About the Role As a Software Engineer on our Cloud team, you'll play...  ...systems running within their own cloud infrastructure. These datasets are uniquely valuable assets... 
    Full time
    Work at office

    edison%20scientific

    San Francisco, CA
    4 days ago
  • $215k - $260k

     ...only vertically integrated AI infrastructure company built from the...  ...Join Crusoe Energy as a Staff Data Engineer, an early and pivotal hire...  ...Engineering Teams: Partner with software engineers, data scientists,...  ...advanced SQL skills for data modeling, query optimization, and... 
    Full time
    Temporary work
    Work at office
    Remote work

    Crusoe

    San Francisco, CA
    4 days ago
  • $231k - $340k

     ...Role Overview As a Backend Platform Engineer at Harvey, you will help build and...  ...and easy to monitor Collaborate with Model Infrastructure team to tackle challenges unique to...  ...confidence What You Have ~8+ years of software engineering experience (post-BS/MS),... 
    Full time
    Shift work

    Harvey

    San Francisco, CA
    8 days ago
  •  ...ecosystem. This is a product and platform engineering role with end-to-end ownership, from designing connector infrastructure and semantic models to shipping the experiences customers...  ...helps customers put their existing software and knowledge to work through... 
    Full time

    Perplexity

    San Francisco, CA
    8 days ago
  • $405k

     ...committed researchers, engineers, policy experts, and...  ...Platform Safety builds the infrastructure that lets Anthropic bring its most capable models to more people, safely...  ...development. As a Staff+ engineer, you'll help...  ..., build and review software Have designed and operated... 
    Work at office
    Local area
    Visa sponsorship
    Flexible hours

    Anthropic

    San Francisco, CA
    8 days ago
  • $194k - $267k

     ...building the trusted, neutral infrastructure that enables organizations...  ...users’ experiences. Okta's Core Engineering team is responsible for...  ...managers. We're looking for a staff level backend engineer to...  ...authorizationDemonstrably followed best software engineering... 
    Permanent employment
    Local area
    Worldwide
    Flexible hours

    Okta

    San Francisco, CA
    1 day ago
  • $232k

     ...networking APIs, developer experience, and observability.As a Staff Software Engineer on the Android Mobile Networking team, you will play a...  ...direction and evolution of Uber's Android networking infrastructure. You will tackle complex engineering challenges at scale,... 
    Full time
    Work at office
    Immediate start
    Remote work
    Worldwide

    Uber

    San Francisco, CA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Staff Software Engineer, Model Infrastructure [Remote]. Be the first to apply!