Staff Software Engineer, Model Infrastructure [Remote]
$236k - $340kharvey
- Remote job
Why Harvey
At Harvey, we’re transforming how legal and professional services operate. By combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise, we’re reshaping how critical knowledge work gets done for decades to come.
This is a rare chance to help build a generational company at a true inflection point. We have strong product-market fit and world-class investor support. We’re scaling fast and defining a new category in real time. The work is ambitious, the bar is high, and the opportunity for growth — personal, professional, and financial — is unmatched.
Our team moves fast, takes ownership, and is deeply committed to the mission — operating with intensity, staying close to our customers, and pushing each other for excellence. We live by three values: Decisiveness, Simplicity, and Job's Not Finished. We act quickly on clear judgment over perfect information, we believe simplicity is what scales, and we're never satisfied with where we are. If you want to do the best work of your career alongside people who share that drive, we'd love to build with you.
At Harvey, the future of professional services is being written today — and we’re just getting started.
Role Overview
As a Staff Software Engineer on the Model Infrastructure team, you'll lead the design and development of the systems that power every AI request at Harvey. You'll partner closely with AI Research, Product Engineering, Infrastructure, and external model providers to build a platform that is highly reliable, scalable, observable, and efficient.
What You’ll Do
Lead the design and implementation of Harvey's Model Infrastructure platform.
Build systems to ensure high availability, low latency, and operational excellence for AI inference.
Design and improve Harvey's Unified Model Controller (UMC) and Model Selector platform to automatically detect model degradations and intelligently route traffic based on reliability, latency, quality, compliance, and cost.
Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers.
Integrate new model providers and maintain provider APIs and SDKs, enabling Harvey to rapidly adopt emerging frontier models.
Improve observability through health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry.
Partner with Product Engineering to support model launches, experimentation, and proactive monitoring of production AI workloads.
Drive infrastructure efficiency through capacity planning, utilization optimization, and cost visibility.
Collaborate with AI Research to build the infrastructure foundation for future model evaluation, training, and deployment.
Lead cross-functional technical initiatives and mentor engineers across the organization.
What You’ll Build
Model Reliability & Operations
Model health monitoring
Automated failover and recovery
Capacity provisioning
Operational tooling and incident automation
Unified Model Controller (UMC)
Policy-based model routing
Intelligent Model Selector
Model health monitoring
Traffic management
Reliability and latency optimization
Provider Platform
Multi-provider architecture
API and SDK integrations
OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, and future providers
Rapid adoption of new frontier models
Observability & Cost Platform
Token usage analytics
Cost attribution
Latency and reliability dashboards
Capacity forecasting
Utilization optimization
AI Platform Foundation
Infrastructure supporting model evaluation
Model deployment and operations
Future model training platform
Agent infrastructure and CcaaS
What You Have
10+ years of software engineering experience building large-scale distributed systems.
Experience designing and operating highly available production services.
Strong programming skills in Go, Java, Python, Rust, or C++.
Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
Experience leading technical projects across multiple engineering teams.
Ability to balance long-term architecture with pragmatic execution.
Strong communication and collaboration skills.
Passion for building foundational platforms that enable other engineering teams.
Nice to Have
Experience with AI infrastructure, LLM serving, or machine learning platforms.
Experience with model routing, inference gateways, or policy-based serving systems.
Experience working with OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, or open-source LLMs.
Experience with Kubernetes, cloud infrastructure, and service mesh technologies.
Experience with large-scale observability and SRE best practices.
Experience with data infrastructure technologies such as Kafka, Spark, Flink, Airflow, or Iceberg.
Familiarity with GPU infrastructure or model training platforms.
Compensation
$236,000 - $340,000 USD
Depending on your location, an Applicant Privacy Notice may apply to you. You can find all of our Applicant Privacy Notices here .
#LI-AK1
Harvey is an equal opportunity employer and does not discriminate on the basis of race, gender, sexual orientation, gender identity/expression, national origin, disability, age, genetic information, veteran status, marital status, pregnancy or related condition, or any other basis protected by law.
We are committed to providing reasonable accommodations to applicants with disabilities, and requests can be made by emailing View email address on jobs.ashbyhq.com
$250k - $300k
...intelligence . As the only vertically integrated AI infrastructure company built from the ground up, we own and operate... ...with us at Crusoe. About This Role: The Senior Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building a...SuggestedFull timeTemporary work$190k - $265k
...and running the world's best data and AI infrastructure platform so our customers can use deep... ...to improve their business. Founded by engineers — and customer-obsessed — we leap at every... ...everything from data apps, AI agents, model training, model serving, and Vector...SuggestedLocal areaWorldwide- ...About Flow Flow Engineering is an AI-native requirements platform for modern... ...About the role Flow is hiring a staff software engineer with an infrastructure focus to build and scale the core... ...AI engineers to define APIs, data models, and integration points that are robust...SuggestedFlexible hours
$175k - $230k
...robots-we're building the essential infrastructure for next generation biologics. Our technology... ..., cloud-integrated hardware-software systems, dextrous manipulation, and imitation... ...to be traceable. We're hiring a Staff Software Engineer to be the first dedicated owner of...SuggestedH1bWork at officeRemote workVisa sponsorshipFlexible hoursShift workAfternoon shift- ...ambition in the physical world. Critical infrastructure is constrained by labor shortages, hazardous... ...the Role At Watney, ML Infrastructure engineers turn data collected from a live fleet of robots into better models. The fleet produces large volumes of video and...SuggestedFull time
$160k - $280k
...About the Role We're looking for a software engineer to build the Kubernetes-native control... ...they need, whether that's a cluster, a model deployment, or a capacity change, and... ...packing, right-sizing), and think about GPU infrastructure as software to be engineered. A...Full time$300 per month
.... As the only vertically integrated AI infrastructure company built from the ground up, we own... ...Crusoe’s Data Center Infrastructure Engineering (DCIE) team is fundamental to our mission... ...seeking a highly skilled and motivated Software Engineer to join Crusoe’s Data Center Infrastructure...Full timeTemporary work- ...Staff Software Engineer Opportunity Lunar is a stealth technology company building a new type of software platform for health systems.... ...generation platform is a fundamental rebuild of the information infrastructure that runs health systems. All of it – from doctors,...
- ...California or New York, New York. About the role As a Staff Software Engineer working on Platform, you will play a foundational role in... ...local dev environments, build systems, CI/CD, test infrastructure, and time-to-first-commit for new hires Build and maintain...Work at officeLocal areaWork from homeWorldwideMonday to Friday
$250k - $295k
...Staff Software Engineer Unify is building the first AI-native outbound platform where agents and sellers work side by side. We help revenue... ...inherit. Rapidly prototype with third-party APIs, databases, model-serving platforms, and infra vendors to accelerate customer...$260k - $350k
...re just getting started. Role Overview As the Engineering Manager for Model Infrastructure, you'll lead the team responsible for the platform powering... ...You'll Do Lead and grow a high-performing team of software engineers responsible for Harvey's Model...Full time$220k - $265k
...Nscale is the GPU cloud engineered for AI. We provide cost-effective, high-performance infrastructure for AI start-ups and large... ...the Role We're hiring a Staff Cloud Native Software Engineer to build, operate... ...reconciliation loops, and the object model. Strong networking...Flexible hours$163k - $204k
...process. About the Role: We're looking for seasoned full-stack software engineers to join the teams behind Gusto's AI product platforms — the... ...upon — to serve small business owners every day. As a Staff Software Engineer, you'll operate across the full lifecycle of...Full timeWork at officeLocal areaRemote work2 days per week3 days per week- ...distributed computing and make it accessible to software developers of all skill levels. We’re... ...About the Role Anyscale is seeking a Staff Software Engineer to lead the technical vision for our Infrastructure team. As a Staff Engineer, you will be responsible...Full timeFlexible hours
- ...Stripe is a financial infrastructure platform for businesses... .... Pair with the engineer manager to mentor and... ...10+ years of industry software engineering experience... ...demonstrated impact at a staff or equivalent scope (leading... ..., BIN sponsorship models, or issuing bank...
$192k - $264k
...at the center of everything we do. Staff Software Engineer - Analytics Compute Platform Team... ...Help unify and evolve our core data model, including how non-event data (profile... ...OLAP databases, query engines, or data infrastructure Experience with SQL, query planning...Full timeWork at officeHome officeFlexible hours- ...Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million... ...from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack...Full timeWork at officeFlexible hours
$199k - $270k
Staff Software Engineer - AI Research InfrastructureP-1215At Databricks, we are obsessed with enabling... ...enables companies to develop AI models and agents using their own data, with... ...Staff Software Engineer, AI Research Infrastructure, you will be developing and running the...Local areaWorldwide$208.45k - $364.8k
...process here.Pinterest is seeking a Staff Software Engineer to help evolve the serving platforms... ...efficiency goals.Partner with product, ML, infrastructure, and Core/Ads engineering teams to... ...page to learn more about our working model.In-Office Requirement Statement:We...Work at officeLocal areaRelocationRelocation package$245.4k - $429.45k
...AI in our recruiting process here.Sr. Staff Software Engineer, Product ML InfrastructureMillions of... ...with AI.Pinterest’s Product ML Infrastructure (PMLI) team enables fast, safe, and efficient... ...will set technical direction across model training and serving, with a focus on...Work at officeLocal areaRemote workRelocationRelocation package$232k
...development, and evolution of the build systems and CI infrastructure that every Android engineer at Uber depends on.Every change to Uber's Android apps... ...run only the tests a change can actually affect.As the Staff Android Engineer on the team, you will be the technical...Full timeWork at officeLocal areaRemote work$198k - $299k
...data continuously. Our in-house OLAP engine, Nova, processes trillions of events in... ...worldwide.That makes Nova the critical infrastructure in the loop, and as non-stop agents... ...grows dramatically.We're looking for a Staff Software Engineer who wants to go deep on the engine...Immediate startWorldwideHome office- ...About the Role The Cloud Infrastructure team owns the foundational cloud primitives and deployment models that power Perplexity's products, from multi-tenant public... ...infrastructure as code (Terraform) and strong software engineering skills in at least one of Python, Go, or...Full time
$300 per month
...an ambitious team run by scientists and engineers from leading institutions across... ...chemistry, and AI. About the Role As a Software Engineer on our Cloud team, you'll play... ...systems running within their own cloud infrastructure. These datasets are uniquely valuable assets...Full timeWork at office$215k - $260k
...only vertically integrated AI infrastructure company built from the... ...Join Crusoe Energy as a Staff Data Engineer, an early and pivotal hire... ...Engineering Teams: Partner with software engineers, data scientists,... ...advanced SQL skills for data modeling, query optimization, and...Full timeTemporary workWork at officeRemote work$231k - $340k
...Role Overview As a Backend Platform Engineer at Harvey, you will help build and... ...and easy to monitor Collaborate with Model Infrastructure team to tackle challenges unique to... ...confidence What You Have ~8+ years of software engineering experience (post-BS/MS),...Full timeShift work- ...ecosystem. This is a product and platform engineering role with end-to-end ownership, from designing connector infrastructure and semantic models to shipping the experiences customers... ...helps customers put their existing software and knowledge to work through...Full time
$405k
...committed researchers, engineers, policy experts, and... ...Platform Safety builds the infrastructure that lets Anthropic bring its most capable models to more people, safely... ...development. As a Staff+ engineer, you'll help... ..., build and review software Have designed and operated...Work at officeLocal areaVisa sponsorshipFlexible hours$194k - $267k
...building the trusted, neutral infrastructure that enables organizations... ...users’ experiences. Okta's Core Engineering team is responsible for... ...managers. We're looking for a staff level backend engineer to... ...authorizationDemonstrably followed best software engineering...Permanent employmentLocal areaWorldwideFlexible hours$232k
...networking APIs, developer experience, and observability.As a Staff Software Engineer on the Android Mobile Networking team, you will play a... ...direction and evolution of Uber's Android networking infrastructure. You will tackle complex engineering challenges at scale,...Full timeWork at officeImmediate startRemote workWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Staff Software Engineer, Model Infrastructure [Remote]. Be the first to apply!
- entry level infrastructure engineer San Francisco, CA
- principal infrastructure engineer San Francisco, CA
- infrastructure engineering manager San Francisco, CA
- infrastructure developer San Francisco, CA
- security infrastructure engineer San Francisco, CA
- infrastructure engineer San Francisco, CA
- lead infrastructure engineer San Francisco, CA
- remote infrastructure engineer San Francisco, CA
- data infrastructure engineer San Francisco, CA
- senior infrastructure engineer San Francisco, CA



