Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

ML Infrastructure Engineer - Distributed Training & Systems

Full-time

MaxIT Consulting - Max Corporate Group

ML Infrastructure Engineer Distributed Training & Systems

San Francisco CA Hybrid Full-Time Permanent Employment Mid-Level ASAP Start

MaxIT Consulting Max Corporate Group is looking for a Research Engineer to build the training and environment infrastructure of a fast-growing AI company in San Francisco. The company builds evaluation environments and post-training infrastructure for long-horizon enterprise agents and uses that stack to train and deploy custom models for very large technology companies.

It is a small founding-stage team out of top AI labs and technology companies already profitable and with real enterprise revenue. This role sits under the whole company: the research and deployed teams build directly on what you own.

THE CHALLENGE

As agent workflows get longer evaluation and training infrastructure become the bottleneck. You will own the systems that let long runs be paused rewound and continued and that start watch and shut down thousands of heterogeneous environments cheaply and reliably.

Within your first 90 days you will own one of the core systems end to end: checkpoint and restore the environment orchestrator or the simulated companies used for training and evaluation.

KEY RESPONSIBILITIES

  • Build checkpoint and restore of full machine state including memory and GPU state across many machines.
  • Build the orchestrator that starts monitors and shuts down thousands of environments from a single container to a multi-GPU cluster .
  • Provision real cloud accounts for agents with hundreds created and deleted on demand.
  • Build realistic simulated companies with code people tickets threads and documentation for agent training and evaluation.
  • Be the infrastructure foundation that the research and deployed teams depend on.

WHAT WE ARE LOOKING FOR

  • One to five years of full-time experience (at least one year full time).
  • Deep experience in at least one of: distributed systems containers and operating-system internals ML infrastructure or model training .
  • Has trained or served models at real scale ( distributed training inference or serving ) or built the environment or benchmark behind a published result.
  • A publication at a top venue (such as ICLR or NeurIPS ).
  • Has built and owned infrastructure that other people depended on beyond a single research script.
  • Comfortable choosing the right tools: no specific language or framework is required.
  • Professional English as the role works daily with technical teams and enterprise customers.

NICE TO HAVE

  • Experience with checkpointing cluster orchestration or scheduling of many heterogeneous jobs.
  • Compiler inference or serving work on large models.
  • Simulation or benchmark environments (robotics sim agent gyms browser or operating-system environments multi-agent benchmarks).
  • Masters degree from a strong computer science program.
  • Interest in joining an early-stage company for the long-term upside.

POSITION DETAILS

Work arrangement: Hybrid three to four days a week in the San Francisco office
Schedule: Full-Time six-day work week
Employment type: Permanent employment
Seniority: Mid-Level (1 to 5 years)
Start date: ASAP
Location: San Francisco California United States

Competitive base salary founding-level equity a performance bonus tied to customer deployments and health dental and vision coverage. Flexible arrangements for candidates outside the Bay Area are reviewed case by case.

WORK AUTHORIZATION

Candidates must be legally authorized to work in the United States. Visa sponsorship for new petitions is not available; H-1B transfers and OPT with sufficient time remaining are considered.

HOW TO APPLY

If you want to own the infrastructure that frontier-level agent training runs on we would like to hear from you. Apply through this platform including your expected gross annual salary in USD and your availability to start.

Vacancy posted 8 days ago
Similar jobs that could be interesting for youBased on the ML Infrastructure Engineer - Distributed Training & Systems in San Francisco, CA vacancy
  • $117.2k - $313.7k

     ...you are the future of Salesforce.Distributed Systems Software Engineer - Public Cloud (Senior/Lead/Principal...  ...-ready.Your Impact:Deliver cloud infrastructure automation tools, frameworks,...  ...compensation, promotion, benefits, training, assessment of job performance, discipline... 
    Training
    Full time

    Salesforce

    San Francisco, CA
    23 hours ago
  • $200k - $280k

     ...Engineering San Francisco Full-time $200,000 - $280,000 About the Role Join our ML Infrastructure team to build the systems that train, deploy, and serve our AI models at scale. You'll work at...  ...with ML training frameworks and distributed training ~ Familiarity with... 
    Training
    Full time
    Work at office

    Lattice

    San Francisco, CA
    1 day ago
  •  ...Job Title: Machine Learning Infrastructure Engineer Location: San Francisco, CA Metro Area...  ...Engineer to help architect the compute, training, and execution frameworks powering...  ...hold direct ownership over scaling distributed systems across massive hardware clusters while... 
    Training
    Full time
    Work at office
    Flexible hours

    Objective Partners

    San Francisco, CA
    2 days ago
  • $160k - $250k

     ...world's largest autonomous logistics system, delivering critical supplies...  ...speeds.About You and The RoleAs an ML Training & Inference Infrastructure Engineer on the Data Platform team you...  ...processing pipelines, dataset management, distributed training, continuous model... 
    Training
    Local area

    ZipLine

    San Francisco, CA
    3 days ago
  • $148.5k - $223.9k

     ...frameworks in developing distributed filesystems in an...  ...mentoring junior engineers, and providing...  ...public cloud infrastructure - AWS/Azure/GCP.-...  ...distributed systems at scale. - Proficiency...  ...with Big-Data/ML and S3 Hands-on experience...  ..., benefits, training, assessment of... 
    Training
    Full time
    Immediate start

    Salesforce

    San Francisco, CA
    3 days ago
  •  ...virtually, a part of being a distributed-first company. Responsibilities As a Senior ML System Engineer in the Rovo GenAI Platform...  ...and maintain the core infrastructure to allow machine learning engineers...  ...scientists to develop, train, evaluate, deploy, and operate... 
    Training
    Work at office

    Atlassian

    San Francisco, CA
    4 days ago
  •  ...Gritt is an intelligent system that combines...  ...and AI to build the infrastructure that pulls society forward...  ...Role: Software - ML Infrastructure Engineer Location: SF Bay...  ...deploy scalable AI training and validation pipelines...  .... Spin up distributed pipelines for data ingestion... 
    Training

    Gritt

    South San Francisco, CA
    5 hours ago
  •  ...We are a team of engineers, scientists, roboticists...  ...The Team The Infrastructure team builds and...  ...PI does: from training state-of-the-art...  ...platform for internal systems such as...  ...platforms (GCP, AWS) and distributed systems: compute...  ...with large-scale ML training,... 
    Training
    Full time
    Local area
    Remote work

    Physical Intelligence

    San Francisco, CA
    3 days ago
  •  ...We are a group of engineers, scientists,...  ...at Pi builds the systems that scale research...  ...intersection of our AI, infrastructure, and external...  ..., caching, distributed systems, and infrastructure...  ...need to be an ML researcher....  ...responsible for training policies for individual... 
    Training
    Full time
    Remote work

    Physical Intelligence

    San Francisco, CA
    3 days ago
  • $189.6k - $237k

    Scale’s ML platform (RLXF) team builds our internal distributed framework for large language model training and inference. The platform has been powering...  ...:Strong excitement about system optimizationExperience...  ...ML systemsStrong software engineering skills, proficient in... 
    Training
    Full time

    Scale AI

    San Francisco, CA
    23 hours ago
  • $166k - $343k

    Presales Systems Engineer - HPE Networking (Northern California)This role...  ...experience with networking infrastructure technologies in all arenas....  ...on datacenter, campus and distributed branch networks. The...  ...work experience, education/training, and/or skill level. - United... 
    Training
    Full time
    Work experience placement
    Local area
    Immediate start
    Remote work
    Work from home

    Hewlett Packard Enterprise

    San Francisco, CA
    1 day ago
  •  ...As a Machine Learning Systems Administrator / DevOps Engineer, you will maintain and build the core infrastructure behind both our model training and our inference...  ...Prior work supporting ML/AI infrastructure, including...  ..., RoCE) and distributed storage Experience... 
    Training
    Work at office
    Remote work
    Relocation package

    Zyphra

    San Francisco, CA
    28 days ago
  • $215k - $260k

     ...vertically integrated AI infrastructure company built from the...  ...looking for a Staff Engineer to be the technical lead and systems architect for Crusoe's...  ...MDT moves petabyte-scale training data between hyperscaler...  ...You'll Bring to the Team:Distributed Systems Depth: Hands-on... 
    Training
    Temporary work

    Crusoe

    San Francisco, CA
    3 days ago
  • $170k - $225k

     ...company with employees distributed across the US and...  ...Machine Learning Engineer to take technical...  ...our core ranking system. Every job...  ...most consequential ML systems we run....  ...produce cleaner training data and unlock downstream...  ...stores, training infrastructure, serving,... 
    Training
    Work at office
    Immediate start
    Flexible hours

    Taskrabbit

    San Francisco, CA
    4 days ago
  •  ...Protocol Learning: training and serving large models...  ...on top of that: infrastructure orchestration, distributed compute, and the...  ...resource management systems that provision and...  ...infrastructure for distributed ML. GPU clusters,...  ...and platform engineering (required) : Production... 
    Training
    Full time
    Remote work
    Visa sponsorship
    Relocation package
    Flexible hours

    Pluralis Research

    San Francisco, CA
    6 days ago
  •  ...enterprise grade search infrastructure. Powering search and...  ...to determine where new ML capabilities can...  ...quality agentic search systems, addressing tail latency...  ...systems, mentor senior engineers, and align technical and...  ...and operations.Strong distributed-systems expertise... 
    Work at office
    Local area

    Atlassian

    San Francisco, CA
    3 days ago
  • $215k - $260k

     ...and intelligence. As the only vertically integrated AI infrastructure company built from the ground up, we own and operate...  ...Crusoe.About the Role:We are seeking a Staff Software Engineer to lead distributed systems design and development within Crusoe Cloud's Cloud Monitoring... 
    Full time
    Temporary work

    Crusoe

    San Francisco, CA
    1 day ago
  • $166k - $225k

     ...world's best data and AI infrastructure platform so our...  ...their business. Founded by engineers — and customer obsessed...  ...building the next generation distributed data storage and processing systems that can outperform...  ...relevant certifications and training, and specific work... 
    Training
    Local area
    Worldwide

    DataBricks

    San Francisco, CA
    1 day ago
  • Title: ML Engineer Location: San Francisco, CA (Onsite)...  ...class of neurosymbolic AI systems, combining statistical...  ...ML pipelines for training, validation, and deployment within distributed computing environments...  ...generation predictive infrastructure for healthcare.Opportunities... 
    Training

    Spectraforce Technologies

    San Francisco, CA
    23 hours ago
  • $191.48k - $260.83k

     ...career. Senior MLOps Engineer, Forward...  ...work across model training and refresh workflows...  ...reliable backend systems, validate the outcome...  ...quality, score distributions, threshold...  ...experience in MLOps, ML platform engineering...  ..., and production infrastructure.Experience... 
    Training
    Full time
    Shift work
    Night shift

    6sense

    San Francisco, CA
    3 days ago
  • $175k - $250k

     ...cutting-edge AI/ML products that solve...  ...AI Research & Engineering team. Operating at...  ...production-grade systems, you will drive the...  ...with core infrastructure and platform teams...  ...techniques in post-training fine-tuning, preference...  ...expertise with distributed machine learning... 
    Training
    Flexible hours

    Ellipsis Health

    San Francisco, CA
    14 hours ago
  •  ...Description Distributed Systems Software Engineer - Public Cloud (Senior/Lead/Principal) Note: By applying to the Public Cloud - Distributed...  ...core back-end systems. Your Impact: Deliver cloud infrastructure automation tools, frameworks, workflows, and... 
    Full time

    Salesforce

    San Francisco, CA
    more than 2 months ago
  •  ...world. We are a group of engineers, scientists,...  ...scale and optimize our training systems and core model code. You’ll own critical infrastructure for large-scale training...  ...the intersection of ML, software engineering...  ...metrics/logging. Scale distributed training: Work with... 
    Training
    Full time

    Physical Intelligence

    San Francisco, CA
    3 days ago
  •  ...About the Team Training Runtime designs the core distributed machine-learning training runtime that powers everything...  ...About the Role As a Training: ML Framework Engineer, you will work on improving the...  ..., understanding distributed systems, and who cannot stand having... 
    Training
    Full time
    Work at office
    Relocation package

    OpenAI

    San Francisco, CA
    7 days ago
  • $184k - $272k

     ...Build and evolve our training infrastructure on Kubernetes with Infrastructure...  ..., multi-node distributed jobs, capacity...  ...operators and workflow engines that keep long-running...  ...observability across the ML stack — utilization,...  ...intensive production systems. ~ Excellent Python... 
    Training
    Full time
    Live in
    Work at office
    Local area
    Remote work
    Work from home
    Flexible hours

    Waabi

    San Francisco, CA
    19 days ago
  • $267k

     ...problems. We work closely with engineering, product and data science...  ...Generative AI, Computer Vision, ML Optimization, Geospatial AI....  ..., automation, and decision systems across Mobility, Delivery,...  ...experience with PyTorch and distributed training.Proven ability to set the... 
    Training
    Full time
    Work at office
    Remote work

    Uber

    San Francisco, CA
    3 days ago
  • $166k - $210.25k

     ...SummaryAs a Senior Applied ML Engineer on the Applied AI...  ...performance of our infrastructure. Your work will span...  ...optimization techniques.Build Systems: Design end-to-end ML...  ...: Architect, train, and deploy state-of-...  ...computer systems and distributed environments.Minimum... 
    Training
    Local area
    Worldwide

    DataBricks

    San Francisco, CA
    1 day ago
  • $70 per hour

     ....S. states. Software Engineering builds the brains of Waymo...  ...with hardware and systems engineers. If you’re a...  ...You will: Train and fine-tune a large...  ...a solid grounding in ML fundamentals: precision...  ...Experience with JAX/Flax, distributed training on TPUs or GPUs... 
    Training
    Hourly pay
    Full time
    Internship
    Summer internship
    Relocation package

    Waymo

    San Francisco, CA
    23 hours ago
  • $400k

     ...and judgment. We are training frontier models with...  ...training depends on systems that can keep large accelerator...  .... We’re looking for engineers to build and improve...  ...researchers on the infrastructure behind large-scale...  ..., coordinating distributed workloads, and recovering... 
    Training
    Full time
    Immediate start
    Visa sponsorship
    Work visa
    Relocation package

    Thinking Machines Lab

    San Francisco, CA
    3 days ago
  • # Infrastructure Engineer (Storage)Lightning AIUnited StatesFull timeRemoteUSD...  ...for developing, training, and deploying AI systems—designed to take ideas from...  ...power large-scale AI/ML training, inference, and...  ...reliability, and scaling distributed storage systems across our... 
    Training
    Work at office
    Remote work

    NEPSE Trading

    San Francisco, CA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to ML Infrastructure Engineer - Distributed Training & Systems. Be the first to apply!