Software Engineer, ML Infrastructure

Responsibilities

• Own the backend of Arc’s Model Performance Dashboard and define how virtual cell model performance is measured.
• Design and build evaluation tools that let researchers compare models and measure the impact of each dataset version.
• Redesign and operate the single-cell data pipeline that creates training sets, including agent-based dataset discovery, processing, curation, and automated quality checks.
• Own lab-in-the-loop infrastructure that connects model performance results to experiments designed by Arc’s science leads.
• Build or select validation frameworks, tests, evaluation systems, QC gates, and policies for agent-written code and applications.
• Own the train-and-evaluate CI system, support multi-node training, diagnose distributed-training failures, and improve throughput and GPU utilization on Arc-operated clusters.
• Work directly with ML researchers, scientists, and stakeholders to define requirements and set technical direction.

Requirements

• Bachelor’s degree with 8 years of relevant experience or Master’s degree with 6 years of relevant experience.
• Experience owning a production data or ML platform end to end, including pipelines, storage, versioning, and technical direction.
• Experience designing and building systems with AI tools and agents and making their output verifiable through evaluations, QC gates, or tests.
• Ability to learn unfamiliar technical domains well enough to make sound design decisions and work directly with stakeholders without an intermediary PM.
• Experience improving others through code review, pairing, and documenting engineering patterns.
• Experience diagnosing and fixing performance issues in Python or PyTorch systems using profilers and measurements of GPU utilization, memory, and data loading.
• Preferred: experience with single-cell or perturbation data, including AnnData or h5ad.
• Preferred: experience building or running model evaluations, benchmarks, evaluation harnesses, and PyTorch/GPU inference.
• Preferred: experience with multi-node distributed training using DDP, FSDP, NCCL, or SLURM, or improving training throughput.
• Preferred: experience leading an effort with one or two other engineers.

Benefits

• Base salary is accompanied by an annual discretionary bonus and eligibility for the Institute’s Compound Cash Grant Program, which provides multi-year cash grants paid in semi-annual installments.
• This is an onsite role requiring work from the Palo Alto office at least three days per week.
• The role offers direct collaboration with ML researchers, scientists, and experimental biology teams on Arc’s virtual cell initiatives.