Software Engineer, Evaluation Platform / Infra

Responsibilities

• Design, build, and maintain a platform for authoring, running, tracking, and analyzing model evaluations.
• Develop evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications end to end.
• Build abstractions for evaluation tasks, environments, graders, datasets, and model outputs.
• Improve reproducibility and trust through versioning, provenance, observability, failure recovery, and quality controls.
• Partner with researchers to convert bespoke workflows and bottlenecks into self-serve systems used across teams.
• Collaborate with the Research Tooling team and other pre-training, post-training, and applied teams.

Requirements

• Bachelor’s degree or equivalent practical experience in computer science, engineering, machine learning, or a related field.
• At least two years of post-graduate software engineering or ML engineering experience, excluding internships.
• Hands-on experience building or maintaining evaluations, benchmarks, graders, or model-quality systems for large language or multimodal models.
• Strong software engineering fundamentals and experience building reliable, maintainable systems.
• Proficiency in at least one backend programming language, with Python and Rust used primarily.
• Experience with databases, data pipelines, distributed systems, or other data-intensive infrastructure.
• Ability to work across the stack and own projects from problem discovery through deployment and operation.
• Experience collaborating with cross-functional partners and subject-matter experts.
• Preferred experience includes building frameworks, SDKs, developer tools, distributed job execution, workflow orchestration, sandboxed environments, large-scale data processing, polished data-inspection interfaces, model evaluation, research collaboration, or end-to-end work on small teams.

Benefits

• Annual salary range of $300,000–$475,000 USD.
• Health, dental, and vision benefits.
• Unlimited paid time off.
• Paid parental leave.
• Relocation support as needed.
• Visa sponsorship is available.
• The role is based in San Francisco, California.