Member of Technical Staff - Evaluations

Responsibilities

• Own the internal evaluation suite and infrastructure end to end, keeping it reliable, optimized, reproducible, and scalable.
• Design held-out evaluations for capabilities including agentic workflows, computer use, and cybersecurity, with documented construction and contamination checks.
• Build evaluations targeted at small edge models and debug unexpected scores by reviewing individual generated outputs.
• Convert customer and open-source feedback into environments for data, training, and evaluation, and partner with other teams on their evaluation needs.
• Release open-source evaluations, write papers, and publish failure-mode reviews of popular evaluations.
• Drive post-training decisions through internal evaluations and help co-own the evaluation function.

Requirements

• Strong evaluation-design ability, including translating high-level capabilities into accurate, general-purpose, hard-to-game evaluations that distinguish model quality.
• Prior experience building evaluations that demonstrate coverage, reproducibility, contamination handling, and useful signal, including at least one benchmark, evaluation paper, or substantial contribution built from scratch.
• Experience with evaluation fundamentals including contamination, saturation, judge bias, and variance.
• Hands-on experience running and managing evaluations within an evaluation, training, engineering, or DevOps team.
• Strong software engineering skills sufficient to own evaluation infrastructure and debug inference and infrastructure failures at scale independently.
• Helpful qualifications include model training experience, publications or research experience, deep systems expertise, and familiarity with small edge models.

Benefits

• Full-time hybrid role based in San Francisco.
• Competitive base salary plus equity; final offers reflect skills, experience, and location.
• Unlimited paid time off and company-wide Refill Days and long weekends.
• Comprehensive health coverage, with details varying by location.
• Paid leave for pregnancy, parenting, medical needs, and caring for family.
• Real compute for running evaluations at scale and the opportunity to open-source and publish evaluation work.