AI Evaluation Engineer

To support high-quality AI evaluations at scale, the full-time AI Evaluation Infrastructure Engineer will build infrastructure and tooling that enables rapid and reliable assessment of AI models and products, ensuring meaningful results and effective feedback loops. Key responsibilities Build an execution engine for scoring candidate versions against task sets quickly Create tooling to validate tasks from production logs before evaluation Develop systems for measuring judge-to-human agreement and monitoring drift over time Required qualifications Experience building production platforms or infrastructure, including data pipelines Strong statistical literacy, including knowledge of confidence intervals and variance Experience evaluating LLM or ML systems, or deep systems engineering experience with an interest in AI evaluation Product instinct for internal tools that are user-friendly and effective Ability to collaborate across ambiguous product, data, and engineering challenges