Software Engineer for AI Model Evaluation

This role focuses on advancing the evaluation and development of frontier coding agents, blending AI research, software engineering, and model evaluation. You will design benchmarks, methodologies, and data systems that define how next-generation coding models are assessed and enhanced. Key Responsibilities

• Design and own evaluation frameworks for coding agents, including benchmark specifications, scoring methodologies, rubrics, and quality standards.
• Lead end-to-end research initiatives aimed at measuring and improving coding model performance across various software engineering tasks.
• Develop high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes, identifying systematic weaknesses and translating findings into actionable improvements for training and evaluation.
• Build tooling and infrastructure to support large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Establish best practices for coding-agent assessment, ensuring methodological rigor, reproducibility, and measurement quality.
• Collaborate closely with researchers, engineers, and applied AI teams to design experiments and evaluate emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives that communicate model performance and insights.

• Strong software engineering background with expertise in Python, C++, or comparable programming languages.
• 3+ years of experience in software engineering, machine learning, AI research, evaluation, or related technical disciplines.
• Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.
• Proven ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills with the ability to investigate model behavior and derive insights from complex technical systems.
• Excellent written and verbal communication skills, with the ability to clearly articulate technical findings to diverse audiences.
• Comfortable operating in fast-moving research environments with significant ambiguity and evolving priorities.

• Experience working on frontier AI systems, coding agents, or model evaluation research.
• Deep interest in understanding how data, evaluations, and feedback mechanisms influence model capabilities.
• Track record of independently driving ambiguous technical or research projects from conception to execution.
• Experience designing benchmarks or datasets for machine learning systems at scale.
• Familiarity with agentic workflows, tool use, reinforcement learning, or post-training methodologies.
• Publications, open-source contributions, or demonstrated technical leadership in AI research.

Full-time position with remote work flexibility.

Annual salary range of $220, 000 - $500, 000.

Open to candidates with the required skills and experience, regardless of location.