Member of Technical Staff, Coding Research

Role Overview

Advance the evaluation and development of frontier coding agents by designing the benchmarks, methodologies, and data systems used to measure and improve next-generation coding models. This role combines AI research, software engineering, and model evaluation.

Key Responsibilities

• Design and own evaluation frameworks for coding agents, including benchmark specifications, scoring methods, rubrics, and quality standards.
• Lead end-to-end research initiatives to measure and improve coding-model performance across a range of software engineering tasks.
• Develop high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes, identify systematic weaknesses, and translate findings into improvements for training and evaluation.
• Build tooling and infrastructure for large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Establish rigorous, reproducible best practices for coding-agent assessment and measurement quality.
• Partner with researchers, engineers, and applied AI teams to design experiments and assess emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives that communicate model performance and insights.

Qualifications

• Strong software engineering background and expertise in Python, C++, or comparable programming languages.
• At least 3 years of experience in software engineering, machine learning, AI research, evaluation, or a related technical discipline.
• Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.
• Ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills for investigating model behavior and deriving insights from complex technical systems.
• Excellent written and verbal communication skills, including the ability to explain technical findings to varied audiences.
• Comfort working in a fast-moving research environment with ambiguity and evolving priorities.

Preferred Qualifications

• Experience with frontier AI systems, coding agents, or model-evaluation research.
• Interest in how data, evaluations, and feedback mechanisms influence model capabilities.
• A record of independently driving ambiguous technical or research projects from concept through execution.
• Experience designing benchmarks or datasets for machine learning systems at scale.
• Familiarity with agentic workflows, tool use, reinforcement learning, or post-training methods.
• Publications, open-source contributions, or demonstrated technical leadership.

Work Terms

• Full-time, remote role.

Compensation

• Annual compensation range: $400, 000 to $800, 000.