Member of Technical Staff

Responsibilities

• Build and own end-to-end evaluation suites for LLM capabilities, including task specifications, grading logic, and researcher dashboards.
• Operate evaluation harnesses as production services with sandboxed execution, tool calling, multi-turn and agentic tasks, deterministic replay, and caching.
• Own data pipelines for collection, filtering, deduplication, labeling, synthetic data generation, and quality gates.
• Build and maintain RL training environments, including reward functions, simulators, verifiers, and task generators.
• Collaborate with researchers to turn hypotheses into runnable evaluations and environments and interpret results.
• Measure evaluation regressions, environment health, throughput, and cost while identifying misleading metric changes.
• Improve research-code quality through testing, versioned datasets and evaluations, and clear training, inference, and evaluation boundaries.

Requirements

• At least 3 years of software engineering experience, including 1–2 years of hands-on LLM training, evaluation, or data-infrastructure experience.
• Deep Python fluency and practical distributed-systems experience across unreliable workers and machine fleets.
• Experience building evaluation infrastructure, including tasks, graders, and supporting systems.
• Working knowledge of supervised fine-tuning and reinforcement learning for language models.
• Production-oriented engineering practices, including testing and reproducibility.
• Ability to work effectively with researchers and turn ambiguous ideas into measurable systems.
• Preferred qualifications include agentic evaluation, tool use, code execution, sandbox environments, multi-turn tasks, training frameworks, inference stacks, data-labeling operations, human-preference collection, open-source evaluation frameworks, public benchmarks, verifiers, and simulation environments.

Benefits

• Full-time position with a US base salary range of $180,000–$450,000 annually.
• Additional compensation components and benefits may be included depending on the specific role and shared if an employment offer is extended.