Senior Software Engineer for LLM Evaluation

Role Overview

This role creates and curates high-quality datasets and reference solutions to train, benchmark, and advance large language models for coding tasks. You will produce precise code examples and corrections across multiple languages, evaluate and improve AI-generated code for efficiency and reliability, build verification agents that check code quality and identify error patterns, and work with researchers and cross-functional teams to raise the engineering capabilities of AI-driven coding systems.

About the organization

Based in San Francisco, the organization accelerates frontier AI research for labs and helps enterprises deploy advanced AI systems. It combines high-quality data, advanced training pipelines, and expert AI researchers specializing in coding, reasoning, STEM, multilinguality, multimodality, and agents to advance research and deliver production-ready AI systems that perform reliably at scale.

Key Responsibilities

• Create and curate code examples, turnkey solutions, and corrections for model training and benchmarking in Python, JavaScript including ReactJS, C and C++, Java, Rust, and Go.
• Evaluate, refine, and benchmark AI-generated code to ensure efficiency, scalability, reliability, and alignment with industry best practices.
• Collaborate with researchers and cross-functional teams to improve enterprise-level AI-driven coding solutions and measure performance against engineering benchmarks.
• Design and build agents and automated verification mechanisms that can verify a software solution and detect recurring error patterns.
• Formulate hypotheses and evaluate model capabilities across stages of the software engineering lifecycle, including prototyping, architecture and API design, production implementation, launch, experimentation, monitoring, and maintenance.
• Author clear, structured evaluation rationales and documentation that explain why a given solution is correct or how it should be improved.

Qualifications

• Several years of professional software engineering experience, including at least 2 years of continuous full-time experience at a top-tier product company, for example Google, Amazon, Apple, Meta, Netflix, Microsoft, Datadog, Dropbox, Shopify, PayPal, or IBM Research.
• Strong track record building full-stack applications and deploying production-grade, scalable software using modern languages and tooling.
• Deep knowledge of software architecture, system design, debugging, testing, and performing code quality assessments and reviews.
• Proficiency in one or more of these languages: Python, JavaScript and ReactJS, C, C++, Java, Rust, Go.
• Excellent oral and written communication skills, able to present concise, structured evaluation reasoning for researchers and engineers.

Work Terms

• Location: Remote.
• Engagement type: Contractor, no medical or paid leave provided.
• Commitment: Flexible, minimum 10 hours per week up to 40 hours per week. Partial overlap with Pacific Time hours is required.
• Duration: Initial 1 month engagement, starting next week, with potential extensions based on performance and fit.