Senior Software Engineer, LLM Evaluation

Role Overview

Help advance large language models by creating high-quality software engineering datasets, benchmarks, and evaluations alongside AI researchers. You will work across Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go to develop reference solutions, improve code samples, and assess AI-generated output for real-world engineering quality.

Key Responsibilities

• Curate code examples, build solutions, and correct code across Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go for AI model training initiatives.
• Evaluate and refine AI-generated code for efficiency, scalability, and reliability.
• Partner with cross-functional teams to improve AI-driven coding solutions against industry performance benchmarks.
• Build agents that verify code quality and identify recurring error patterns.
• Develop and assess model capabilities across the software engineering lifecycle, including prototyping, architecture design, API design, production implementation, launch, experiments, monitoring, operations, and maintenance.
• Design automated verification mechanisms for solutions to software engineering tasks.

Qualifications

• Several years of software engineering experience, including at least 2 years of continuous full-time experience at a top-tier product company.
• Strong full-stack application development experience and the ability to deploy scalable, production-grade software with modern languages and tools.
• Deep knowledge of software architecture, design, development, debugging, and code-quality and code-review assessment.
• Excellent written and verbal communication skills, with the ability to provide clear, structured evaluation rationales.

Work Terms

• Remote contract engagement.
• Flexible schedule of at least 10 hours per week, with up to 40 hours per week available.
• Some overlap with Pacific Time is required.
• Independent contractor role, with no medical benefits or paid leave.