Senior Software Engineer for LLM Evaluation

Design and evaluate high-quality datasets and evaluations that advance large language models for code. You will work with researchers to curate code examples, produce precise solutions and corrections across multiple languages, assess AI-generated implementations for production readiness, and help build verification agents that detect errors and measure model capabilities across the software engineering lifecycle.

• Curate and author code examples, solutions, and corrections in Python, JavaScript including ReactJS, C and C++, Java, Rust, and Go for model training and benchmarking.
• Evaluate AI-generated code for correctness, efficiency, scalability, and reliability, and provide structured feedback and improvements.
• Collaborate with cross-functional research and engineering teams to align datasets and evaluations with industry performance benchmarks.
• Design and build agents or automated verification mechanisms that can validate solutions to software engineering tasks and identify recurring error patterns.
• Formulate and test hypotheses about model performance on steps of the software engineering cycle, including prototyping, architecture and API design, production implementation, launch, experimentation, monitoring, and operational maintenance.

• At least 3 years of professional software engineering experience.
• Proven experience building full-stack applications and deploying scalable, production-grade software using modern languages and tooling.
• Deep knowledge of software architecture, design, development practices, debugging, and code quality or review assessment.
• Strong written and oral communication skills, able to produce clear, structured evaluation rationales and code documentation.

• Commitment: flexible engagement, minimum 10 hours per week, up to 40 hours per week.
• Engagement type: Contractor, no medical or paid leave provided.
• Duration: 1 month initial term, with potential extension based on performance and fit.
• Location: Remote, candidates must be based in the United States, Canada, or Western Europe countries such as Austria, Belgium, France, Germany, and similar WEU locations.

Applicants must be legally located in the United States, Canada, or a Western European country as listed above. This role is offered to independent contractors only, and does not include employee benefits.

• The application takes approximately 15 to 30 minutes to complete.
• Completion of an AI video interview is required as part of the evaluation.