Senior Software Engineer for LLM Evaluation

Role Overview

Build and evaluate high-quality datasets and verification tools that improve large language models on software engineering tasks. You will curate and correct code examples, assess AI-generated solutions across multiple languages, and design automated checks and agents to measure and improve model performance in real engineering workflows.

• Create datasets used for training and benchmarking LLMs, working closely with researchers to define scope and quality standards.
• Curate code examples, author precise solutions, and correct code in Python, JavaScript including ReactJS, C and C++, Java, Rust, and Go.
• Evaluate and refine AI-generated code for efficiency, scalability, reliability, and maintainability.
• Develop verification mechanisms and agents that automatically verify solutions and surface error patterns.
• Collaborate with cross-functional teams to align AI-driven coding solutions with industry performance benchmarks and enterprise needs.
• Formulate and test hypotheses about software engineering steps, including prototyping, architecture design, API design, production implementation, launch, experimentation, monitoring, and operational maintenance, and evaluate model capabilities on these phases.

• Minimum 3 years of professional software engineering experience.
• Proven experience building full-stack applications and deploying scalable, production-grade software using modern languages and tools.
• Strong knowledge of software architecture, system design, debugging, and code quality assessment and review practices.
• Practical experience with one or more of the following languages: Python, JavaScript (including ReactJS), C, C++, Java, Rust, Go.
• Excellent oral and written communication skills, with the ability to produce clear, structured evaluation rationales and documentation.

• Engagement type: Contractor, no medical or paid leave provided.
• Commitment: Flexible, minimum 10 hours per week, up to 40 hours per week.
• Duration: 1 month initial term, with potential extensions based on performance and fit.
• Work location: Remote. Candidates must be based in the United States, Canada, or Western European countries, for example Austria, Belgium, France, Germany.

• Candidates must be located in the US, Canada, or a WEU country as listed above. Employment is on a contractor basis; confirm you can contract under these terms before applying.

• The application requires about 15 to 30 minutes to complete.
• An AI video interview is a required step in the evaluation process.