Senior Software Engineer for LLM Evaluation [Remote]

This role focuses on creating innovative datasets for training and advancing large language models, working closely with researchers to enhance AI-driven coding solutions. As a Software Engineering evaluator, you will curate code examples, provide precise solutions, and make corrections, primarily using Python, while also engaging with JavaScript (including ReactJS), C/C++, Java, Rust, and Go.

Key Responsibilities:

• Curate code examples, build solutions, and correct code primarily in Python, with additional work in JavaScript (including ReactJS), C/C++, Java, Rust, and Go.
• Evaluate and refine AI-generated code to ensure efficiency, scalability, and reliability.
• Collaborate with cross-functional teams to enhance AI-driven coding solutions against industry performance benchmarks.
• Develop agents and automated verification tools in Python to verify code quality and identify error patterns.
• Hypothesize on software engineering cycle steps (prototyping, architecture design, API design, production implementation, launch, experiments, monitoring, operational maintenance) and evaluate model capabilities.
• Design verification mechanisms to automatically verify solutions to software engineering tasks.

Qualifications:

• Minimum of 3 years of software engineering experience.
• Strong expertise in Python, with deep knowledge of frameworks, tooling, and best practices for building production-grade software.
• Experience in building full-stack applications and deploying scalable software using modern languages and tools.
• Deep understanding of software architecture, design, development, debugging, and code quality/review assessment.
• Excellent oral and written communication skills for clear, structured evaluation rationales.

Work Terms:

• Flexible engagement with a minimum commitment of 10 hours per week, up to 40 hours per week.
• Contractor position (no medical/paid leave).
• Initial duration of 1 month with potential extensions based on performance and fit.
• Candidates must be based in the United States.

Compensation:

Details regarding compensation will be discussed during the interview process.

Eligibility:

• The application process takes 15, 30 minutes.
• Completion of an AI video interview is required.