Senior Software Engineer for LLM Evaluation and Repository Validation

Lead hands-on engineering work to create verifiable software engineering tasks drawn from public repository histories and to evaluate how large language models perform on realistic code problems. This role focuses on selecting and preparing high-quality open-source repositories, automating development environments, triaging issues, and measuring test coverage and bug-fix scenarios to produce LLM evaluation and training data.

• Analyze and triage GitHub issues across trending open-source libraries, identifying items suitable for LLM evaluation.
• Set up and configure repositories, including Dockerization and local environment automation, to reproduce and test real projects.
• Evaluate unit test coverage and test quality, and modify or run codebases locally to assess LLM-driven bug fixes.
• Collaborate with researchers to select repositories and incidents that challenge current LLM capabilities.
• Contribute as a technical lead, with opportunities to guide and coordinate a small team of junior engineers.
• Work on building verifiable, human-in-the-loop tasks derived from public repository histories, expanding dataset coverage across languages and difficulty levels.

• Strong experience with at least one of these languages: Python, JavaScript, Java, Go, Rust, C, C++, C#, or Ruby.
• Demonstrated proficiency with Git, Docker, and basic CI or pipeline setup.
• Proven ability to understand and navigate complex, real-world codebases and reproduce their environments locally.
• Comfortable running, modifying, and testing projects to evaluate behavior and quality.
• Experience with well-maintained, widely used public repositories, such as projects with 500+ stars, is required.
• Experience contributing to or evaluating open-source projects is a plus.

• Prior participation in LLM research, evaluation, or dataset construction projects.
• Experience building or testing developer tooling, automation agents, or developer-facing integrations.

• Location: Remote.
• Commitment: 20 hours per week, with some overlap required with Pacific Standard Time hours.
• Engagement type: Contractor assignment, no medical or paid leave provided.
• Contract duration: 3 months, with an expected start date as soon as next week.

• Fully remote work.
• Opportunity to work on cutting-edge AI and LLM evaluation projects with leading organizations in the field.