Senior Software Engineer for LLM Evaluation & Repository Validation

Role Overview: This position focuses on developing and evaluating large language model (LLM) training datasets to address real-world software engineering challenges. The role emphasizes building verifiable software engineering tasks using public repository histories, while enhancing dataset diversity across various programming languages and difficulty levels.

Key Responsibilities:

• Analyze and triage GitHub issues for trending open-source libraries.
• Set up and configure code repositories, including Dockerization and environment setup.
• Evaluate unit test coverage and quality.
• Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
• Collaborate with researchers to identify challenging repositories and issues for LLMs.
• Lead a team of junior engineers on collaborative projects.

Qualifications:

• Minimum of 3 years of overall experience.
• Strong proficiency in Python.
• Experience with Git, Docker, and basic software pipeline setup.
• Ability to navigate and understand complex codebases.
• Comfortable running, modifying, and testing real-world projects locally.
• Experience contributing to or evaluating open-source projects is a plus.

Nice to Have:

• Previous involvement in LLM research or evaluation projects.
• Experience in building or testing developer tools or automation agents.

Work Terms:

• Commitment: At least 4 hours per day and a minimum of 20 hours per week with 4 hours overlap with PST. Options for time commitment include 20 hrs/week, 30 hrs/week, or 40 hrs/week.
• Employment Type: Contractor assignment (no medical/paid leave).
• Contract Duration: 3 months; expected start date is next week.
• Location: Candidates from India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, or Mexico are eligible.

Compensation: Competitive compensation commensurate with experience.

Eligibility:

• Must be eligible to work in the specified locations.

Evaluation Process:

• Two rounds of interviews: 60 minutes technical and 30 minutes technical & cultural discussion.