Senior Software Engineer, Python for LLM Evaluation

Role Overview: This position focuses on developing evaluation and training datasets for large language models (LLMs) to tackle real-world software engineering challenges. The role involves creating verifiable software engineering tasks based on public repository histories, enhancing dataset diversity across programming languages and difficulty levels.

Key Responsibilities:

• Analyze and triage GitHub issues across trending open-source libraries.
• Set up and configure code repositories, including Dockerization and environment setup.
• Evaluate unit test coverage and quality.
• Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
• Collaborate with researchers to design and identify repositories and issues that pose challenges for LLMs.
• Lead a team of junior engineers on collaborative projects.

Qualifications:

• Minimum of 3 years of overall experience in software engineering.
• Strong proficiency in Python or similar programming languages.
• Experience with Git, Docker, and basic software pipeline setup.
• Ability to understand and navigate complex codebases.
• Comfortable running, modifying, and testing real-world projects locally.
• Experience contributing to or evaluating open-source projects is a plus.

Nice to Have:

• Previous participation in LLM research or evaluation projects.
• Experience building or testing developer tools or automation agents.

Work Terms:

• Commitments Required: At least 4 hours per day and a minimum of 20 hours per week with 4 hours of overlap with PST. Options available for 20, 30, or 40 hours per week.
• Employment Type: Contractor assignment (no medical/paid leave).
• Duration of Contract: 3 months; expected start date is next week.
• Location: Candidates must be based in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, or Mexico.

Compensation: Competitive compensation based on experience.

Eligibility: Open to candidates with the required skills and location constraints as specified.

Evaluation Process: The evaluation will consist of two rounds of interviews, totaling approximately 75 minutes (60 minutes technical + 15 minutes technical & cultural discussion).