Software Engineer - LLM Evaluation & Repository Validation

About the Role

We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.

Key Responsibilities

Review and triage issues in open-source GitHub projects.
Prepare repositories for evaluation, including containerisation and environment automation.
Assess how thorough and well written the unit tests are.
Edit and run code locally to see how LLMs perform on bug-fixing tasks.
Help pick repositories and issues that are hard for LLMs.
Write up findings clearly for researchers and reviewers.
Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.

Who We're Looking For

Required Qualifications

3+ years of software engineering experience.
Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
Proficiency with Git, Docker and basic pipeline setup.
Ability to navigate complex code bases and run real-world projects locally.
Fluent written English.
Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.

Preferred Qualifications

Prior LLM research or evaluation projects.
Experience building or testing developer tools or automation agents.
Open-source contribution or evaluation experience.

Rights, Confidentiality and Data Handling

Submit only work that you created yourself or that you have the right to share.
Do not submit material covered by an employer, client or third-party confidentiality agreement, or material you have no right to share.
Do not submit export-controlled, military or defence-related technical data.
Remove credentials, secrets and personal data from anything you submit.
Rex may ask for evidence of ownership or licence, may verify submissions, and may reject or remove any submission.

Legal and Compliance Notice

All content you submit must be lawful, and you must have the right to share it.
Do not submit confidential or proprietary material belonging to an employer, client or third party, or personal data without a lawful basis.
Rex may verify identity, credentials, ownership and licensing; may reject or remove any submission; and may report unlawful submissions to the relevant authorities.
Engagement and payment are subject to successful verification and to applicable sanctions, export-control, tax and employment laws.
The contractor agreement will set out ownership and licence terms for the work you deliver (to be confirmed by Rex Legal).

Compensation

USD $50–100 per hour, adjusted by country of residence.