SWE-Bench AI Task Auditor - Freelance AI Trainer Project

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a SWE-Bench AI Task Auditor - Freelance AI Trainer Project based in Austria.

This freelance opportunity is designed for experienced software engineering professionals who want to contribute their expertise to the development and evaluation of advanced AI systems.
You will audit software engineering tasks used to train and assess AI models, ensuring they reflect realistic and technically rigorous development scenarios.
The role focuses on evaluating complex codebases, validating task logic, and determining whether tests and evaluation criteria are reliable and reproducible.
You will identify integration problems, test failures, and technical inconsistencies and provide actionable feedback to improve task quality.
Your expertise will directly contribute to making AI training workflows more accurate, practical, and aligned with real-world software engineering standards.
The project offers a flexible freelance environment where your specialized technical knowledge can have a meaningful impact on AI development.
This is an opportunity to apply deep software engineering experience to challenging technical evaluation work while working independently on project-based assignments.

Accountabilities

• Evaluate software engineering tasks for technical accuracy, realism, solvability, reproducibility, and alignment with established SWE-Bench standards.
• Review complex codebases and real-world application scenarios to determine whether tasks accurately reflect practical software engineering challenges.
• Assess the reliability and completeness of tests, evaluation criteria, and expected outcomes associated with assigned tasks.
• Identify codebase integration issues, logic errors, test failures, and other technical inconsistencies that could affect task quality or evaluation accuracy.
• Troubleshoot complex technical scenarios and provide clear, actionable feedback to improve task implementation and evaluation workflows.
• Apply specialized software engineering knowledge to ensure AI training and evaluation tasks meet high standards of technical rigor and practical relevance.
• Document findings clearly and consistently so that identified issues can be addressed efficiently.

Requirements

• Demonstrable professional experience and deep technical expertise in software engineering, including complex codebase navigation and real-world application development.
• Strong familiarity with software engineering concepts and the ability to evaluate tasks against practical development standards, including SWE-Bench-related requirements.
• Strong analytical and problem-solving skills, with the ability to rigorously test, troubleshoot, and evaluate complex technical scenarios.
• Strong attention to detail and the ability to identify subtle logic, integration, testing, and reproducibility issues.
• Excellent written communication skills for providing precise, constructive, and actionable technical feedback.
• Ability to work independently and apply specialized expertise to project-based freelance assignments.
• Candidates will be evaluated for a specific area of specialization based on their professional background; expertise across multiple domains is not required.
• Ability to provide a secure computer and reliable high-speed internet connection for freelance project work.

Benefits

• Compensation of $70–$100 per hour, with the exact rate determined based on experience, expertise, and geographic location.
• Final compensation may vary from the published range following evaluation of your qualifications and location.
• Freelance contract structure offering flexibility for project-based work.
• Remote work environment.
• Opportunity to contribute directly to the training and evaluation of advanced AI systems.
• Opportunity to apply specialized software engineering expertise to technically challenging AI evaluation projects.
• Health insurance, paid time off, and other company-sponsored employee benefits are not provided under the freelance contract.
• Contractors are responsible for supplying their own secure computer and high-speed internet connection.

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.