AI QA Trainer - LLM Evaluation [Remote]

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a AI QA Trainer - LLM Evaluation based in Australia.

This role offers the opportunity to contribute directly to the evolution of advanced AI systems by improving their reliability, accuracy, and safety.
You will work on evaluating large language models through structured testing, quality analysis, and adversarial scenarios.
The position combines AI quality assurance, data evaluation, and technical problem-solving to strengthen model performance.
You will design evaluation frameworks, identify failure patterns, and help improve AI reasoning capabilities across diverse use cases.
Working with cutting-edge AI technologies, you will influence how future systems handle information, reasoning, and real-world tasks.
This is a remote contract opportunity for professionals passionate about AI quality, testing, and responsible model development.

Accountabilities:

The role focuses on ensuring the quality, reliability, and safety of advanced AI models through comprehensive evaluation and testing processes. You will collaborate on identifying weaknesses, improving evaluation methodologies, and creating actionable insights that enhance model performance.

• Evaluate large language models across areas such as factual accuracy, reasoning quality, hallucination detection, safety, and reliability.
• Design and execute test plans, regression suites, and evaluation frameworks to measure model performance.
• Identify and document model failures, including prompt vulnerabilities, incorrect outputs, bias issues, and reasoning inconsistencies.
• Develop clear evaluation rubrics, pass/fail criteria, and quality benchmarks for AI system assessment.
• Perform adversarial testing, red-teaming exercises, and robustness evaluations to identify potential risks.
• Verify grounding, retrieval-augmented generation outputs, tool usage accuracy, and workflow reliability.
• Analyze model behavior and provide recommendations for improving prompts, guardrails, and evaluation strategies.
• Support automation initiatives using tools such as Python or SQL to improve testing efficiency.
• Maintain detailed documentation of findings, root-cause analysis, and reproducible issue reports.
• Collaborate with technical teams to improve AI evaluation metrics, dashboards, and quality monitoring processes.

Requirements:

The ideal candidate brings strong experience in AI quality assurance, machine learning evaluation, software testing, or related technical disciplines. You should be comfortable analyzing complex AI behaviors and communicating technical findings clearly.

• Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, Statistics, or a related technical field preferred.
• Experience testing, evaluating, or validating AI/ML systems, particularly large language models.
• Strong understanding of LLM evaluation concepts, including hallucination detection, factual consistency, safety testing, and reliability analysis.
• Experience designing QA frameworks, evaluation criteria, regression testing processes, or quality benchmarks.
• Familiarity with prompt engineering, system prompts, retrieval-augmented generation (RAG), and AI safety practices.
• Experience with adversarial testing, red teaming, bias assessment, or compliance-oriented AI reviews.
• Knowledge of test automation tools and programming languages such as Python or SQL.
• Familiarity with LLM evaluation tools, experiment tracking platforms, or AI testing frameworks.
• Strong analytical skills with the ability to identify patterns, investigate failures, and propose improvements.
• Excellent written and verbal communication skills, with the ability to clearly explain technical reasoning and findings.

Benefits:

• Competitive contractor compensation ranging from $6 to $65 per hour , depending on experience, expertise, and geographic location.
• Fully remote work environment with flexibility to contribute from anywhere.
• Opportunity to work on advanced AI systems and influence the development of future AI capabilities.
• Hands-on exposure to cutting-edge language models, evaluation methodologies, and AI safety practices.
• Ability to apply your technical expertise to impactful real-world AI improvement projects.
• Flexible contract-based opportunity designed for experienced AI and QA professionals.
• Contractors provide their own secure computer and high-speed internet connection.
• Company-sponsored benefits such as health insurance and paid time off are not included for this contract role.

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.