AI Evaluation Specialist

AI Evaluation Specialist

Client is engaging AI Evaluation Specialists to assess and elevate the quality of AI assistant outputs for an enterprise AI training initiative. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.

Scope of Work

1. Evaluate AI-generated outputs against detailed rubrics and defined quality standards, focusing on accuracy, relevance, and adherence to guidelines.
2. Apply consistent, impartial judgment across a high volume of examples, ensuring a fair and reliable assessment process.
3. Identify reasoning gaps, tool-use failures, or logic errors in AI assistant responses, providing actionable feedback for iterative improvement.
4. Produce clear, concise written feedback on both strengths and areas for improvement, directly influencing model refinement and AI adoption practices.
5. Participate in discussions regarding rubric interpretation and evolving quality standards, contributing to process optimization and best practices.
6. Maintain meticulous documentation of evaluations and recommendations, ensuring transparency and traceability in assessment workflows.

Preferred Qualifications

1. Experience in grading, quality assurance, editorial review, assessment, annotation, or similar fields demanding careful analysis and detailed feedback.
2. Advanced, daily use of AI assistants (such as ChatGPT, Claude, or similar) as an essential work and productivity tool.
3. Demonstrated ability to synthesize complex information and communicate findings effectively in writing.
4. Background in process improvement, rubric development, or operational quality assessment in an enterprise or educational context.
5. Strong critical thinking skills with a focus on consistency, integrity, and fairness in evaluations.
6. Comfort working independently on large volumes of similar examples while maintaining high attention to detail.
7. Collaborative mindset for sharing insights, discussing ambiguous cases, and refining evaluation criteria as models evolve.

About the company

Avance Consulting is a global leader in innovative talent solutions for diverse industries. Since 2007, we have been serving nearly a third of Fortune 500 companies and most of the top 100 technology companies worldwide.

Our team of 700+ experts offers Engineering Services, Information Technology, Digital and Executive Search solutions from our offices in North America, UK, Europe and APAC.

We understand the local markets and cultures and tailor our services to meet the needs of our clients. Our clients value our flexibility and agility, which enable us to deliver high-quality results at any scale across the globe.

This allows us to work with numerous industries at varying scale across the globe while ensuring the excellent service standard that our clients have come to expect.

At Avance you can learn new skills and technologies, have more advancement opportunities and career growth, being a part of our productive atmosphere, and a positive reinforcement culture.