Agentic AI Evaluation Engineer

Job Summary The Agent Evaluation team is responsible for testing whether AI agents return the correct and expected responses. We build the framework, metrics, and test cases that validate agent behavior, accuracy, and reliability before release. Our goal is to ensure agents perform consistently and meet product and user expectations. Responsibilities Design and develop agent evaluation pipelines across development, staging, and production environments. Define and standardize evaluation metrics and benchmarks for conversational AI quality (accuracy, relevance, CX, safety). Build automated and human-in-the-loop evaluation systems to assess agent performance. Manage and curate evaluation datasets, test sets, and annotation workflows. Enable continuous evaluation and monitoring of agents in production. Integrate evaluation into CI/CD pipelines to support safe and efficient releases. Conduct experiments, A/B testing, and case studies to drive improvements in agent quality. Partner with engineering and product teams to deliver high-quality AI solutions. Create technical documentation and drive best practices across teams. Mentor junior engineers and contribute to team growth. Qualifications 5‑7 years of relevant experience in AI evaluation, customer support AI, or chatbot platforms. Strong understanding of responsible AI, including bias, fairness, and hallucination mitigation. Proficiency with large language models (LLMs) and machine learning (ML) techniques. Experience with benchmarking, CI/CD, and automation of evaluation pipelines. Excellent communication skills and ability to work cross‑functionally. Preferred Skills Experience in customer support AI or chatbot platforms. Understanding of responsible AI (bias, fairness, hallucination mitigation). Education & Certifications Bachelor’s degree in a related field (preferred but not required). Relevant certifications are a plus. Compensation Primary location: Pay range $142,651.46 – $213,977.19. Base pay is one part of the Total Rewards package and may include bonuses. Benefits Eligible employees receive comprehensive benefits covering physical, financial, and emotional well-being, including health, dental, vision, and various support resources. Equal Opportunity Comcast is an equal opportunity workplace. We will consider all qualified applicants for employment without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, veteran status, genetic information, or any other basis protected by applicable law. #J-18808-Ljbffr Blueface Ltd