Customer Support Specialists, AI Response Evaluation

Customer Support Specialists, AI Response Evaluation at Mpathic Keeping the human in AI. mpathic is a trusted leader in advancing quality and safety in AI systems through expert-led evaluation and human data. We partner with leading technology companies to support red teaming, trust & safety, expert annotation, and model evaluation across high-stakes domains. About the Role mpathic is seeking part-time Customer Support Specialists to support a red-teaming and quality assurance (QA) campaign focused on evaluating AI system behavior in customer-facing interactions. In this role, you will review AI-generated responses and multi-turn conversations to assess quality, accuracy, tone, and safety—similar to evaluating customer support agents. You will identify issues such as incorrect information, poor customer experience, inappropriate tone, or failure to resolve user needs effectively. This is not a traditional support role. Instead, it focuses on evaluating AI performance, applying your customer support expertise to assess how well AI systems handle real-world customer interactions. What You’ll Be Working On You will help identify, prevent, and characterize risks that emerge when users interact with AI-powered support systems. Responsibilities may include: Reviewing AI-generated customer support responses for accuracy, clarity, and helpfulness Evaluating tone, empathy, and professionalism in customer interactions Identifying poor support behaviors (e.g., dismissiveness, lack of resolution, incorrect guidance) Assessing whether AI responses appropriately address customer intent and resolve issues Evaluating how AI handles frustration, confusion, or escalating user behavior Identifying gaps in responses, including missing steps, unclear instructions, or lack of follow-through Detecting overconfident, misleading, or incorrect responses Evaluating multi-turn conversations for consistency, escalation handling, and resolution quality Identifying patterns of agreement without problem-solving (e.g., empty empathy or unhelpful reassurance) Participating in or reviewing red teaming exercises to surface failure modes in support scenarios Supporting quality assurance (QA) to ensure consistency across evaluations Documenting edge cases, failure patterns, and customer experience risks Providing structured written feedback to internal teams Collaborating with interdisciplinary teams on AI quality, safety, and evaluation frameworks Maintaining strict confidentiality and quality standards What We’re Looking For Successful candidates are detail-oriented, customer-focused, and experienced in evaluating or delivering high-quality support in fast-paced environments. Professional experience in one or more of the following: Customer support (chat, email, or phone) Call centers or contact centers Technical support or help desk roles Strong understanding of: High-quality customer experience and service standards Effective communication in customer-facing interactions Common support workflows (triaging, troubleshooting, escalation) How customers interpret and respond to support interactions Ability to identify: Poor or incomplete resolutions Incorrect or misleading information Inappropriate tone (e.g., robotic, dismissive, overly casual) Lack of empathy or ineffective de-escalation Failure to follow support best practices Strong written communication skills and ability to clearly explain reasoning Experience with or interest in: Evaluating conversations or QA for support teams Working with AI tools, chatbots, or automated systems Assessing how systems perform under real customer behavior Comfort with: Tech tools and platforms (Slack, spreadsheets, dashboards) Evaluating AI-generated responses (no coding required, but must be tech-comfortable) Ambiguity, iteration, and feedback-driven workflows Willingness to: Sign NDAs and work with sensitive content Nice to Have (Not Required) Experience in QA, coaching, or training within support teams Background in trust & safety or content moderation Experience with chatbot or AI-assisted support tools Familiarity with CX metrics (CSAT, QA scoring, resolution rate) Interest in AI, automation, or improving customer experience systems Compensation $30-40/hour, depending on experience and specific project tasks/difficulty #J-18808-Ljbffr