Data Scientist

Data Scientist

Location - Louisville, Kentucky (Hybrid)

Duration: 6+ months

Mode of interview: 2 videos

We are seeking a Data Scientist to help build and maintain a robust evaluation framework for conversational AI systems. This role will focus on developing automated AI quality measurement solutions, including LLM-as-a-Judge systems, to assess hallucination rates, intent accuracy, transcript quality, and responsible AI metrics at scale. As a key member of our AI Quality team, you will work closely with annotation specialists, product teams, and governance stakeholders to ensure our AI experiences are accurate, safe, compliant, and customer-centric.

Design and develop LLM-as-a-Judge evaluation frameworks for conversational AI systems.

Build automated monitoring solutions to evaluate millions of voice and chat-based customer interactions.

Collaborate with user experience teams to identify core metrics and evaluation criteria.

Validate model performance against human-labeled datasets using statistical best practices.

Measure and report precision, recall, false-positive rates, and false-negative rates.

Develop statistical sampling methodologies and quality measurement frameworks.

Continuously calibrate and improve evaluation models as production AI systems evolve.

Analyze hallucination rates, intent classification accuracy, transcript quality (WER), guardrail effectiveness, vulnerability to jailbreak attempts, and fairness metrics.

Create dashboards and reporting to support governance, legal, compliance, and Responsible AI reviews.

Collaborate with annotation teams to improve label quality and gold-standard datasets.

Support future multilingual AI evaluation initiatives.

Master's degree or PhD in Data Science, Statistics, Computer Science, Machine Learning, or related field, or equivalent experience.

3+ years of experience in data science, machine learning, analytics, or AI model evaluation.

Experience analyzing voice-based language data.

Strong experience with Python and data science tooling.

Experience designing experiments and evaluating machine learning models.

Knowledge of statistical analysis, sampling methodologies, and model validation techniques.

Experience communicating technical findings to both technical and non-technical stakeholders.

Experience with Generative AI, LLMs, or conversational AI systems.

Familiarity with AI safety, Responsible AI, fairness, bias, and governance frameworks.

Experience developing automated evaluation systems.

Experience with cloud-based AI platforms and experimentation environments.