Director AI Evaluation (, PA, United States)

Overview

Location: Work from home (Pennsylvania). Shift: Days (United States of America). Scheduled Weekly Hours: 40. Exemption Status: Yes. Job summary: The Director of AI Evaluation owns how Geisinger defines, proves, and sustains quality across its AI portfolio, including internally built models and vendor-provided systems. This hands-on technical leader also manages the people who build and validate AI, including data scientists and senior analysts who evaluate AI systems. The Director sets the standard for quality, leads the team that enforces it, and reports findings to the VP of AI, executive leaders, and the board. This role combines people leadership with technical authority to guide major AI programs toward evidence that withstands scrutiny.

Responsibilities

• Reports to the VP of AI; directly manages the data science line and matrix-manages the Senior Analysts, AI Evaluation.
• Determine the quality standard for high-value AI initiatives (internal or vendor-provided) from design through production; hold both built and bought systems to the same standard.
• Own the methodology for pre-production validation and live production monitoring.
• Lead the health and development of the data science and evaluation teams, providing hands-on technical guidance on validation studies, equity audits, monitoring plans, and escalation playbooks.
• Maintain the evaluation toolkit, reusable playbooks, and templates to accelerate new programs.
• Translate failure modes into concrete, measurable production-monitoring metrics; collaborate with the AI Platform team on backend implementation.
• Track AI system performance against clinical thresholds; monitor user adoption, engagement, and time-to-action; distinguish genuine value from alarm fatigue.
• Link each AI initiative to the outcomes it aims to improve (e.g., mortality, time-to-treatment, boarding time, denial rate, cost per case) using pre-launch baselines and appropriate horizons.
• Monitor equity by assessing performance gaps across relevant subgroups to surface disparate impact early.
• Ensure compliance with organizational policies and procedures; perform duties aligned with job obligations.

Qualifications

• People-leadership experience — managing, developing, and growing technical staff; building teams.
• Strong foundation in experimental design and causal inference; ability to select appropriate methods for given situations.
• Hands-on experience designing and running model evaluation studies in production settings.
• Experience evaluating LLM or generative AI systems, or comparable complex ML systems with ambiguous or noisy ground truth.
• Ability to translate ambiguous failure modes into concrete, defensible evaluation designs and monitoring metrics.
• Strong fluency in Python and SQL; comfortable with modern ML tooling and cloud-native data environments.
• Experience evaluating fairness and equity in ML systems.
• Clear written communication for evaluation memos and specifications used by non-technical decision-makers.
• Healthcare, clinical, or regulated-industry experience strongly preferred.

Education & Experience

• Bachelor’s Degree in Related Field (Required).
• Minimum 8 years related work experience (Required).
• Minimum 3 years managerial/supervisory experience (Required).

Values

• OUR PURPOSE & VALUES: Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.
• KINDNESS: Treat everyone as we would hope to be treated ourselves.
• EXCELLENCE: Strive for excellence in all we do.
• LEARNING: Share knowledge to prepare caregivers for tomorrow.
• INNOVATION: Seek new and better ways to care for patients and the community.
• SAFETY: Provide a safe environment for patients and members.

We offer healthcare benefits for full-time and part-time positions from day one, including vision and dental coverage. We value collaboration, cooperation, and collegiality and believe a diverse workforce strengthens our team. We are an affirmative action, equal opportunity employer. All qualified applicants will receive consideration for employment regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status. For more information, visit or connect with us on social media.