Director AI Evaluation

Job Summary

The Director of AI Evaluation owns how Geisinger defines, proves, and sustains quality across its entire AI portfolio – internally built models and vendor‑provided systems alike. This is a hands‑on technical leader who also manages the people who build and prove: the data scientists who develop production machine learning and fine‑tuned AI systems, and the senior analysts who evaluate them. Every high‑value AI initiative, bought or built, needs a single credible standard for what constitutes quality, who validates it, and how it stays good in production. The Director sets that standard, leads the team that enforces it, and reports findings to the VP of AI, executive leaders, and the board. The role is a manager who develops a multidisciplinary team, a technical authority who defines evaluation methods across the enterprise, and a quality owner who guides every major AI program toward evidence that withstands scrutiny.

Job Duties

• Reports to the VP of AI; directly career‑manages the data science line and matrix–manages the Senior Analysts, AI Evaluation.
• Determines the quality standard for any high‑value AI initiative at Geisinger – internally built or vendor‑provided – from design through production.
• Holds bought systems to the same standard as built ones, generating local evidence on whether a tool works here for Geisinger’s clinicians and patients rather than accepting vendor aggregate or cherry‑picked results.
• Owns the methodology that holds initiatives to that standard: pre‑production validation and live production monitoring.
• Owns the health of the data science team – attracting and retaining strong technical talent, developing careers, and keeping the bench deep, engaged, and growing – and leads and develops the evaluation team alongside it.
• Provides hands‑on technical guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks.
• Owns the evaluation toolkit and reusable playbooks and templates that let each new program move faster than the last.
• Translates program‑specific failure modes into concrete, measurable production‑monitoring metrics; defines what is measured and how, while the AI Platform team builds the backend.
• Tracks AI System Performance – the single most important accuracy indicator for each system, against thresholds set to clinical tolerance.
• Tracks User Adoption – engagement, override rates, and time‑to‑action – distinguishing genuine workflow misalignment and alarm fatigue from poor predictive value.
• Connects each AI to the Outcome it was deployed to improve (mortality, time‑to‑treatment, boarding time, denial rate, cost per case) over a pre‑launch baseline and a use‑case‑appropriate horizon, holding both tangible returns and harder‑to‑quantify value in view.
• Monitors Equity – the maximum performance gap on the Pillar 1 metric across the subgroups that matter for the initiative, so disparate impact surfaces early.

Required Skills and Qualifications

• People‑leadership experience – managing, developing, and growing technical staff; building teams, not just leading projects.
• Strong foundation in experimental design and causal inference, with judgment about which method fits which situation.
• Hands‑on experience designing and running model evaluation studies in real production settings.
• Experience evaluating LLM or generative AI systems, or comparable experience with complex ML systems where ground truth is ambiguous or noisy.
• Proven ability to translate ambiguous failure modes into concrete, defensible evaluation designs and monitoring metrics.
• Strong fluency in Python and SQL; working comfort with modern ML tooling and cloud‑native data environments.
• Experience in evaluating fairness and equity in ML systems.
• Clear written communication – the role produces evaluation memos and specifications that non‑technical decision‑makers rely on.
• Healthcare, clinical, or regulated‑industry experience strongly preferred.

Education

Bachelor's Degree – Related Field of Study (Required)

Experience

Minimum of 8 years – Related work experience (Required). Minimum of 3 years – Managerial/Supervisory (Required).

Certification(s)

None required.

Benefits

We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. Additional benefits include a collaborative environment, opportunities for learning and career development, and a culture that values diversity and inclusion.

Equality, Diversity & Inclusion

We are an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. Women and minorities are encouraged to apply.