Tech Lead Data Scientist, AI Evaluation & Monitoring

Job Summary The Tech Lead Data Scientist, AI Evaluation & Monitoring is the principal technical expert responsible for evaluating, monitoring, and optimizing AI systems in production at Geisinger. This hands‑on technical leadership role sets technical direction for AI evaluation across a growing portfolio, leads a team of data analysts, and partners with AI program teams to improve how AI is validated, monitored, and improved over time. Responsibilities Own the technical evaluation methodology across the enterprise, including pre‑production validation, production monitoring, and ongoing optimization. Guide program teams in designing validation studies, equity audits, monitoring plans, and escalation playbooks for their AI systems. Translate program‑specific failure modes into concrete, measurable metrics and instrument production monitoring. Develop and maintain evaluation toolkits such as LLM‑as‑Judge frameworks, golden sets, simulation harnesses, experimental study designs, drift detection, and subgroup fairness analysis. Create reusable evaluation playbooks and templates that accelerate new program onboarding. Provide technical direction, design review, and mentorship for a team of data analysts. Coach teams to produce rigorous evaluation deliverables for governance review. What You Will Not Own People management or formal performance evaluations for the analyst team. Program‑level product strategy or go/no‑go decisions. Final clinical validation judgment for AI safety. Software infrastructure behind evaluation tooling (handled by the AI Platform team). Required Skills & Qualifications 6+ years in data science, statistics, ML engineering, or applied quantitative research, with senior technical experience on cross‑functional projects. Strong foundation in experimental design and causal inference, with judgment on method selection. Hands‑on experience designing and running model evaluation studies in real production settings. Experience evaluating LLM or generative AI systems, or comparable complex ML systems with imperfect ground truth. Demonstrated ability to translate ambiguous failure modes into concrete evaluation designs and monitoring metrics. Fluency in Python and SQL; comfortable with modern ML tooling and cloud‑native data environments. Experience with fairness and equity evaluation for ML systems. Track record of providing technical leadership and mentorship without formal people‑management authority. Clear written communication skills for producing evaluation memos and specifications relied upon by non‑technical decision makers. Clinical, healthcare, or regulated‑industry experience strongly preferred. Preferred Qualifications MS or PhD in a quantitative field, or equivalent experience. Education: Bachelor’s Degree in a related field (required). Benefits Healthcare benefits for full‑time and part‑time employees from day one, including vision, dental, and domestic partners. Equal Opportunity Employer Geisinger is an affirmative action, equal‑opportunity employer and all qualified applicants will receive consideration for employment regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran. #J-18808-Ljbffr