AI VALIDATION ANALYST

AI Validation Analyst

Working at Moffitt is both a career and a mission: to contribute to the prevention and cure of cancer. As the only National Cancer Institute-designated Comprehensive Cancer Center based in Florida, Moffitt employs some of the best and brightest minds from around the world. Join a dedicated team of nearly 10,000 who are shaping the future we envision. Moffitt has been recognized as a Best and Brightest Company to Work for in the Nation, a Digital Health Most Wired Organization and continually named one of the Tampa Bay Time's Top Workplaces. A National Cancer Institute (NCI)-designated Comprehensive Cancer Center since 2001.

Position Highlights

The AI Validation Analyst is the technical specialist on the AI Governance Operations team, responsible for executing local validation testing, managing continuous performance monitoring, and coordinating incident response for AI tools across Moffitt's portfolio. This role bridges the gap between clinical/operational end users and the technical realities of AI performance — translating model outputs, drift metrics, and validation results into governance-actionable findings. The Analyst works closely with faculty validation consultants, IT/platform teams, clinical departments, and vendor technical contacts.

Responsibilities

Local Validation Testing and Analysis

• Design and execute validation test plans for AI tools deployed or proposed for deployment at Moffitt, with emphasis on Tier 1 (high-risk) clinical AI.
• Conduct local performance validation using Moffitt data, including sensitivity/specificity analysis, subgroup performance evaluation (demographics, disease subtypes), and failure mode analysis
• Coordinate with faculty validation consultants (pathology, radiology, data science) to execute domain specific validation protocols.
• Evaluate vendor-provided validation evidence for sufficiency, including assessment of training data representativeness, external validation studies, and applicability to Moffitt's oncology patient population.
• Produce validation reports documenting methodology, findings, residual risks, and recommendations for AIGC review.

Continuous Monitoring

• Develop and manage the continuous monitoring program for deployed AI tools, including definition of performance metrics, drift detection thresholds, and alert triggers appropriate to each risk tier.
• Monitor AI tool performance for accuracy degradation, data drift, concept drift, and bias emergence using vendor-provided dashboards, internal analytics, and manual audit sampling.
• Execute scheduled review cycles aligned to risk tier (Tier 1: 6- month, Tier 2: 12-month, Tier 3: 24 month) and produce monitoring reports for AIGC.
• Identify when performance degradation or environmental changes warrant escalation, revalidation, or tool retirement.

Escalation and Incident Response

• Serve as the first responder for AI-related adverse events (aiAE), near-misses, and performance incidents.
• Coordinate incident investigation, root cause analysis, and remediation in partnership with clinical quality, patient safety, IT, and vendor contacts.
• Maintain an aiAE log integrated with Moffitt's existing quality and safety reporting systems
• Support the Director in executing emergency pause decisions when AI tools present immediate risk.

Risk Evaluation Support

• Provide technical input to the 7-question risk scoring process, particularly for questions related to clinical decision influence, human override capability, and scope of harm.
• Contribute to Tier 1 parallel specialist review tracks, specifically the Clinical Safety Track (validation requirements, failure modes, human oversight design).
• Score the residual risk factors owned by the Clinical Safety specialist track (Nature of Impact, Scale of Impact, Reversibility, Likelihood of Harm, Autonomy Level) as part of the Integrated Residual Risk Profile for Tier 1 and Tier 2 reviews; provide technical input to Data & Privacy track on Transparency factor assessments.
• Assess AI tools for equity and bias risks, particularly in oncology specific populations.

Technical Documentation and Knowledge

• Maintain technical documentation standards for validation protocols, monitoring configurations, and incident response procedures.
• Prepare and maintain "AI Governance Model Cards" for all the AI tools deployed inside Moffitt.
• Stay current on AI/ML validation methodologies, healthcare AI safety research, and regulatory guidance (FDA AI/ML SaMD action plan, CHAI/Joint Commission recommendations).
• Contribute to the development of Moffitt-specific validation standards for common AI tool categories (clinical decision support, imaging AI, NLP, ambient documentation).

Credential and Qualifications

• Bachelor's Degree in data science, biostatistics, biomedical informatics, computer science, health informatics, clinical research, or related field required.
• 3+ years of experience in data analysis, biostatistics, clinical research analytics, AI/ML model evaluation, health informatics, or a related technical role.
• Hands-on experience with quantitative analysis — statistical testing, performance metric calculation (sensitivity, specificity, AUC, PPV/NPV), or equivalent analytical methods.
• Experience working with healthcare data (EHR data, clinical datasets, imaging data, or claims data)
• Experience producing technical reports or analyses for non-technical audiences.

Preferred Experience

• Experience in an NCI-designated cancer center or academic medical center.
• Experience validating clinical AI/ML tools (e.g., clinical decision support, diagnostic imaging AI, NLP for clinical text).
• Experience with model monitoring tools or platforms (e.g., Fiddler AI, Arthur AI, Evidently, vendor-specific monitoring dashboards).
• Experience with FDA SaMD regulatory processes or clinical trial data analysis.
• Experience with bias and equity auditing in AI systems.
• Familiarity with oncology-specific data, workflows, and patient populations.