Principal AI/ML Engineer - Validation & Evaluation Governance

City Rochester

State MN

Remote NO

Department Information Technology

Why Mayo Clinic

Mayo Clinic is top-ranked in more specialties than any other care provider according to U.S. News & World Report. As we work together to put the needs of the patient first, we are also dedicated to our employees, investing in competitive compensation and comprehensive benefit plans – to take care of you and your family, now and in the future. And with continuing education and advancement opportunities at every turn, you can build a long, successful career with Mayo Clinic.

Benefits Highlights

• Medical: Multiple plan options.
• Dental: Delta Dental or reimbursement account for flexible coverage.
• Vision: Affordable plan with national network.
• Pre-Tax Savings: HSA and FSAs for eligible expenses.
• Retirement: Competitive retirement package to secure your future.

Responsibilities

As the Principal AI/ML Engineer - Validation & Evaluation Governance within AI Validation & Monitoring (AVM), you will serve as the enterprise subject-matter authority for validation and evaluation requirements, evidence sufficiency, and complex-case consultation. You will define and interpret standards aligned with AIA Governance Policy and patient-safety risk categories; establish retrospective and prospective pathways; set expectations for performance, safety, human-computer interaction (HCI), workflow, guardrails, production parity, material change, and revalidation; and authoritatively review the Validation, Performance, Safety, HCI, workflow, and revalidation content and evidence for methodological adequacy, traceability, limitations, and decision readiness.

You will provide final AVM technical direction and precedent for high-risk, novel, disputed, patient-facing, agentic, vendor-limited, retrospective-method, and other precedent-setting questions. You will direct corrections, alternate methods, added evidence, interim controls, limitations, and escalation; convene cross-functional expertise; and convert recurring gaps into reusable standards and enablement.

• Providing strategic and technical leadership for enterprise validation and evaluation governance and evidence standards.
• Defining risk-proportionate requirements for pilots, full implementations, changes, legacy products, and other pathways.
• Reviewing intended use, validation strategy, performance expectations, thresholds, acceptance criteria, and residual risk.
• Evaluating test plans and datasets for representativeness, traceability, limitations, production parity, functionality, robustness, calibration, subgroup and equity evidence, and uncertainty.
• Reviewing safety controls, guardrails, task boundaries, safe refusal, escalation, fallback, human oversight, and related mitigations.
• Reviewing HCI, clinical workflow, usability, automation-bias risk, patient-facing behavior, training, accessibility, and human-use evidence.
• Determining when retrospective evidence is sufficient and when prospective testing, simulation, UAT, human-factors work, a pilot, or another method is required.
• Establishing material-change and revalidation triggers, including strengthened early monitoring when retrospective evidence replaces a pilot.
• Issuing traceable review outcomes: concurrence, edits, revision, additional evidence, consultation, escalation, or insufficient basis.
• Leading consultation and setting precedent for high-risk, novel, disputed, vendor-constrained, patient-facing, autonomous, agentic, or out-of-method cases.
• Coordinating with clinical product teams and enterprise partners, including Research Shield, Kern Center, AVSP/CCaTS, MCP Evaluate and Deploy, FAST, Epic, DTO, IT, Architecture, data platforms, Patient Safety, Clinical Informatics, Legal, and relevant committees.
• Developing policies, standards, pathway guidance, playbooks, rubrics, decision aids, evidence examples, standard findings, case libraries, and escalation criteria.
• Leading training, calibration, office hours, case consultation, quality assurance, and coaching for AVM staff and Governance Operations Product Leads.
• Translating recurring gaps into improved guidance, vendor expectations, data needs, TRex workflows, evidence objects, dashboards, and technology priorities.
• Communicating findings, limitations, uncertainty, and recommendations clearly to clinical, technical, operational, governance, and executive audiences.
• Collaborating with post-deployment governance so monitoring, reporting, significant-change, and PDRS evidence can trigger retesting, workflow evaluation, or revalidation.
• Setting precedent without owning the complete assessment, routine case coordination, product-team gap resolution, study execution, monitoring operations, technology build/run, clinical judgment, or final approval.
• Provide mentorship, guidance, and technical leadership to junior engineers. May have supervisory responsibilities. 

Qualifications

• A master’s degree in engineering, computer science, mathematics, health science, or a related field with 7 years of relevant experience, or a bachelor’s degree with 9 years of relevant experience. 
• Extensive (7+ years) experience applying AI and machine learning in production healthcare environments or similar highly regulated or technology focused industries, showcasing an acute understanding of healthcare technology.
• Demonstrated leadership in managing complex projects, with a proven ability to navigate intricate project requirements and deliver successful outcome.
• Proven success in fostering collaboration across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
• Demonstrated expertise in cloud infrastructure environment and software development tools.
• Experience working with large, complex, and heterogeneous data sets, preferably in healthcare. 
• Strong skills in AI/ML techniques and frameworks.
• Expertise with best practices in data engineering, data science, AI Engineering, and the MLOps communities.
• In-depth knowledge of healthcare domain, including clinical workflows, electronic health records, medical terminologies, regulatory requirements, and industry standards.
• Demonstrated leadership in administration, education, software development, and technical reporting.
• Experience mentoring and training less-experienced team members, coupled with strong interpersonal, communication, and time management skills.

Preferred Qualifications:

• A Ph.D. or other doctorate is preferred. 
• Experience with healthcare industry informatics standards, best practices, and common data models. Participation in national or international standards organizations or other domain-specific professional organizations, or extensive implementation experience with common data, development, and deployment standards.
• Excellent communication, collaboration, and stakeholder management skills, with the ability to effectively engage with diverse stakeholders and translate complex technical concepts and results to non-technical audiences.
• Demonstrated experience leading technical/quantitative teams in a regulated environment.
• Familiarity with systems or quality engineering best practices, regulatory standards, and compliance frameworks, with the ability to adapt these effectively to different project scenarios.
• Demonstrated experience creating risk management files and verification/validation strategies for digital health technology products within the healthcare industry.
• Demonstrated expertise in user-centered design, human factors engineering, usability testing methodologies, and evaluation across AI product development. Ability to lead expert reviews using established usability practices and methods. Presents findings in easy-to-understand terms for the business or clinical practice.
• Strong problem-solving abilities, critical thinking skills, and a passion for driving innovation and positive change in healthcare through AI technology.
• Demonstrated hands-on leadership using the TRex assessment application to review AI tools deployed in Epic, ANIMATE, or comparable clinical environments, including evidence sufficiency, validation pathways, HCI, guardrails, production parity, material change, revalidation, and precedent-setting decisions. 
  

Exemption Status

Exempt

Compensation Detail

$163,280.00 - $236,745.60/ year. Education, experience and tenure may be considered along with internal equity when job offers are extended.

Benefits Eligible

Yes

Schedule

Full Time

Hours/Pay Period

80

Schedule Details

M-F 8am-5pm This is a hybrid position and must be located within 100 miles of a Mayo Clinic campus for occasional on-site expectations based on business needs.

Weekend Schedule

NA

International Assignment

No

Site Description

Just as our reputation has spread beyond our Minnesota roots, so have our locations. Today, our employees are located at our three major campuses in Phoenix/Scottsdale, Arizona, Jacksonville, Florida, Rochester, Minnesota, and at Mayo Clinic Health System campuses throughout Midwestern communities, and at our international locations. Each Mayo Clinic location is a special place where our employees thrive in both their work and personal lives. Learn more about what each unique Mayo Clinic campus has to offer, and where your best fit is.

Equal Opportunity

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status or disability status. Learn more about the "EOE is the Law" . Mayo Clinic participates in E-Verify and may provide the Social Security Administration and, if necessary, the Department of Homeland Security with information from each new employee's Form I-9 to confirm work authorization.

Recruiter

Laura Percival