AI/ML Engineer - Validation & Evaluation

City Rochester

State MN

Remote NO

Department Information Technology

Why Mayo Clinic

Mayo Clinic is top-ranked in more specialties than any other care provider according to U.S. News & World Report. As we work together to put the needs of the patient first, we are also dedicated to our employees, investing in competitive compensation and comprehensive benefit plans – to take care of you and your family, now and in the future. And with continuing education and advancement opportunities at every turn, you can build a long, successful career with Mayo Clinic.

Benefits Highlights

• Medical: Multiple plan options.
• Dental: Delta Dental or reimbursement account for flexible coverage.
• Vision: Affordable plan with national network.
• Pre-Tax Savings: HSA and FSAs for eligible expenses.
• Retirement: Competitive retirement package to secure your future.

Responsibilities

AI/ML Engineers at AI Validation & Monitoring (AVM) apply data, systems, computer science, clinical workflow, and governance expertise to help ensure that clinical AI products are supported by proportionate, traceable, and technically sound validation evidence. They work with AIA Governance Operations, clinicians, product teams, human-factors and user-experience specialists, data scientists, IT, architecture, patient safety, legal and regulatory partners, vendors, and other stakeholders to translate AIA Governance Policy and approved validation methods into clear evidence expectations and review conclusions.

As the AI/ML Engineer - Validation & Evaluation, with the functional assignment of Validation Guidance, Guardrails, and Test Evidence, you will perform routine and moderately complex subject-matter review of the Validation, Performance, and Safety content and supporting evidence prepared. You will apply approved rubrics, evidence-sufficiency criteria, checklists, templates, and standard methods; identify missing evidence, unsupported conclusions, methodological limitations, and inconsistencies; document corrections, clarification questions, limitations, and escalation triggers; and draft traceable AVM concurrence or consultation comments. High-risk, novel, disputed, exceptional, or out-of-method questions are escalated to the Senior or Principal AI/ML Engineer comparisons, residual risk, and whether assessment conclusions are supported by the available evidence.

• Reviewing validation strategy and pathway selection, including retrospective analysis, prospective testing, simulation, user acceptance testing, human-factors evaluation, pilots, and other risk-proportionate methods, for alignment with intended use, workflow, and patient-safety risk.
• Evaluating performance expectations, metrics, thresholds, acceptance criteria, standard-of-practice 
• Reviewing test plans and validation datasets for representativeness, traceability, limitations, production parity, functionality, robustness, calibration, uncertainty, bias, subgroup and equity evidence, and the adequacy of retesting evidence.
• Reviewing risk and safety controls, guardrails, task boundaries, safe refusal, escalation, fallback, human oversight, training, and other mitigations needed for the intended clinical or operational use.
• Evaluating human-computer interaction, clinical workflow, usability, automation-bias risk, patient-facing behavior, accessibility, and human-use evidence, and identifying when additional evaluation, clarification, or remediation is required.
• Assessing whether prior validation remains applicable after changes to the model, data, thresholds, workflow, population, interface, autonomy, guardrails, training, or intended use, and documenting needs for targeted retesting, re-evaluation, or revalidation.
• Producing clear, traceable review findings and recommended outcomes, including concurrence, concurrence with edits, revision required, additional evidence, consultation, escalation, or insufficient basis, and communicating technical findings to Product Leads and other technical and non-technical stakeholders.
• Maintaining and applying test-plan, validation-dataset, guardrail, subgroup, and production-parity templates; policy-to-evidence crosswalks; evidence examples; standard finding and clarification language; and recurring-gap themes that improve guidance, training, tools, and Product Lead self-sufficiency. 

This vacancy is not eligible for sponsorship/ we will not sponsor or transfer visas for this position. Also, Mayo Clinic DOES NOT participate in the F-1 STEM OPT extension program.

Qualifications

• A master’s degree in engineering, computer science, mathematics, health science, or a related field and 1 year experience, or a bachelor’s degree with 3 years of experience. 
• Experience applying AI and machine learning in production environments or similar highly regulated or technology focused industries, showcasing an understanding of healthcare technology.
• Skill in cloud infrastructure environment and software development tools.
• Experience working with large, complex, and heterogeneous data sets, preferably in healthcare. 
• Skill in AI/ML techniques and frameworks.
• History of collaborating across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
• Familiarity with best practices in data engineering, data science, AI Engineering, and the MLOps communities.
• Strong interpersonal, communication, and time management skills.
  Preferred Qualifications:
• A Ph.D. or other doctorate degree is preferred. 
• Knowledge of the healthcare domain, including clinical workflows, electronic health records, medical terminologies, regulatory requirements, and industry standards.
• Familiarity with systems or quality engineering best practices, regulatory standards, and compliance frameworks, with the ability to adapt these effectively to different project scenarios.
• Expertise in user-centered design, human factors engineering, usability testing methodologies, and evaluation across AI product development. Ability to conduct expert reviews using established usability practices and methods. Presents findings in easy-to-understand terms for the business or clinical practice.
• Ability to articulate complex technical concepts to diverse audiences, facilitating clear understanding and engagement from technical and non-technical stakeholders.
• Ability to manage a varied workload of projects with multiple priorities and stay current on healthcare trends.
• Experience with healthcare industry informatics standards, best practices, and common data models.
• Demonstrated hands-on experience conducting clinical AI validation and evaluation assessments using with TRex for products deployed into clinical environments such as EPIC or ANIMATE, including test-plan review, production-parity assessment, guardrail evaluation, human-factors evidence, and revalidation decisions. 
  

Exemption Status

Exempt

Compensation Detail

$116,043.20 - $168,292.80/ year. Education, experience and tenure may be considered along with internal equity when job offers are extended.

Benefits Eligible

Yes

Schedule

Full Time

Hours/Pay Period

80

Schedule Details

M-F 8am-5pm This is a hybrid position and must be located within 100 miles of a Mayo Clinic campus for occasional on-site expectations based on business needs.

Weekend Schedule

NA

International Assignment

No

Site Description

Just as our reputation has spread beyond our Minnesota roots, so have our locations. Today, our employees are located at our three major campuses in Phoenix/Scottsdale, Arizona, Jacksonville, Florida, Rochester, Minnesota, and at Mayo Clinic Health System campuses throughout Midwestern communities, and at our international locations. Each Mayo Clinic location is a special place where our employees thrive in both their work and personal lives. Learn more about what each unique Mayo Clinic campus has to offer, and where your best fit is.

Equal Opportunity

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status or disability status. Learn more about the "EOE is the Law" . Mayo Clinic participates in E-Verify and may provide the Social Security Administration and, if necessary, the Department of Homeland Security with information from each new employee's Form I-9 to confirm work authorization.

Recruiter

Laura Percival