Senior AI Governance Scientist

Role Description

Leads advanced technical evaluation and assurance activities within our AI Governance function. This role is hands-on and execution-focused, responsible for designing, conducting, and scaling rigorous evaluations of AI systems—including traditional machine learning, generative AI, and agentic AI—to assess safety, reliability, robustness, and alignment with intended use.

• Plays a critical role in operationalizing AI governance through experimentation, red teaming, and evaluation frameworks.
• Partners closely with engineering, product, and research teams to embed evaluation practices into the AI development lifecycle.
• Executes comprehensive red team and stress-testing exercises to identify vulnerabilities, failure modes, and safety risks across AI systems.
• Designs, implements, and refines evaluation methodologies and protocols to assess AI performance, safety, reliability, and alignment with intended use cases.
• Evaluates the adequacy and sufficiency of existing AI evaluations, identifies gaps in coverage or rigor, and recommends targeted improvements.
• Designs and conducts reproducible experiments to measure AI value, impact, and risk, applying statistical methods and causal inference techniques where appropriate.
• Develops and maintains automated testing frameworks and evaluation pipelines that scale across the organization’s AI portfolio.
• Researches and applies novel attack vectors and stress-testing approaches for generative AI and agentic systems.
• Creates and curates benchmarks, datasets, and metrics aligned to specific AI capabilities, risk profiles, and governance requirements.
• Documents evaluation methodologies, findings, and recommendations in clear, governance-ready technical reports for review by governance bodies and cross-functional stakeholders.
• Partners with product, engineering, and research teams to integrate evaluation and assurance practices into AI design, development, and deployment workflows.
• Performs other duties as assigned.
• Complies with all policies and standards.

Qualifications

• Bachelor's Degree in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent applied research experience required. Master's Degree preferred.
• 5+ years AI/ML research or applied AI development, including at least 3 years focused on model evaluation, safety, robustness or validation required. 7+ years preferred.

Requirements

• Strong technical foundation in machine learning and deep learning, with hands-on experience evaluating or developing modern AI systems required.
• Demonstrated experience designing and executing AI evaluation, testing, or validation methodologies across multiple AI paradigms required.
• Solid understanding of statistical analysis, experimental design, and data analysis techniques relevant to AI evaluation required.
• Experience designing evaluation methodologies and publication record preferred.
• Familiarity with Python‑based AI/ML stack using PyTorch and Databricks, with agentic AI frameworks for single‑ and multi‑agent systems.
• Strong focus on LLM observability, MLOps, and evaluation using various testing frameworks.

Benefits

• Competitive pay
• Health insurance
• 401K and stock purchase plans
• Tuition reimbursement
• Paid time off plus holidays
• Flexible approach to work with remote, hybrid, field or office work schedules