Senior AI Scientist

About the RoleThis role sits at the intersection of AI safety research, LLM evaluation, and applied data science. You will design and run adversarial evaluations against production AI agents, develop quantitative frameworks to measure evaluation quality, and translate findings into actionable insights for application teams and business stakeholders. You will also mentor junior data scientists and contribute to the growth of our AI safety research practice.ResponsibilitiesDesign and execute adversarial evaluations of generative AI and agentic systems, probing for safety failures, policy violations, and unexpected behaviors under realistic conditionsApply rigorous experimental design and statistical methodology to AI safety research questions, including evaluation of model robustness and the reliability of automated evaluation systemsDevelop and validate quantitative frameworks for assessing LLM outputs and measuring evaluation quality at scaleBuild and maintain data pipelines to process large-scale evaluation outputs, aggregate metrics, and surface trends for research and stakeholder consumptionEngage directly with internal stakeholders including application teams, security analysts, and business leaders to explain findings and translate complex analytical results into clear, actionable recommendationsMentor and develop junior data scientists and analysts on the teamStay current with emerging literature in adversarial ML, LLM safety, and AI evaluation; contribute to the team's evolving research agendaParticipate in special projects and perform other duties as assignedQualificationsMinimum 5 years of experience in data science, applied ML research, or AI evaluation rolesHands-on experience with large language models and agentic AI systems, with working knowledge of LLM behavior, failure modes, and safety evaluation techniquesFamiliarity with adversarial AI concepts including jailbreaks, prompt injection, and model robustness, whether through direct research or applied workStrong programming skills in Python, including experience with data pipelines, large-scale experimentation, and ML libraries such as PyTorch, Hugging Face, or Scikit-learnSolid foundation in statistical reasoning and experimental design: hypothesis formulation, significance testing, and the ability to identify confounds and methodological flaws in existing analysesExperience in financial services, AI safety, trust and safety, or a regulated industry is a strong plusClear, confident communication skills with the ability to explain technical findings to both research peers and non-technical stakeholdersBachelor's degree in Computer Science, Statistics, Applied Mathematics, or a related quantitative field; Master's degree or equivalent research experience preferredSpecial FactorsSponsorshipVanguard is not offering visa sponsorship for this position.About VanguardAt Vanguard, we don't just have a mission—we're on a mission.To work for the long-term financial wellbeing of our clients. To lead through product and services that transform our clients' lives. To learn and develop our skills as individuals and as a team. From Malvern to Melbourne, our mission drives us forward and inspires us to be our best.How We WorkVanguard has implemented a hybrid working model for the majority of our crew members, designed to capture the benefits of enhanced flexibility while enabling in-person learning, collaboration, and connection. We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience.SummaryLocation: Malvern, PA; Charlotte, NC; Dallas/Ft. Worth, TX; Philadelphia, PAType: Full time