Associate AI Evaluation Scientist

Role Description

The Associate AI Evaluation Scientist is an individual contributor role focused on contributing to the data science execution of AI evaluations under the guidance of senior scientists and engineers. This role supports the development and application of evaluation methods, metrics, and datasets that inform model quality, product performance, and AI risk management.

In this role, you will work hands‑on with machine learning models—particularly LLM‑based systems—helping to run evaluations, analyze results, and maintain evaluation workflows. You will gain experience translating evaluation standards into reusable code, collaborating with engineering partners, and communicating findings to technical stakeholders. This position is well suited for an early‑career data scientist who is excited to deepen their skills in applied ML evaluation, experimentation, and responsible AI practices.

Key Responsibilities

• Implement and run established evaluation methods for LLMs and ML systems, including performance, robustness, fairness, and safety‑related metrics.
• Contribute to the creation, curation, and maintenance of evaluation datasets such as golden sets, challenge sets, and test corpora aligned with banking use cases.
• Support the development and maintenance of evaluation pipelines, harnesses, and dashboards used in experimentation and model assessment.
• Perform empirical analyses such as error analysis, metric comparisons, and basic statistical testing to surface model behavior and performance issues.
• Assist with integrating evaluation results into experimentation workflows, model reviews, and deployment readiness checks.
• Document evaluation methodologies, assumptions, and results to support reproducibility and team knowledge sharing.
• Collaborate with senior scientists, engineers, and product partners to understand evaluation requirements and apply best practices.
• Learn and apply responsible AI principles, including reproducibility, transparency, and risk‑aware evaluation.

Qualifications

• 1–3 years of experience in data science, machine learning, analytics, or a related field, including internships, co‑ops, or substantial academic projects.
• Bachelor’s or Master’s degree in Computer Science, Data Science, Statistics, Engineering, or a related quantitative discipline.
• Proficiency in Python and familiarity with data science libraries (e.g., pandas, numpy, scikit‑learn); exposure to PyTorch or TensorFlow is a plus.
• Foundational understanding of statistics, model evaluation metrics, and experimental analysis.
• Experience working with structured or unstructured datasets and writing clear, maintainable analysis code.
• Interest in AI evaluation, model quality, safety, or responsible AI, with a desire to grow deeper technical expertise in this area.
• Strong written and verbal communication skills, with the ability to explain analysis results to technical teammates.
• Familiarity with LLMs, NLP, or ML experimentation workflows is an asset but not required.

Benefits

• Health insurance
• Tuition reimbursement
• Accident and life insurance
• Retirement savings plans

Company Description

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.

As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset.