Senior AI Data Scientist

When you join the growing BILH team, you're not just taking a job, you're making a difference in people's lives.

The Senior AI Data Scientist role serves as a technical leader for complex machine learning and generative AI initiatives within the organization. This role is responsible for designing, fine-tuning, evaluating, and governing large and small language models (LLMs) and advanced machine learning models used in healthcare and operational settings. The Senior AI Data Scientist owns model quality, evaluation rigor, and responsible AI outcomes, while mentoring junior team members and guiding project-level technical direction. The position emphasizes model development, experimentation strategy, evaluation frameworks, and applied generative AI. The Senior AI Data Scientist partners closely with engineering, clinical, and data teams to ensure AI solutions are accurate, safe, explainable, and fit for production use.

Job Description:

Essential Duties & Responsibilities including but not limited to:


1. Lead the design, development, and evaluation of advanced machine learning and generative AI models across healthcare use cases.


2. Serve as a subject matter expert on large and small language models (LLMs), including architecture tradeoffs, inference behavior, strengths, and limitations.


3. Design and oversee LLM fine-tuning strategies, including instruction tuning and parameter efficient fine-tuning approaches, ensuring alignment with domain needs and model governance standards.


4. Define and execute model evaluation frameworks for LLMs, including quantitative metrics, qualitative review, bias analysis, hallucination assessment, and safety testing.


5. Guide and review prompt design and optimization strategies, ensuring robustness, consistency, and interpretability of generative AI outputs.


6. Lead and improve retrieval of augmented generation (RAG) solutions, including data selection, grounding quality, relevance evaluation, and retrieval performance analysis.


7. Conduct advanced exploratory data analysis, feature engineering, and statistical modeling to support predictive and prescriptive analytics.


8. Translate complex clinical or business problems into clear modeling strategies and evaluation plans.


9. Mentor entry level and midlevel data scientists through model reviews, best practice guidance, and collaborative problem solving.


10. Partner with data engineering, software engineering, and clinical teams to ensure AI models are deployable, monitored, and compliant with governance requirements.


11. Define and promote Responsible AI best practices, including fairness, bias mitigation, transparency, documentation, and appropriate clinical use.


12. Communicate model behavior, tradeoffs, and outcomes to technical leaders, clinicians, and executive stakeholders.


Minimum Qualifications:


Education:

Bachelor's degree in Data Science, Computer Science, Artificial Intelligence, Statistics, Mathematics, or related quantitative discipline required.


Master's degree preferred.


Licensure, Certification & Registration:

None required.


Experience:

5+ years of experience in data science, machine learning, or applied AI roles.


Demonstrated experience leading model development and evaluation efforts, including generative AI or NLP-based solutions.


Experience mentoring or guiding junior data scientists.


Skills, Knowledge & Abilities:

Deep understanding of machine learning and generative AI fundamentals.


Handson experience working with LLMs, including finetuning concepts and inference tradeoffs.


Strong expertise in model evaluation, including both traditional ML metrics and qualitative LLM assessment.


Advanced proficiency using Python for data analysis, modeling, and experimentation.


Strong analytical judgment and ability to assess model risks and limitations.


Excellent written and verbal communication skills, especially for explaining complex AI behavior.


Ability to operate independently on complex problems while providing technical leadership to others.


Preferred Qualifications & Skills:

Advanced experience with LLM evaluation frameworks, test harnesses, or human-in-the-loop review.


Experience designing or governing RAG architectures.


Familiarity with healthcare data, clinical terminology, or regulated data environments.


Knowledge of AI ethics, fairness, bias mitigation, and responsible AI governance.


Experience contributing to AI standards, documentation, or best practice frameworks.


Exposure to production monitoring or model lifecycle management in regulated environments.


Dept./Unit-Specific Skills:

Key Business Relationships: (Title and Purpose)


1 AI Data Scientists


Technical leadership, mentoring, and model review


2 Data Engineering


Data availability, quality, and pipeline alignment


3 Clinical Stakeholders


Validation of model relevance and safety


4 AI Governance / Compliance


Responsible AI alignment and oversight


Pay Range:
$120,000.00 USD - $165,000.00 USD

The pay range listed for this position is the annual base salary range the organization reasonably and in good faith expects to pay for this position at this time. Actual compensation is determined based on several factors, that may include seniority, education, training, relevant experience, relevant certifications, geography of work location, job responsibilities, or other applicable factors permissible by law.


As a health care organization, we have a responsibility to do everything in our power to care for and protect our patients, our colleagues and our communities. Beth Israel Lahey Health requires that all staff be vaccinated against influenza (flu) as a condition of employment.

More than 35,000 people working together. Nurses, doctors, technicians, therapists, researchers, teachers and more, making a difference in patients' lives. Your skill and compassion can make us even stronger.

Equal Opportunity Employer/Veterans/Disabled