Generative AI Scientist

Role Description

Join a recently formed team focused on Model Risk and Responsible AI. The Generative AI Scientist - Risk will apply knowledge and experience to real world problems and seek to utilize their skills to reduce the cost of healthcare and improve health quality and outcomes. As a Data Scientist on this team, you will focus on three main project areas: Model Validation, Model Metrics and Monitoring, and Responsible AI.

• Conduct independent model validation of existing models for benchmarking, assessment, and gauging effectiveness.
• Determine aspects of model drift and related data drift for the purpose of model risk management (MRM) to reduce risk and find opportunities for new revenue growth and innovation.
• Apply deep expertise with AI/ML/GenAI model development, including hands-on experience with model building and evaluation.
• Benchmark and potentially rebuild existing models using updated data and newer algorithms.
• Actively drive improvements in model monitoring activities, including methods for model registration and metadata management.
• Complete all responsibilities as outlined in the annual performance review and/or goal setting.
• Complete all special projects and other duties as assigned.
• Must be able to perform duties with or without reasonable accommodation.

Qualifications

• Graduate Degree in a quantitative discipline such as Computer Science/Engineering, Statistics, Operations Research covering Advanced Statistics, Machine learning and AI.
• Experience with the latest techniques in natural language processing including transformers, fine-tuning LLMs, measuring/benchmarking and deploying LLMs with tools such as HuggingFace, Langchain, LLAMA/Mistral, and OpenAI, vector databases.
• 1+ years of hands-on data science/AI experience, using typical machine learning and data science tools including pandas, scikit-learn, keras, nltk, and TensorFlow/PyTorch, GPU.
• Experience building production-grade machine learning deployments on AWS, Azure, or GCP.
• Experience working with Apache Spark™ and large-scale distributed datasets.
• Experience communicating technical concepts to non-technical and technical audiences is a plus.
• Passion for collaboration, learn it all mindset and driving value with AI.

Requirements

• Familiarity with healthcare payor ecosystem and related data.
• General understanding of Responsible AI (RAI), including explainability (XAI), AI NIST RMF, and related AI risk management frameworks.
• Experience evaluating models for bias and fairness, with aptitude for detecting bias in model design and data, using metrics such as SHAP and LIME.
• Understanding appropriate model metrics and techniques for managing, evaluating and monitoring GenAI models and LLMs.
• Understanding and familiarity with model governance and data governance best practices.
• Strong understanding of technology systems for model development (e.g., Python, DataRobot, AWS Sagemaker), model deployments (AWS, Azure, DataRobot, DataBricks), model monitoring (AWS Model Monitor, MLFlow, NannyML, FiddlerAI, Arize) and related tools for model management and metadata management.

Benefits

• Base compensation ranges from $110,000 to $130,000 per year.
• Competitive benefits package including medical, dental, vision, disability, and life insurance coverage.
• 401(k) savings plans.
• Paid family leave.
• 9 paid holidays per year.
• 17-27 days of Paid Time Off (PTO) per year, depending on specific level and length of service with Cotiviti.