Senior Machine Learning Engineer

Evaluation Engineer

Evaluation is the bottleneck in healthcare AI — you can't ship what you can't measure. You'll build the systems that tell us whether our models are safe, accurate, and ready for real patients: evaluation frameworks, synthetic data pipelines, automated benchmarks, and LLM-as-judge systems. This is a high-leverage engineering role where your work directly gates what goes to production.

What You'll Do

• Design and build evaluation frameworks for LLM safety, clinical accuracy, and conversational quality
• Develop synthetic data generation pipelines to stress-test models across diverse clinical scenarios
• Build automated and human-in-the-loop evaluation pipelines at scale
• Create benchmarks, metrics, and LLM-as-judge systems for healthcare tasks and conversational experience
• Analyze failure modes and translate findings into actionable model improvements by collaborating with the LLM post-training team
• Collaborate with research, engineering, and clinical teams to define and raise the quality bar

What You Bring

Must-Have

• MS or PhD in CS or related field
• 5+ years in ML engineering, evaluation systems, or applied ML
• Strong software engineering skills — Python, PyTorch, and production-quality code
• Hands-on experience with LLM evaluation, benchmarking, or synthetic data generation
• Comfort building robust data analysis and evaluation infrastructure, not just running experiments
• Experience with UI/UX and front-end development toolkits such as Streamlit, Gradio, React, etc.

Nice-to-Have

• Experience in healthcare AI
• Experience with RL/RLVR/RLHF or safety evaluation

Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @ hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.