Machine Learning Evaluation Specialist

Machine Learning Evaluation Specialist

What if your years of research and domain expertise could directly shape how the next generation of AI models think and reason? That's exactly what this role offers.

We're looking for ML experts and domain specialists to design fiendishly difficult evaluation problems that push state-of-the-art AI systems to their limits. This isn't about textbook prompts — it's about crafting the kind of challenges that only someone with deep, hard-earned expertise can build and judge.

Your work will directly influence how leading AI models are measured, improved, and deployed.

• Organization: Alignerr
• Type: Hourly Contract
• Location: Remote
• Commitment: 10–40 hours/week

What You'll Do

• Design complex, original machine learning problems rooted in your specialized domain of expertise
• Craft evaluation tasks that go beyond standard ML pipelines and require genuine advanced knowledge to solve
• Draw from your own research experience to build challenges that would stump a highly capable AI
• Write clear, precise problem statements with defined evaluation criteria and gold-standard solutions
• Assess AI-generated ML solutions for correctness, creativity, and methodological rigor
• Document problem difficulty, required domain knowledge, and expected failure modes
• Collaborate asynchronously with a global team of researchers and engineers

Who You Are

• You hold graduate-level expertise (MS or PhD preferred) in a scientific or technical domain intersecting with machine learning
• You have strong working knowledge of core ML methods — model selection, feature engineering, evaluation metrics, and pipeline design
• You're deeply familiar with active, open research problems in your field
• You can pinpoint exactly where general ML knowledge falls short and specialized domain insight becomes essential
• You have experience conducting or publishing original research (highly valued)
• You communicate complex ideas clearly and precisely in writing
• You're self-motivated and thrive when working independently on intellectually demanding problems

Example Domains (Not Exhaustive)

We're actively seeking specialists across a wide range of fields, including:

• Computational biology, genomics, or bioinformatics
• Climate science and environmental modeling
• Medical imaging and healthcare ML
• Materials science and computational chemistry
• Astrophysics and signal processing
• NLP for low-resource or specialized corpora
• Robotics, control theory, or reinforcement learning
• Financial modeling and quantitative analysis

If your domain isn't listed, we still want to hear from you — if your expertise is deep and niche, it's valuable.

Why Join Us

• Work at the frontier — contribute directly to AI evaluation and safety research that matters
• Make your expertise count — finally, a role that rewards deep domain knowledge, not just generalist skills
• Collaborate globally — work asynchronously alongside top researchers and engineers around the world
• Full autonomy — flexible schedule, no micromanagement, work when and how you do your best thinking
• Ongoing opportunity — strong potential for contract extension and deeper research involvement
• Build your profile — be recognized as a contributor to cutting-edge AI development

Apply now