AI Evaluation Researcher

Join a pioneering AI initiative focused on developing the next generation of evaluation benchmarks for frontier AI models.

We are seeking researchers from computational STEM disciplines—as well as computationally intensive social sciences and humanities—to bring the rigor of real-world research into AI evaluation.

In this role, you will transform scientific methodologies such as experimental design, hypothesis testing, and data-driven analysis into sophisticated, multi-step benchmark tasks that challenge state-of-the-art AI systems. Working closely with AI researchers, you'll help uncover subtle reasoning errors and methodological flaws that only experienced researchers can identify.

This is a fully remote, full-time engagement requiring approximately 35 hours per week .

Requirements

Key Responsibilities

• Design complex, research-oriented benchmark tasks inspired by real-world scientific workflows, including study design, experimentation, hypothesis testing, and data analysis.
• Develop comprehensive reference solutions using Python , notebooks, and computational tools with the rigor expected in professional research.
• Define clear evaluation standards that distinguish sound scientific reasoning from plausible but incorrect conclusions.
• Review AI-generated solutions, identifying methodological weaknesses, analytical errors, and flawed reasoning that experienced researchers would recognize immediately.
• Collaborate with AI researchers and fellow domain experts to improve benchmark quality, consistency, and scientific rigor.
• Contribute to the continuous refinement of evaluation methodologies for advanced AI systems.

Required Qualifications

• Master's degree, PhD, or equivalent practical experience in a STEM discipline , computational social science, computational humanities, or another research-intensive field involving programming and data analysis.
• Minimum 1 year of experience in an active research role within academia, industry, government laboratories, or a similar research environment.
• Demonstrated experience performing computational research involving Python , data analysis, simulation, modeling, machine learning, or scientific computing.
• Strong understanding of experimental design, hypothesis testing, statistical analysis, and rigorous interpretation of research findings.
• Working knowledge of Git , integrated development environments (IDEs), and notebook platforms such as Jupyter or Google Colab .
• Experience with AI evaluation, benchmark development, AI training, or task authoring is preferred.
• Excellent analytical thinking, attention to detail, creativity, and the ability to solve complex, open-ended problems independently.
• Strong written communication skills for documenting technical methodologies and research findings.
• Ability to commit approximately 35 hours per week on a consistent basis.

Preferred Qualifications

• Experience designing reproducible computational experiments or research workflows.
• Familiarity with machine learning, large language models, or AI-assisted research tools.
• Background in benchmark design, scientific software development, or computational research infrastructure.
• Experience mentoring researchers, reviewing scientific work, or contributing to peer-reviewed publications.

Why Join

• Help shape how next-generation AI systems are evaluated using rigorous scientific methodologies.
• Collaborate with leading AI researchers working on frontier models and advanced evaluation frameworks.
• Apply your research expertise to improve AI reasoning, reliability, and scientific accuracy.
• Contribute to impactful work that advances the quality and robustness of AI systems across multiple disciplines.
• Enjoy the flexibility of a fully remote engagement while working on cutting-edge AI research initiatives.

Equal Opportunity

We are committed to fostering an inclusive and diverse environment where all qualified applicants receive equal consideration. Reasonable accommodations are available throughout the application and engagement process.

Contract & Engagement Details

• Independent contractor engagement.
• Fully remote with flexible working hours.
• Expected commitment of approximately 35 hours per week .
• Project duration may be extended, shortened, or concluded based on project requirements and individual performance.
• Work does not require access to confidential or proprietary information from any current or former employer.
• Payments are issued weekly based on approved work completed.
• At this time, we are unable to support H1-B or STEM OPT candidates.