Astrophysicist for AI Model Evaluation

Design graduate-level, research-focused computational problems that test whether advanced AI systems can perform real scientific workflows in astrophysics and cosmology. You will author original tasks that require use of scientific software to run simulations, analyze outputs, design experiments, and extract hidden information from data. Each task will be exercised against top AI models and iteratively refined until it matches the intended difficulty and reasoning requirements.

• Create original, challenging problems that require skilled use of specialized scientific software and realistic research workflows.
• Author problems of two main types: ones with fully specified setups that require exact, multi-step computational answers, and investigative tasks where the AI must plan measurements or queries to uncover information not directly visible.
• Implement problem setups, oracle functions, and solution validators in Python.
• Run and evaluate problems against state-of-the-art AI models, analyze their behavior, and iterate on problem design to achieve target difficulty and robustness.
• Ensure problems emphasize strategic reasoning and avoid solvability by surface-level pattern matching or trivial computation.
• Work within remote compute sandboxes and a Linux/terminal environment to validate reproducibility and execution of the tasks.

• Graduate-level training in a relevant STEM field, MS or PhD preferred, or equivalent research experience.
• Proven, hands-on proficiency with at least one relevant scientific software library, demonstrated through research publications, open-source contributions, or professional work. Strong preference for experience in astrophysics and cosmology tools, specifically astropy and related packages for cosmological calculations, angular power spectra, galaxy survey analysis, and observational data reduction pipelines. Experience with other specialized domain software will also be considered.
• Strong Python programming skills, including writing problem setups, validators, and reproducible execution scripts.
• Comfortable working in Linux and terminal environments, able to use remote compute sandboxes for testing tasks.
• Independent, iterative problem designer mindset, able to refine tasks based on feedback and model performance.
• Availability for at least 15 to 20 hours per week.

• Experience across multiple relevant domains or scientific tools.
• Familiarity with benchmark or evaluation design for scientific tasks.
• Background in teaching, exam, or problem-set design at the undergraduate or graduate level.
• Experience with computational reproducibility, containerized environments, or workflow portability.

• Location: Remote.
• Employment type: hourly.
• Minimum weekly commitment: 15 to 20 hours.
• Work involves independent design and remote validation using provided compute sandboxes and Linux/terminal tools.

• 70 - 100 hourly.