Astrophysics & Cosmology Computational Problem Designer [Remote]

This role focuses on creating a large-scale evaluation benchmark for advanced AI reasoning in scientific and engineering domains. As a task designer, you will develop challenging computational problems that assess whether AI systems can effectively utilize real scientific software tools to tackle research-grade issues, ranging from querying simulations to designing experimental strategies and extracting hidden information from data.

This position is distinct from typical annotation or labeling roles. You will be responsible for designing original, graduate-level computational problems rooted in authentic scientific workflows, calibrating them against cutting-edge AI models, and iterating on the problem design to ensure the appropriate level of difficulty.

• Design problems that necessitate advanced use of domain-specific scientific software libraries.
• Create tasks that require precise outputs from fully specified setups, testing the solver''s ability to implement complex multi-step scientific workflows.
• Develop sequences of queries or experiments to uncover non-visible information, requiring strategic reasoning about measurements, interpretation of partial observations, and efficient narrowing of possibilities.
• Engage in a calibration loop for each task, testing against state-of-the-art AI models and refining problem design to achieve target difficulty levels.

We are particularly interested in candidates with extensive, hands-on experience in:

• Astrophysics & Cosmology: Proficiency with astropy and related tools for cosmological calculations, angular power spectra, galaxy survey analysis, and observational data reduction pipelines.
• Experience with other specialized software in this domain will also be considered.

The ideal candidate will possess graduate-level expertise (MS or PhD preferred) in the specified domain, along with real hands-on experience using the relevant software tools. You should have:

• Written code that utilizes these libraries to solve actual research problems, with a clear understanding of their limitations, edge cases, and the nature of genuinely challenging problems.
• The ability to think like a puzzle designer, crafting problems where difficulty arises from reasoning strategy rather than mere computation, and where careful analysis is required to identify the correct approach.

• Graduate-level training in a relevant STEM domain (MS, PhD, or equivalent research experience).
• Demonstrated proficiency with at least one of the specified scientific software libraries, supported by research publications, open-source contributions, or professional experience.
• Strong Python programming skills for writing problem setups, oracle functions, and solution validators.
• Ability to work independently and iterate on problem designs based on calibration feedback.
• Comfort with Linux/terminal environments and remote compute sandboxes.
• Availability for at least 15, 20 hours per week.

• Experience across multiple listed domains or tools.
• Familiarity with benchmark or evaluation design.
• Background in scientific pedagogy or exam/problem-set design.
• Experience with computational reproducibility and containerized environments.