Computational Bayesian Statistics Expert for AI Evaluation Design [Remote]

This role involves designing original, graduate-level computational problems that challenge AI systems to utilize real scientific software tools for solving complex research-grade problems. You will create tasks that test AI''s ability to query simulations, interpret outputs, design experimental strategies, and extract hidden information from data.

Unlike typical annotation roles, your focus will be on calibrating these problems against cutting-edge AI models, iterating on their design to ensure the difficulty aligns with target benchmarks.

• Design computational problems that require advanced use of domain-specific scientific software libraries.
• Create tasks that test solvers on multi-step scientific workflows, ensuring precise outputs from fully specified setups.
• Develop complex queries or experimental designs to uncover non-visible information, requiring strategic reasoning.
• Engage in a calibration loop, testing each task against state-of-the-art AI models and refining designs based on feedback.

We are particularly interested in candidates with extensive hands-on experience in:

• Computational Bayesian Statistics and Applied Mathematics, utilizing libraries such as PyMC, PyStan, PyJAGS, and CmdStanPy.
• Applied mathematics and numerical PDEs, including tools like FEniCS, FEniCSx, DOLFINx, scikit-fem, FiPy, Devito, and Dedalus.
• Computational topology, using GUDHI.
• Differential algebra, including DACEyPy.

Experience with MCMC, Bayesian modeling, finite element or finite difference methods, mesh-based numerical modeling, computational topology, or other specialized Python-based computational methods is also valuable. Candidates are not required to have experience with all listed packages, but proficiency in any one will be highly regarded.

The ideal candidate will possess:

• Graduate-level expertise (MS or PhD preferred) in the relevant domain, with practical experience using specific software tools.
• Proven ability to write code that utilizes these libraries to address real research problems, understanding their limitations and complexities.
• A puzzle designer mindset, capable of constructing problems that emphasize reasoning strategy over mere computation.

• Graduate-level training in a relevant STEM domain (MS, PhD, or equivalent research experience).
• Demonstrated proficiency with at least one of the specified scientific software libraries, supported by research publications, open-source contributions, or professional experience.
• Strong Python programming skills for writing problem setups, oracle functions, and solution validators.
• Ability to work independently and iterate on problem designs based on calibration feedback.
• Comfort with Linux/terminal environments and remote compute sandboxes.

This is a remote, hourly position.

Hourly compensation ranges from $70 to $100.