Bioinformatics & Single-Cell Genomics Problem Designer for AI Evaluation [Remote]

This role focuses on creating a large-scale evaluation benchmark for advanced AI reasoning in scientific and engineering domains. As a task designer, you will develop challenging computational problems that assess AI systems'' ability to utilize real scientific software tools for solving research-grade issues, from querying simulations and interpreting outputs to designing experimental strategies and extracting hidden information from data.

This position is distinct from typical annotation or labeling roles. You will be responsible for designing original, graduate-level computational problems rooted in real scientific workflows, calibrating these problems against cutting-edge AI models, and refining the designs to ensure the appropriate level of difficulty.

• Design problems that necessitate sophisticated use of domain-specific scientific software libraries.
• Create tasks that require precise outputs from fully specified setups, testing solvers on complex multi-step scientific workflows.
• Develop sequences of queries or experiments to uncover non-visible information, requiring strategic reasoning about measurements, interpretation of partial observations, and efficient narrowing of possibilities.
• Engage in a calibration loop where tasks are tested against state-of-the-art AI models, refining problem designs until they meet targeted difficulty levels.

We are particularly interested in candidates with extensive, hands-on experience in:

• Experience with tools such as scanpy, scvelo, squidpy, and gudhi for single-cell RNA-seq analysis, trajectory inference, spatial transcriptomics, and topological data analysis.
• Ability to design problems related to cell-type annotation, pseudotime ordering, multi-omic integration, spatial variable gene identification, and persistence-based analysis pipelines.

The ideal candidate will possess graduate-level expertise (MS or PhD preferred) in the relevant domain, with practical experience using specific software tools rather than just theoretical knowledge. You should have:

• Written code that utilizes these libraries to address actual research problems, understanding their limitations and edge cases.
• The ability to think like a puzzle designer, creating problems where difficulty arises from reasoning strategy rather than mere computation.
• A knack for constructing tasks that require careful analysis to identify the correct approach among multiple plausible solutions.

• Graduate-level training in a relevant STEM domain (MS, PhD, or equivalent research experience).
• Demonstrated proficiency with at least one of the specified scientific software libraries, supported by research publications, open-source contributions, or professional experience.
• Strong Python programming skills for writing problem setups, oracle functions, and solution validators.
• Ability to work independently and iterate on problem designs based on calibration feedback.
• Comfortable working in a Linux/terminal environment with remote compute sandboxes.
• Availability for at least 15, 20 hours per week.

• Experience across multiple listed domains or tools.
• Familiarity with benchmark or evaluation design.
• Background in scientific pedagogy or exam/problem-set design.
• Experience with computational reproducibility.