Data Scientist, AI & Machine Learning (PhD Student)

Job Type: Flexible Contract (Remote)

Cobalt is hiring current Masters and PhD students to review reinforcement learning and data science tasks and environments used to test whether frontier AI models can carry out real research, analysis, and engineering work. Your job is to make sure each one is correct, fair, and actually measures what it claims to measure before it is accepted. If you enjoy finding the flaw in an experiment, the leak in a dataset, the bug in a grader, or the loophole in a reward function, this role is built around that skill.

What you will do

• Review tasks end to end, including the task description, environment code, datasets, dependencies, reference solution, and automated tests or graders, and confirm that each part is correct and consistent with the others.
• Run reference solutions and graders yourself to confirm that correct solutions pass, incorrect solutions fail, and results are reproducible. Because model training and reinforcement learning results vary between runs and random seeds, you will check that tests are deterministic or use tolerances that are clearly justified.
• Audit the data behind each task, checking for leakage between training and test sets, mislabeled or inconsistent records, and evaluation metrics that do not match the stated goal.
• Look for ways an agent could pass a task without doing the intended work, such as exploiting a reward signal, reading the answer from the environment or the data, or satisfying a weak test, and document each issue so the task author can fix it.
• Review how a frontier AI model attempted each task and judge whether its failure reflects a real gap in capability, or instead reflects ambiguous wording, a hidden assumption, a broken environment, flawed data, or an unfair grader.
• Write clear, specific feedback to task authors and work with them through revisions until the task meets the acceptance standard.
• Track recurring problems across tasks and help improve the review checklist and quality standards over time.

Example tasks you might review include implementing proximal policy optimization from a paper’s description alone, finding the bug in a flawed reward function or data pipeline, building a model on a messy tabular dataset without leaking information from the test set, reproducing a reported result and explaining a discrepancy, and running a sound statistical comparison between two models or agents across multiple random seeds.

Who we are looking for

• Currently enrolled in a Masters or PhD program in machine learning, data science, computer science, statistics, robotics, or a closely related field.
• Hands-on experience with machine learning, including reinforcement learning, through research, a thesis, coursework projects, or industry work. You should be comfortable reading an implementation or analysis and saying whether it is correct.
• Strong working knowledge of Python and its data stack, including pandas and NumPy, at least one major framework such as PyTorch or JAX, and familiarity with environment interfaces such as Gymnasium. Working knowledge of SQL is a plus.
• Comfort with the Linux command line, Docker, and Git, enough to set up an environment, run a task, and trace a failure.
• A solid grasp of experimental design and statistics, including variance across seeds, confidence intervals, validation strategy, and when a difference between two results is meaningful.
• Careful, direct written communication. Much of the job is explaining precisely what is wrong and why.

Strong advantages

• Experience as a teaching assistant who wrote assignments or autograders, or as a peer reviewer for a conference, workshop, or journal.
• A publication or preprint in machine learning or applied statistics, which we value but do not require.
• Experience working with healthcare, clinical, or biological data, such as electronic health records, medical imaging, genomics, or laboratory data. Some projects involve these domains, and reviewers who understand them are especially valuable.

Why Cobalt AI

• Work on frontier AI evaluation. Your judgment directly determines which tasks are used to measure what leading models can and cannot do.
• Build skills that matter for your research. Reviewing many tasks and model attempts gives you a close view of how frontier models are trained and assessed, and where they break.
• Fit the work around your degree. Set your own schedule, with flexible weeks of 10 to 40 hours that can scale down during busy academic periods.
• Grow with us. Strong reviewers can move into task design, project leadership, and research collaborations on future projects.