Machine Learning Engineer for AI Model Benchmarking [Remote]

This position offers an exciting opportunity for experienced machine learning engineers and researchers to act as human baseliners for evaluating open-ended machine learning research tasks. Your role will involve performing realistic AI R&D problems, providing essential human reference points to interpret AI agent performance. You will work independently in a sandboxed environment, utilizing your preferred tools and workflows to complete self-contained ML research tasks, which will serve as benchmarks for evaluating frontier-model agents.

• Attempt open-ended machine learning research tasks under a fixed time and compute budget.
• Work independently in a sandboxed Linux environment with internet access.
• Utilize your preferred tooling, including IDEs and AI coding assistants such as Cursor, Claude Code, and ChatGPT.
• Record your full working session via screen recording.
• Complete a short pre-task and post-task questionnaire.
• Submit your final work product, screen recording, and completed questionnaires; successful candidates will be considered for longer commitments.

Selected candidates are expected to commit to a minimum of 20 hours per week, with more availability strongly preferred.

Candidates must meet all of the following requirements:

• 3+ years of machine learning experience (time spent in a PhD program counts toward this requirement; undergraduate and master’s experience does not).
• Graduated from a top-100 university or worked at a FAANG or comparable company.
• Experience with at least one major ML framework such as PyTorch, JAX, or TensorFlow.
• Deep, hands-on expertise in at least one of the following focus areas:
• Pretraining under tight data and compute budgets.
• PPO, reward shaping, custom gym/gymnasium environments, and throughput tuning.
• Full fine-tuning, LoRA, QLoRA, DPO, RLHF, RLAIF, and distillation.
• Large-scale corpus filtering, deduplication, subsampling, and benchmark contamination avoidance.
• Architecture design under strict parameter-count or size constraints.
• Modifying pretrained architectures, including attention patterns, pooling heads, or training objectives.
• Contrastive training for embedding or retrieval models.
• Generative vision or video modeling.
• Multilingual or low-resource language experience.
• Image or video data pipelines at scale.
• Experience balancing competing model objectives such as safety and capability.
• Prior work as an ML evaluator, red-teamer, or baseliner.

Candidates must have strong practical experience in at least one of the following:

• Pretraining: training transformer language models from scratch.
• Reinforcement learning: training agents in custom or existing environments.
• Post-training: fine-tuning and aligning LLMs.
• Dataset curation: building and cleaning large text corpora for LLM training.
• Model architecture: designing and modifying neural network architectures.

• One baseline attempt per contractor per task.
• Each task may only be attempted once by a given contractor.
• All work is confidential and covered by NDA.
• Compute and environment are provided; no personal GPU is required.