ML Engineer - Model Evaluation Expert

Job Description

Job Description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Machine Learning Engineer — Model Evaluation & Experimentation
Type: Contract
Compensation: $60–$90/hour
Location: Remote
Commitment: 35 hours/week

Role Responsibilities

• Design tasks by transforming real ML research ideas into well-defined, multi-step tasks.
• Run experiments by implementing changes, executing training experiments, and analyzing results to define correct solutions.
• Explore reinforcement learning concepts such as reward functions and training behavior in task development.
• Evaluate frontier models' performance on tasks and identify areas of improvement.
• Collaborate with researchers to ensure tasks are consistent, rigorous, and fair.

Qualifications

Must-Have

• MSc or PhD in machine learning , computer science , or a related STEM field.
• 1+ years in a research or research-engineering role.
• Experience in training and evaluating ML models and conducting end-to-end experiments.
• Proficiency in Python and Git .
• Strong understanding of large language models and their evaluation.

Preferred

• Basic knowledge of reinforcement learning .
• Experience in AI training, model evaluation, or benchmark/task authoring.

Compensation & Legal

• Hourly contractor
• Paid weekly

Application Process (Takes 20–30 mins to complete)

• Upload resume
• AI interview based on your resume
• Submit form

Resources & Support

• For details about the interview process and platform information, please check:
• For any help or support, reach out to: View email address on us.fitly.work

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.