ML Engineer - Model Evaluation Expert
$60 - $90 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Machine Learning Engineer — Model Evaluation & Experimentation
Type: Contract
Compensation: $60–$90/hour
Location: Remote
Commitment: 35 hours/week
Role Responsibilities
- Design tasks by transforming real ML research ideas into well-defined, multi-step tasks.
- Run experiments by implementing changes, executing training experiments, and analyzing results to define correct solutions.
- Explore reinforcement learning concepts such as reward functions and training behavior in task development.
- Evaluate frontier models' performance on tasks and identify areas of improvement.
- Collaborate with researchers to ensure tasks are consistent, rigorous, and fair.
Qualifications
Must-Have
- MSc or PhD in machine learning , computer science , or a related STEM field.
- 1+ years in a research or research-engineering role.
- Experience in training and evaluating ML models and conducting end-to-end experiments.
- Proficiency in Python and Git .
- Strong understanding of large language models and their evaluation.
Preferred
- Basic knowledge of reinforcement learning .
- Experience in AI training, model evaluation, or benchmark/task authoring.
Compensation & Legal
- Hourly contractor
- Paid weekly
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 6 days ago
Similar jobs that could be interesting for youBased on the ML Engineer - Model Evaluation Expert in San Francisco, CA vacancy
- ...with safety, privacy, and real-world responsibility in mind. Our ML team comes from a culture of academic research driven to... ...privacy for the sake of ML advancement. Responsibilities Own LLM evaluation processes and methods with a focus on generating benchmarks representative...SuggestedLocal areaShift work
$213k - $263k
...ground for the Waymo Driver. The Simulator Evaluation team faces the ultimate data challenge:... ...are seeking visionary machine learning engineers and researchers to architect the... ...measure the realism of our multimodal world models. Your work will define the state of the...SuggestedFull timeRemote workWaymo
San Francisco, CA19 hours ago- Design Arena hosts cutting-edge ML evaluation platforms and data engines. This role focuses on building systems that learn from human preference signals to evaluate frontier AI models. You will collaborate with researchers and scale data pipelines for large-scale experiments...Suggested
- ...-Scientist. It is a discovery engine that transforms messy biological... ...THE ROLE We are hiring an ML Engineer, Discovery... ...combine analysis modules, ML models, domain logic, and agentic reasoning... ...specific workflows for compound evaluation, program prioritization, and multi...SuggestedFull timeWork at office
Mithrl
San Francisco, CA19 hours ago - ...We are looking for a visionary Senior ML Engineer who will bridge the gap between high-level... ...monitoring, and performance tuning of AI models in live environments Automate SDLC... ...; building clustering models; utilizing evaluation frameworks to quantify performance ~ Proficiency...SuggestedFull timeShift work
Palm Venture Studios
San Francisco, CA19 hours ago - ...Responsibilities: Own evaluation pipelines — design, build, and... ...keep our speech and multimodal models honest in production. Harness... ...Required Qualifications: Expert-level PyTorch. Proven software engineer who loves ML; comfortable writing production...Full timeContract workFlexible hoursShift work
Sesame, L.l.c.
San Francisco, CA19 hours ago - ...advances in speech, language models, and clinical AI can fundamentally... ...We are hiring two ML Engineers / Researchers to help build the... ...You do not need to be an expert in both areas. We are looking... ...experiments, build datasets and evaluation systems, train and fine-tune...Full time
Knowtex
San Francisco, CA7 hours ago - ...science. We build foundational understanding of models to advance the frontier of intelligence. About the role: As a ML Engineer, you’ll build and operate the... ...work on the systems that support training and evaluating large models, scaling experimental pipelines...Full timeInternship
Tilde Research
San Francisco, CA19 hours ago $148.5k - $223.9k
...This role is for a Senior Machine Learning Engineer within the Trust Intelligence Platform... ...capabilities like probabilistic modeling and graph analytics, and operationalize... ...tools to help our recruiters assess and evaluate candidates’ resumes and qualifications throughout...Full time- ...experienced Machine Learning Engineer to join our team and help develop... ...-edge speech recognition models that help teach language fluency... ...incredibly exciting time to join an ML team designing a personalized... ...such as training/evaluation datasets and labeling pipelines...Full timeLive inWork at officeWorldwide
Speak
San Francisco, CA19 hours ago $170k - $280k
...mission is to give leaders clarity and engineers time. We help leaders understand... ...We're looking for an Applied ML Engineer to help build and improve... ...primary focus will be on improving model quality through high-quality evaluation datasets, rigorous experimentation,...Odd jobFull timeMacroscope Inc
San Francisco, CA19 hours ago$77k - $202k
...the chance to work alongside healthcare experts and data scientists, directly influencing... ...least 4 years of experience in software engineering or data engineeringWhat Sets You Apart-... ...collaborating closely with team members. We evaluate these factors thoughtfully to establish...Full timeH1b- ...seeking a Machine Learning Infrastructure Engineer to help architect the compute, training,... ...execution frameworks powering next-generation model performance. Backed by top-tier venture... ...and optimize resource efficiency.Evaluate cutting-edge developments in model optimization...Full timeWork at officeFlexible hours
$227.33k - $312.58k
We’re looking for a Staff ML Data Engineer to join Procore’s AI & Frontier Models organization. In this role, you’ll be responsible for designing and building... ...This is an opportunity to shape how Procore builds, evaluates, and deploys frontier models by ensuring the...Full timeWork at officeLocal areaImmediate start3 days per week- ...out 1962 new Machine Learning Engineer opportunities posted on AI Chopping... ...and optimize end-to-end ML pipelines encompassing data collection... ...MLOps practices such as model versioning, experiment... ...Machine Learning Enginer, Core Evaluations The responsibilities include designing...Flexible hours
AI Chopping Block, Inc.
San Francisco, CA3 days ago $162.8k - $203.5k
...mapping data possible, along with algorithms, models, platform services, and map-based user... ...our efforts, we are hiring a Senior ML Engineer who will work end-to-end on creating and... ...launch machine learning models at scale Evaluate machine learning systems against...Hourly payFull timeWork experience placementWork at officeLocal area3 days per weekLyft
San Francisco, CA19 hours ago$162.8k - $203.5k
...lives warrants modern ML utilizing peta-byte scale... ...Machine Learning Engineers work on these challenging... ...We are hiring motivated experts in each of these fields... ...learning, or optimization models Write production... ...learning models at scale Evaluate machine learning...Hourly payFull timeWork experience placementWork at officeLocal area3 days per weekLyft
San Francisco, CA19 hours ago$190k - $205k
...noise characteristics. Design models that are robust,... ...and visual signals Production Engineering Write clean, scalable, well... ...codebase. Build end-to-end ML pipelines including data processing... ...extraction, training, evaluation, and deployment. Optimize...Full timeLive inGridware
San Francisco, CA19 hours ago- ...next-generation multimodal AI models and a proprietary, high-... ...with hands-on support from AMD engineers the team is scaling rapidly to... ...applications. About the role As an ML Engineer at Sciforium, you... ...tool-based systems. Develop evaluation frameworks that measure...Full timeFlexible hours
Sciforium
San Francisco, CA2 days ago - ...AfterQuery is seeking a Founding Machine Learning Engineer, Quality Intelligence to build ML and data systems behind how we measure, improve, and... ...of machine learning, human expertise, and frontier model evaluation. This role is for someone who wants to build practical...
AfterQuery
San Francisco, CA20 minutes ago - ...Perplexity is seeking experienced ML engineers to design, build, and optimize the recommendation... ...and business metrics. Build user modeling that captures intent, preference, and... ...experience for the user. Build the data and evaluation foundations that let these systems...Full time
Perplexity®️
San Francisco, CA19 hours ago - ...Francisco is actively hiring Machine Learning Engineers to extend our API platform into cloud-... ...environments, with a focus on AWS-hosted Codex, model customization, and post-training workflows. You will work across training, evaluation, data pipelines, model behavior, and API/...
- ...pioneering hedge fund in San Francisco is seeking a Founding ML Engineer to architect and build machine learning systems for investment... ...in developing reproducible ML pipelines, implementing evaluation frameworks, and establishing robust MLOps practices. #J-18808...
- ...our training systems and core model code. You’ll own critical infrastructure... ...with researchers and model engineers to translate ideas into... ...leverage role at the intersection of ML, software engineering, and... ..., modalities, and evaluation metrics. What We Hope You’ll...Full time
- The role At Mach9, ML Engineers build the perception models at the core of our AI-enabled CAD system. We build models to extract 3D object and line features... ...ML model backs. Responsibilities Design, train, and evaluate computer vision and 3D ML models for extracting CAD-...
- ...documents codebases autonomously. As a Founding ML Engineer , you’ll architect the intelligence... ...Develop and refine agent behavior models for planning, implementation, and review... ...across the OS Build internal tools for evaluating intelligence, tracing decisions, and tuning...Remote workFlexible hours
- Clay is seeking a Machine Learning Engineer to join the Learning Team and help build a self-learning revenue engine. You'll ship... ...features, design data pipelines, and contribute to ML platform and evaluation systems. You’ll collaborate with product teams to make surfaces...
$198k - $230k
.... We operate a flexible work model that combines both in‑person... ...individual work styles. Senior MLOps Engineer (Applied AI Focus) As a... ...truth generation, model evaluation, and pre/post processing decisions... ...Annotation & Evaluation Expert: You have deep experience setting...Work at officeRemote workWork from homeWorldwideHome officeFlexible hours- A leading AI evaluation platform in San Francisco is looking for a Senior Software Engineer specializing in ML infrastructure. The successful candidate will design and develop robust real-time data and API systems, enabling insights for researchers and developers. Ideal...
- ...don’t believe culture can be engineered - but when it falls into place... ...We’re looking for an ML engineer to design, train, and... ...language-action (VLA) foundation model at the core of Humble’s autonomous... ...training to closed-loop evaluation in simulation and deployment...Local area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to ML Engineer - Model Evaluation Expert. Be the first to apply!
Related searches
- ai ml engineer San Francisco, CA
- senior ml engineer San Francisco, CA
- computer vision machine learning engineer San Francisco, CA
- machine learning engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- junior machine learning research engineer San Francisco, CA
- data scientist machine learning engineer San Francisco, CA
- graduate machine learning engineer San Francisco, CA
- machine learning software engineer San Francisco, CA
- machine learning scientist San Francisco, CA