ML Engineer - Model Evaluation Expert
$60 - $90 per hourFull-time
Mercor
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Machine Learning Engineer — Model Evaluation & Experimentation
Type: Contract
Compensation: $60–$90/hour
Location: Remote
Commitment: 35 hours/week
Role Responsibilities
- Design tasks by transforming real ML research ideas into well-defined, multi-step tasks.
- Run experiments by implementing changes, executing training experiments, and analyzing results to define correct solutions.
- Explore reinforcement learning concepts such as reward functions and training behavior in task development.
- Evaluate frontier models' performance on tasks and identify areas of improvement.
- Collaborate with researchers to ensure tasks are consistent, rigorous, and fair.
Qualifications
Must-Have
- MSc or PhD in machine learning , computer science , or a related STEM field.
- 1+ years in a research or research-engineering role.
- Experience in training and evaluating ML models and conducting end-to-end experiments.
- Proficiency in Python and Git .
- Strong understanding of large language models and their evaluation.
Preferred
- Basic knowledge of reinforcement learning .
- Experience in AI training, model evaluation, or benchmark/task authoring.
Compensation & Legal
- Hourly contractor
- Paid weekly
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on jobs.jobcopilot.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 18 hours ago
Similar jobs that could be interesting for youBased on the ML Engineer - Model Evaluation Expert in San Francisco, CA vacancy
- ...looking for a Senior Machine Learning Engineer to join our Search & Intelligence organization... ...-native experiences, agentic systems, models, evaluation frameworks, and data platforms that... .... You will own projects across the ML lifecycle, from problem definition and...SuggestedWork at officeLocal area
Atlassian
San Francisco, CA5 days ago - ...We are looking for a visionary Senior ML Engineer who will bridge the gap between high-level... ...monitoring, and performance tuning of AI models in live environments Automate SDLC... ...; building clustering models; utilizing evaluation frameworks to quantify performance ~ Proficiency...SuggestedShift work
Palm Venture Studios
San Francisco, CA26 days ago - ...build, and deploy production‑grade ML systems with end‑to‑end ownership of the model lifecycle from conception to... ...of professional experience in ML engineering. Strong programming skills in Python... ...for model training and evaluation. Familiarity with FastAPI, OpenAI...SuggestedFull time
- Machine Learning Engineer, Applied AI As a Machine Learning Engineer you'll join our Product Innovations team and work across the full applied ML stack - deploying models, building the evaluation systems that tell us whether they actually work, and making the data and...SuggestedWork at officeWork from homeHome office
- The role At Mach9, ML Engineers build the perception models at the core of our AI-enabled CAD system. We build models to extract 3D object and line features... ...ML model backs. Responsibilities Design, train, and evaluate computer vision and 3D ML models for extracting CAD-...Suggested
- ML Engineer Tilde Research is a moonshot AI lab advancing mechanistic interpretability, new... ...We build foundational understanding of models to advance the frontier of intelligence.... ...on the systems that support training and evaluating large models, scaling experimental pipelines...Full timeInternship
$213k - $263k
...ground for the Waymo Driver. The Simulator Evaluation team faces the ultimate data challenge:... ...are seeking visionary machine learning engineers and researchers to architect the... ...measure the realism of our multimodal world models. Your work will define the state of the...Full timeRemote work$250k - $400k
...pharma. The challenge isn't just model capability. It's building the... ...these models to be trained, evaluated, and deployed reliably at... ...isolation. It's building the engine that research runs on. You'll... ...Experience building and scaling ML systems in production Strong background...Remote work$160k - $250k
...breaking speeds. About You and The Role As an ML Training & Inference Infrastructure Engineer on the Data Platform team you will be building... ...dataset management, distributed training, continuous model integration, evaluation, and deployment. The ideal candidate has strong...Local areaZipline
San Francisco, CA12 days ago- ...seeking a Machine Learning Infrastructure Engineer to help architect the compute, training,... ...execution frameworks powering next-generation model performance. Backed by top-tier venture... ...and optimize resource efficiency.Evaluate cutting-edge developments in model optimization...Full timeWork at officeFlexible hours
Objective Paradigm
San Francisco, CAa month ago $227.33k - $312.58k
...Description Job Description We’re looking for a Staff ML Data Engineer to join Procore’s AI & Frontier Models organization. In this role, you’ll be responsible... ...is an opportunity to shape how Procore builds, evaluates, and deploys frontier models by ensuring the...Work at officeLocal areaImmediate start3 days per weekProcore
San Francisco, CA16 days ago- ...noise characteristics. Design models that are robust,... ...visual signals Production Engineering Write clean, scalable, well... ...codebase. Build end-to-end ML pipelines including data processing... ...feature extraction, training, evaluation, and deployment. Optimize...Live inLocal area
Gridware
San Francisco, CA24 days ago - ...technical owner to lead post-training projects for their audio models. The role requires translating customer needs into specifications, designing function calling capabilities, and performing evaluations for production use. Candidates should have hands-on experience in...
- ...Infrastructure team at Zensors builds the engine that powers our visual sensing... ...lifecycle of our AI workflow, including model development, evaluation, optimization, deployment, and monitoring... ...As a Machine Learning Engineer in ML Runtime & Optimization , you will develop...
Zensors
San Francisco, CA10 days ago $120k - $200k
...cloud platform, automated build/test/deploy pipelines, and model observability. We value evaluation discipline, including building eval datasets, measuring... ..., and cost. A degree in computer science, AI/ML, or a related field may be useful, but it is not required...Full timeTemporary workRemote workCyrano Technologies Inc
San Francisco, CA3 days ago- ...are now building our Silicon Valley engineering team — a small, senior group focused... ...reasoning loops. • Own prompt engineering, model versioning, evaluation (RAGAS, DeepEval), and LLMOps... ...or related field. • 3–5 years of AI/ML engineering; minimum 2 years building...Work at officeVisa sponsorship
Hop
San Francisco, CA25 days ago $240k - $270k
...fast.That’s where you come in. As an AI/ML Engineer, you’ll join a growing team focused on building... ...adapting or training foundation models (language or multimodal) for novel domainsBonus... ...your personal data for the purposes of evaluating your candidacy for employment at Sigma...Full timeWork at officeSigma Computing
San Francisco, CAa month ago- ...Our client is a well-funded AI startup building production-grade ML infrastructure used by enterprise customers. They are looking for a Senior AI/ML Engineer to own model training pipelines, evaluation systems, and inference serving at scale. Full-time, on-site in San...Full time
Clera
San Francisco, CA26 days ago $150k - $225k
...re looking for an Founding AI Engineer to design, build, and scale the... ...that connects large language models, vector databases, and real-world... ...Develop feedback loops and evaluation systems to measure and improve... ...building and scaling data or ML pipelines in production environments...Full timeWork from homeFlexible hours- ...reach patients. The Role As a Senior AI/ML Engineer at Condor, you will own the design and... ...you will build, train, and serve custom models where they outperform general-purpose approaches... ...-tool interfaces, retrieval systems, evaluation harnesses, and serving infrastructure-...Live inImmediate start
- AI/ML Engineer (Computer Vision) Location: On site, Bay Area, CA A fast-growing applied AI... ...close to product decisions, and shipping models that improve how customers analyze... ...improve model pipelines across training, evaluation, inference, and monitoring. Partner with...
- ...role We're looking for an AI/ML Engineer to join our product... ...that's a novel machine learning model or plain software engineering... ...real product, designing the evaluations that prove they work, and turning... ...worldwide. Collaborate with learning experts and product engineers to...Work at officeWorldwideShift work
- ...is the first Machine Learning Engineer role at Metriport. We have access... ...customers, to training the models, to serving them in production... ...them honest. This is applied ML on messy, high-dimensional, real... ...them: you have opinions about evaluation, monitoring, retraining, and...Work at officeWork from homeRelocationFlexible hours
$131.4k - $236k
...of growth, we’re investing deeply in AI/ML, LLMs, and Industrial IoT to transform how... ...data ingestion and feature development to model training, deployment, and monitoring.... ...instilling best practices for experimentation, evaluation, and model lifecycle management.Partner...MaintainX
San Francisco, CA13 days ago$200.35k - $260.91k
...hiring a Senior Machine Learning Engineer to join our AI team, reporting... ..., and who is deciding. Your models power products customers use... ...and reliability of production ML systems, and help shape AI platform... ...of feature engineering, model evaluation, and MLOps practices.A product...Full time6sense
San Francisco, CA14 days ago- ...with data centers. About the Role At Watney, ML engineers turn a live fleet of robots into better models. The fleet produces large volumes of video from... ...help curate and label the data behind them, and help evaluate what works on a real fleet. What You’ll Do...
Watney
San Francisco, CA3 days ago - ...understand, and ship. We build the evaluation systems, datasets, quality signals, and production ML capabilities that reveal where... ...doAs a Senior Machine Learning Engineer, you’ll tackle the open-ended... ...teams: driving advances in models, agents, and evaluation and turning...Work at officeLocal area
Atlassian
San Francisco, CA11 days ago $145.5k - $249.5k
...Connecting. Growing together. As a Lead AI/ML Engineer on the AI Platform Team within Optum AI,... ...ingestion and generative agents to AI model deployment, clinical applications, and... ...term technical strategy for AI/ML systems, evaluating emerging trends and research to inform...Minimum wageFull timeWork experience placementWork at officeLocal areaRemote workUnitedHealth Group
San Francisco, CA24 days ago$172.5k - $210k
...are looking for a Machine Learning Engineer to develop and optimize our ad science models, open-source LLMs, and the... ...performance and cost. AI and ML sit at the core of InMobi's business... ...efficiency Build monitoring and evaluation solutions that keep our infrastructure...Local areaWorldwideFlexible hoursInMobi
San Francisco, CA5 days ago- ...ensure our vision transformers and spatial models run efficiently on both cloud and edge... ...more at About the Role As an ML / DevOps Engineer, you will play a pivotal role in advancing... .... Systems & Networking Mastery: Expert-level knowledge of Linux administration...Work at office
Zensors
San Francisco, CA26 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to ML Engineer - Model Evaluation Expert. Be the first to apply!
Related searches
- machine learning engineer San Francisco, CA
- senior ml engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- machine learning software engineer San Francisco, CA
- ai ml engineer San Francisco, CA
- computer vision machine learning engineer San Francisco, CA
- machine learning part time San Francisco, CA
- data engineer machine learning San Francisco, CA
- intern - quantum machine learning for quantum computing San Francisco, CA
- internship machine learning San Francisco, CA