ML Engineer - Model Evaluation Expert
$60 - $90 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Machine Learning Engineer — Model Evaluation & Experimentation
Type: Contract
Compensation: $60–$90/hour
Location: Remote
Commitment: 35 hours/week
Role Responsibilities
- Design tasks by transforming real ML research ideas into well-defined, multi-step tasks.
- Run experiments by implementing changes, executing training experiments, and analyzing results to define correct solutions.
- Explore reinforcement learning concepts such as reward functions and training behavior in task development.
- Evaluate frontier models' performance on tasks and identify areas of improvement.
- Collaborate with researchers to ensure tasks are consistent, rigorous, and fair.
Qualifications
Must-Have
- MSc or PhD in machine learning , computer science , or a related STEM field.
- 1+ years in a research or research-engineering role.
- Experience in training and evaluating ML models and conducting end-to-end experiments.
- Proficiency in Python and Git .
- Strong understanding of large language models and their evaluation.
Preferred
- Basic knowledge of reinforcement learning .
- Experience in AI training, model evaluation, or benchmark/task authoring.
Compensation & Legal
- Hourly contractor
- Paid weekly
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the ML Engineer - Model Evaluation Expert in San Francisco, CA vacancy
- ...looking for a Senior Machine Learning Engineer to join our Search & Intelligence organization... ...-native experiences, agentic systems, models, evaluation frameworks, and data platforms that... .... You will own projects across the ML lifecycle, from problem definition and...SuggestedWork at officeLocal area
Atlassian
San Francisco, CA4 days ago $70 per hour
...simulation across 15+ U.S. states. Software Engineering builds the brains of Waymo's fully autonomous... ...rewarding internship! You will: Conduct ML modeling experiments and prototyping to build state-of-the-art evaluation tools. Write code within Waymo’s...SuggestedHourly payFull timeInternshipSummer internshipWaymo
San Francisco, CA1 day ago- ...We are looking for a visionary Senior ML Engineer who will bridge the gap between high-level... ...monitoring, and performance tuning of AI models in live environments Automate SDLC... ...; building clustering models; utilizing evaluation frameworks to quantify performance ~ Proficiency...SuggestedShift work
Palm Venture Studios
San Francisco, CAa month ago - ...Atlassian is seeking a Senior Machine Learning Engineer to join the Search & Intelligence team in... .... You will own projects across the ML lifecycle—from problem definition to... ...teams. You will mentor others and shape evaluation-driven development. #J-18808-Ljbffr Jobleads...Suggested
$180k - $230k
...ML Infrastructure Engineer San Francisco Company Overview Echo Neurotechnologies... ...power massive-scale data, modeling, and analysis platforms,... ...closely with domain experts to architect new cloud... ...pretraining, fine-tuning, and evaluation processes Design and...SuggestedFlexible hoursEcho Neurotechnologies
San Francisco, CA3 days ago$200k - $260k
...Role As the founding machine learning engineer at an early-stage internet marketplace company... .... You will build production models that distinguish between closely related... ...Experience designing and running vision model evaluations that measure improvement beyond loss metrics...Clera
San Francisco, CA1 day ago- ...science. We build foundational understanding of models to advance the frontier of intelligence. About the role: As a ML Engineer, you'll build and operate the infrastructure... ...on the systems that support training and evaluating large models, scaling experimental pipelines...
$250k - $400k
...pharma. The challenge isn't just model capability. It's building the... ...these models to be trained, evaluated, and deployed reliably at... ...isolation. It's building the engine that research runs on. You'll... ...Experience building and scaling ML systems in production Strong background...Remote work- The role At Mach9, ML Engineers build the perception models at the core of our AI-enabled CAD system. We build models to extract 3D object and line features... ...ML model backs. Responsibilities Design, train, and evaluate computer vision and 3D ML models for extracting CAD-...
$180k - $300k
...The role At Mach9, ML Engineers build the perception models at the core of our AI-enabled CAD system. We build models to extract 3D object and line... ...end-to-end from data strategy, to model training, to evaluation. This role is ideal for early-career engineers (new...Full timeInternship- ...build, and deploy production‑grade ML systems with end‑to‑end ownership of the model lifecycle from conception to... ...of professional experience in ML engineering. Strong programming skills in Python... ...for model training and evaluation. Familiarity with FastAPI, OpenAI...Full time
$100 - $150 per hour
...criteria for data science deliverables, including exploratory data analyses , statistical modeling work , machine learning pipelines , and A/B test write-ups . Evaluate AI-generated or human-created work against established criteria to ensure quality and...Contract workSummer workRemote workMercor
San Francisco, CA4 days ago$141k - $249k
...and efficiency. - Comprehensively profile model runtime and memory to pinpoint performance bottlenecks. - Identify and evaluate emerging technologies that can be adopted... ...techniques. - Work with researchers and ML engineers on best-practices for optimal resource...Work at officeWork from homeFlexible hoursWaabi
San Francisco, CAa month ago$160k - $250k
...breaking speeds. About You and The Role As an ML Training & Inference Infrastructure Engineer on the Data Platform team you will be building... ...dataset management, distributed training, continuous model integration, evaluation, and deployment. The ideal candidate has strong...Local areaZipline
San Francisco, CA25 days ago- ...building the design suite for molecules, training frontier AI/ML models to advance drug discovery. You will develop core frameworks for model training and evaluation, in collaboration with researchers and engineers. Your role focuses on creating a scalable, reliable...
$207k - $244k
...Company.About the team/roleWe’re hiring an ML Engineer (P3) to build and ship the AI systems... ...services. Design, develop, and ship ML models and AI systems that Product Engineering... ...the room with alignment, not confusion.Evaluate and iterate fast. Build evaluation frameworks...Work at officeLocal areaRemote workRelocation3 days per weekCheckr
San Francisco, CA4 days ago$230k - $322k
...Design, build, and maintain scalable ML infrastructure to support notification... ...data ingestion, feature computation, model training, evaluation, deployment, and online serving in production... ...scientists, product managers, and engineers across teams to define requirements,...Full timeFor contractorsRemote workReddit
San Francisco, CA1 day ago$200k - $227k
...proven. We build specialized models for mission-critical trust and... ...create the benchmarks and evaluation harnesses that keep decision... ...managers, data scientists, software engineers, fraud intelligence, and... ...Together, you'll design and build ML solutions that have direct,...Work experience placementCasual workLive inWork at officeRemote workAirbnb
San Francisco, CA23 hours ago- ...Atlassian is seeking a Machine Learning Engineer to join our Growth organization. Growth... ...and product signals into scalable ML systems that support the growth funnel,... ...building reliable data foundations, features, models, evaluation frameworks, and decisioning systems...Temporary workWork at officeLocal area
Atlassian
San Francisco, CA4 days ago $118k - $176k
...and other digital services.Snap Engineering teams build fun and technically... ...Build and deploy machine learning models for core Snapchat productsApply established ML techniques to solve well-defined... ...data analysis, model training, evaluation, and production deploymentCollaborate...Full timeLive inWork at officeLocal areaSnap
San Francisco, CA1 day ago- ...looking for a Senior Machine Learning Engineer to join our Search & Intelligence organization... ...experiences, agentic systems, models, evaluation frameworks, and data pipelines that power... ...enterprise B2B environment.Experience with ML platforms, model serving, feature...Work at officeLocal areaImmediate startWorldwide
Atlassian
San Francisco, CA4 days ago - ...Responsibilities Senior Machine Learning Engineer — Agentic Search & Query... ...with product, search infrastructure, modeling, and evaluation teams. We combine applied research with... ...strengthen the team’s engineering and ML practices.Your backgroundOn the first...Work at officeLocal area
Atlassian
San Francisco, CA2 days ago - Title: ML Engineer Location: San Francisco, CA (Onsite) Direct HireCompany Mission Our client... ...create transparent, clinically grounded models. Its early adopters include health... ...with data engineers, clinicians, domain experts, and product teams to align models with...
Spectraforce Technologies
San Francisco, CA21 hours ago - ...Our client is a well-funded AI startup building production-grade ML infrastructure used by enterprise customers. They are looking for a Senior AI/ML Engineer to own model training pipelines, evaluation systems, and inference serving at scale. Full-time, on-site in San...Full time
Clera
San Francisco, CAa month ago - ...are now building our Silicon Valley engineering team — a small, senior group focused... ...reasoning loops. • Own prompt engineering, model versioning, evaluation (RAGAS, DeepEval), and LLMOps... ...or related field. • 3–5 years of AI/ML engineering; minimum 2 years building...Work at officeVisa sponsorship
Hop
San Francisco, CAa month ago $200k - $300k
...Description About the Role This is a founding-team ML / AI Engineer position at a consumer SaaS startup, where you... ...both real-time and batch workloads. Lead model selection, fine-tuning, embeddings, RAG pipelines, evaluation frameworks, and quality measurement....Remote workWintermeyer Ventures
San Francisco, CA10 days ago- ...technical owner to lead post-training projects for their audio models. The role requires translating customer needs into specifications, designing function calling capabilities, and performing evaluations for production use. Candidates should have hands-on experience in...
- ...Job Description - Senior AI/ML Engineer Location: Melbourne (hybrid, 3+ days a week in... ...robust Python solutions for benchmarking, model evaluation and data analysis • Design and... ...the direction of AI. • Become a world expert in AI : You will evaluate every major...Work at office3 days per week
Artificial Analysis
San Francisco, CA1 day ago - ...AI/ML Engineer (Computer Vision) A fast-growing applied AI company is expanding its engineering... ...to product decisions, and shipping models that improve how customers analyze... ...improve model pipelines across training, evaluation, inference, and monitoring. Partner with...
Blue Signal Search
San Francisco, CA1 day ago $84 per hour
.... Position: Atomistic & Surface Modeling Experts (Computational Materials & Catalysis)... ...simulation to build high-quality training and evaluation data. Review and evaluate AI-... ..., chemistry , physics , chemical engineering , or a related field. Clear...Contract workSummer workRemote workMercor
San Francisco, CA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to ML Engineer - Model Evaluation Expert. Be the first to apply!
Related searches
- senior ml engineer San Francisco, CA
- machine learning software engineer San Francisco, CA
- machine learning engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- ai ml engineer San Francisco, CA
- computer vision machine learning engineer San Francisco, CA
- internship machine learning San Francisco, CA
- machine learning San Francisco, CA
- artificial intelligence - machine learning intern San Francisco, CA
- machine learning scientist San Francisco, CA