Data Scientist - AI Evaluation
$100 - $150 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Data Scientist Talent Network
Type: Contract
Compensation: $100–$150/hour
Location: Remote
Role Responsibilities
- Design precise, task-specific grading criteria for data science deliverables, including exploratory data analyses , statistical modeling work , machine learning pipelines , and A/B test write-ups .
- Evaluate AI-generated or human-created work against established criteria to ensure quality and consistency.
- Provide detailed written justifications for evaluations and scores to maintain transparency and accountability.
- Apply consistent, evidence-based judgment to ensure assessments are reproducible and defensible.
- Incorporate structured feedback from senior reviewers and iterate on submitted work for continuous improvement.
- Work independently and asynchronously to meet deadlines and improve AI model performance .
Qualifications
Must-Have
- 1+ years of professional data science experience.
- Experience at a leading technology, research, or quantitative firm.
- Strong command of Python , SQL , statistical modeling , machine learning , and experimentation .
- Exceptional written communication skills.
- Detail-oriented and consistent approach to evaluating complex work.
- Comfort receiving feedback and calibrating judgment against established standards.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 12 days ago
Similar jobs that could be interesting for youBased on the Data Scientist - AI Evaluation in San Francisco, CA vacancy
- ...Mercor's mission is to organize human intelligence to power the AI economy. We partner with leading AI labs and enterprises to... ...offices. About the Role As a Senior Software Engineer (AI Data & Evaluation) at Mercor, you will be at the core of building the data...SuggestedFull timeWork at officeRelocation package
Mercor
San Francisco, CA18 hours ago $100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Summers , and Jack Dorsey . Position: Data Scientist Talent Network Type: Contract... ...pipelines , and technical reports. Evaluate AI-generated or human-created work against...SuggestedContract workSummer workRemote workMercor
San Francisco, CA6 days ago$240k - $280k
A leading software monitoring company is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure for measuring the performance of AI systems. This role involves designing datasets, creating benchmarks, and ensuring AI features behave reliably...Suggested- Mercor is partnering with a leading AI research lab to support a Frontier Code Agents project. You will help evaluate frontier AI coding models through structured technical assessments and focus on realistic data engineering workflows. The role involves reviewing model...Suggested
$238k - $302k
...simulation across 15+ U.S. states. The Large Model Evaluation team is at the nexus of Waymo’s AI ambition . With advancements in Large Language Models... ...that inform model development and deployment. Build data pipelines for signal discovery, data labeling, feature...SuggestedFull timeRemote workWaymo
San Francisco, CA18 hours ago$213k - $263k
...It blends deterministic logic, physical dynamics, and state-of-the-art Generative AI to create a training ground for the Waymo Driver. The Simulator Evaluation team faces the ultimate data challenge: How do you mathematically prove that a virtual world is "real"? We...Full timeRemote workWaymo
San Francisco, CA18 hours ago$238k - $302k
...Francisco, is seeking a skilled professional to lead methodologies for evaluating its autonomous driving technology. This hybrid role involves... ...) and have at least seven years of industry experience in data science. The position offers a competitive salary range of $238...- ...applying machine learning and data mining techniques to real problems... ....WHAT YOU'LL DOAs a Data Scientist II, you will collaborate with... ...impactful advanced analytics and AI solutions, optimize code, and... ...analysisDesign, implement, and evaluate models—from traditional...ApprenticeshipWork at officeLocal areaEasy work
$136.16k - $170.2k
...have the opportunity to thrive.Data Science is at the heart of... ...products and decision-making. Data Scientists at Lyft operate in dynamic... ...Lead complex Machine Learning, AI, and causal inference initiatives... ...offline/online metrics, evaluation frameworks, and A/B testing strategies...Hourly payTemporary workWork at officeLocal area3 days per week$150k - $185k
...Francisco, CaliforniaEngineering - Data Science & Analytics /US Full-... ...looking for a hands-on Data Scientist to own and deliver complex... ...-quality Python, building and evaluating models on large-scale viewership... ...use artificial intelligence (AI) tools to support parts of the...Full timeTemporary work$135k - $242k
...moment where seamless digital experiences and AI-powered innovation will empower customers... ...meaningful outcomes faster. The Principal Data Scientist will work within an Engineering team focused on building frameworks to evaluate AI Systems. This is a critical data science...Full timeFor contractors$170k - $178k
...translating designs into code, or iterating with AI. From idea to product, Figma empowers... ...us!We're looking for a research-minded Data Scientist to join the Core Data team. This team is... ...analytics, AI product measurement, or evaluating the impact of AI-powered featuresA self-...Minimum wageFull timeInternshipLocal areaRemote workFlexible hours$162.92k - $238.95k
...convert pipeline to revenue. Powered by AI, big data, and predictive analytics, our platform... ...'re looking for a Senior Data Scientist (IC4) to join our growing AI and Data Science... ...understanding of feature engineering, model evaluation, and MLOps.Ability to independently...Full time$130.1k - $165k
...for a problem solver who thrives at the intersection of data science, growth strategy, and AI-powered automation. You’ll focus on diving deep into... ...experimentation workflows, and surface intelligent recommendations.Evaluate and implement advanced statistical techniques -...Full timeTemporary workWork at officeLocal areaRelocation packageFlexible hours2 days per week$139.76k - $287.75k
...career you love? It’s Possible.At Pinterest, AI isn't just a feature, it's a powerful... ...performance marketers. We leverage massive data and cutting-edge science to automate and... ...outputsStrong track record of critical evaluation and verification of AI-assisted work (e.g...Work at officeLocal areaRelocationRelocation package$230k - $342k
About the RoleWe’re hiring a Data Scientist to support Real Estate & Workplace (REW), a fast-moving... ...execution. Working at the forefront of AI-native analytics, you’ll help define the... ...inference, forecasting, and model evaluation; fluent in SQL and Python.Proven ability...Work at officeLocal areaImmediate startFlexible hours$200k - $260k
...financial tools they need.The Data Science TeamOur ImpactData is core... ...our co-founders were PhD data scientists at Robinhood. Since our... ...FluencyRapidly learn and adopt emerging AI tools and workflows to accelerate analysisCritically evaluate AI-generated outputs for...Flexible hours$130k - $196.5k
LiveRamp is the data collaboration platform of choice for the world... ..., build experimentation and evaluation frameworks, and tests to quantify... ...Mentor and support other data scientists and analysts through code... ...example, BigQuery, Dataflow, Vertex AI, or similar).Familiarity with...Full timeWork from homeFlexible hoursNight shift$101.5k - $142k
...key investments in applying Generative AI and Machine Learning to some of the world... ...investigations, and litigations.Applied Scientists should have a pragmatic approach to problem... ...searchCreate and implement new metrics, evaluation pipelines, and internal training/test...Full timeWork at office$148.5k - $223.9k
...SalesforceSalesforce is the #1 AI CRM, where humans with agents... ...future of Salesforce.The TeamThe Data Science & Analytics team in... ...driven, and creative Senior Data Scientist to join the team. We explore,... ...our recruiters assess and evaluate candidates’ resumes and qualifications...Full time$139.76k - $287.75k
...career you love? It’s Possible.At Pinterest, AI isn't just a feature, it's a powerful... ...? It’s Possible.We are looking for a Data Scientist to join our marketing org. As a Data Scientist... ...and refine attribution methodologies to evaluate the effectiveness of marketing...Work at officeLocal areaRelocationRelocation package$263k
About the TeamOpenAI’s Agentic Data Science team helps shape how AI agents are built, deployed, and improved... ...new ways to measure security, evaluate defenses, and distinguish genuine risk... ...RoleWe are looking for a senior data scientist to help define what effective...Work at officeLocal areaRemote workFlexible hours- ...Analytics team owns a growing suite of internal AI agents and applications used daily by... ...applied AI product development with core data science, ML, and analytics work. You will... ..., internal product experiences, and the evaluation and observability systems that make AI...Work at office3 days per week
- ...Analytics team is looking for experienced Data Scientists and Senior Data Scientists to guide... ...data-supported wayNotice Regarding Use of AI and Automated Tools: To streamline our hiring... ...: Gem assists our recruiting team by evaluating job related qualifications and...Hourly payWork at officeLocal areaRemote workFlexible hours
$164.43k - $287.75k
...career you love? It’s Possible.At Pinterest, AI isn't just a feature, it's a powerful... ...process here.We are looking for Data Scientists to join our Monetization Engineering organization... ...between metrics change? How should we evaluate overall impact from changes in one...Local areaRelocationRelocation package$227.33k - $312.58k
We’re looking for a Staff ML Data Engineer to join Procore’s AI & Frontier Models organization. In this role, you’ll be responsible for designing and... .... This is an opportunity to shape how Procore builds, evaluates, and deploys frontier models by ensuring the underlying...Full timeWork at officeLocal areaImmediate start3 days per week$162k - $258.75k
...About the Principal Data Scientist at Headspace We are looking for an innovative, strategic... ...causal inference and statistical modeling to evaluate engagement, and long‑term member... ...the ROI dashboards. Drive new uses of AI in analytics and forecasting. Mentor...Full timeCurrently hiringWork at officeLocal areaRemote workShift work3 days per weekHuman Ventures, LLC.
San Francisco, CA4 days ago- ...Role As a Research Engineer, Distributed Data Systems, you will design and scale the... ...powers large-scale multimodal training and evaluation at OpenAI. You’ll manage distributed data... .... About OpenAI OpenAI is an AI research and deployment company dedicated...Full timeWork at officeRelocation package
OpenAI
San Francisco, CA18 hours ago $210k - $385k
A leading financial technology firm seeks a Data Scientist to architect evaluation pipelines and improve answer quality through innovative evaluations. Candidates should possess a PhD or MS in a technical field with at least 4 years of relevant experience, strong skills...- Cohere is looking for a Member of Technical Staff in Data Analysis and Evaluation to ensure the quality and performance of large language models. The role involves designing data collection tasks, collaborating with teams, and applying statistical methods for data evaluation...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Scientist - AI Evaluation. Be the first to apply!
Related searches
- data scientist (hedge fund) San Francisco, CA
- healthcare data scientist San Francisco, CA
- data scientist San Francisco, CA
- entry level data scientist San Francisco, CA
- part time data scientist San Francisco, CA
- ai data scientist San Francisco, CA
- senior data scientist San Francisco, CA
- principal data scientist San Francisco, CA
- entry level data scientist remote San Francisco, CA
- energy data scientist San Francisco, CA