Data Science Expert - AI Evaluation
$120 - $170 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Data Science Expert
Type: Contract
Compensation: $120–$170/hour
Location: Remote
Role Responsibilities
- Design precise, task-specific grading criteria for real-world data science deliverables such as analyses, models, dashboards, and experiment readouts.
- Score AI-generated and human work samples against criteria with detailed written justifications for every score.
- Apply consistent, evidence-based judgment to ensure scores are reproducible and defensible.
- Incorporate structured feedback from senior reviewers and iterate quickly on your work.
- Work independently and asynchronously to meet deadlines while improving AI model performance .
Qualifications
Must-Have
- 5+ years of professional data science experience in industry.
- Background in business operations, product, or growth data science at top-tier technology companies.
- Deep fluency in experiment design and A/B testing, metric definition, SQL/Python analysis, and communicating findings to executive stakeholders.
- Exceptionally strong written communication.
- Detail-oriented, consistent, and comfortable having your judgment reviewed and calibrated against peers.
Preferred
- Prior experience with AI training , evaluation, or human-data projects.
Application Process (Takes 20–30 mins to complete)
- Submit your resume or relevant technical background to get started.
- Qualified applicants may be asked to complete a brief technical assessment or submit additional information.
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 7 days ago
Similar jobs that could be interesting for youBased on the Data Science Expert - AI Evaluation in New York, NY vacancy
- AuraOne is seeking an AI Data Annotation Expert for a remote contractor role focused on evaluating AI data annotation outputs against a versioned rubric. You will compare paired responses, label edge cases, and provide structured feedback to retrain the model. Responsibilities...SuggestedRemote jobFor contractors
$144.5k - $230k
...AI Data Specialist We're looking for a Data Specialist to join one of Accordion's AI... ...teams that combine AI engineering, data science, and product management to transform how... ...problems, from feature engineering through evaluation and deployment. Explore unfamiliar...SuggestedWork at officeLocal areaRemote workShift work2 days per weekAccordion USA
New York, NY5 days ago- ...Job Description Job Description Job Title: AI Data Science Domain Expert Job Type: Contractor (Part-Time) Location: Remote Job Overview... ...-generation AI systems. In this role, you will review, evaluate, and refine AI-generated technical and analytical content...SuggestedRemote jobPart timeFor contractors
$72k - $184.44k
...Description & SummaryThe OpportunityAs an AI Data Scientist Senior Associate, you will... ...throughout the analysis process- Applying data science workflows and troubleshooting techniques... ...preparation, experimentation, and model evaluation techniques to develop and validate AI/ML...SuggestedFull timeH1b- 24-MAG LLC is seeking a highly skilled data scientist or quantitative analyst for a full-time remote consulting role. You will design advanced... ..., and perform rigorous statistical comparisons for frontier AI benchmarks. The role emphasizes transparent reporting,...SuggestedRemote jobFull time
- Welo Data is hiring Data Labeling Associates in New York City for Project Perseus. The role involves evaluating Arabic language AI outputs and ensuring AI safety, requiring professional proficiency in Arabic and experience in writing and AI safety. You’ll critique models...
$100k - $160k
...Portfolio Management team, this role sits at the intersection of data, AI, and PM Infrastructure—driving the MAS data strategy while... ...quantitative, technical, or finance-related field (e.g., Computer Science, Data Science, Statistics, Economics, Engineering, or Mathematics...Full timeTemporary workPart time$26 - $28 per hour
...Description OverviewWelo Data is looking for detail-oriented and reliable individuals... ...Analysts, supporting speech and voice AI systems.This is a high-impact production... ...this role is more execution-focused than evaluation-heavy roles, it still requires strong judgment...Full timeRemote workVisa sponsorshipWelocalize
New York, NYmore than 2 months ago$80 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Summers , and Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type:... ...frontier AI coding agents to complete and evaluate complex data engineering tasks. Review...Contract workSummer workRemote work$30 per hour
...Human Intelligence Behind the World’s Leading AI – NYC Are you sharp, curious, and obsessed with the details? Welo Data is looking for Data Quality Analysts who... ...The Mission: Beyond Annotation Model Evaluation : Critique AI outputs and provide structured...Full timeTemporary workWork at officeWelo Data
New York, NYmore than 2 months ago$34 per hour
...Data Labeling Associate Welocalize is looking for sharp, curious, and detail-oriented individuals to join our team as Data... ...annotation role. You'll be working directly with cutting-edge AI systems evaluating outputs, identifying gaps, and helping improve how these...Full timeContract workRemote workVisa sponsorship- ...professionals for flexible, hourly contract work to support AI research. This ongoing project-based opportunity allows... ...without prior AI experience. The role involves creating expert-level training data and evaluating AI-generated responses, working independently on a...Remote jobHourly payContract workFlexible hours
- 1. Role Overview Mercor is partnering with a leading AI research organization to engage experienced data scientists for a project focused on evaluating how well AI systems perform real‑world data science work. Rather than producing deliverables yourself, you will define...
$195k - $275k
...industry experience to join a data analytics and artificial intelligence... ...capabilities in quantitative AI, combining advanced... ...statistical approaches to support evaluation, monitoring, and continuous improvement... ...Statistics, Physics, Computer Science, Engineering or related...Temporary work- SME Careers is looking for a SQL Engineer to work remotely, focusing on reviewing AI-generated SQL queries and improving AI models. In this role, you will create detailed prompts and evaluate AI responses for accuracy while ensuring model integrity. Candidates should have...Remote job
$77k - $202k
...SummaryAt PwC, our people in data and analytics focus on leveraging... ...data engineering, Generative AI, and advanced analytical... ...and emerging technologies to evaluate investment opportunities, assess... ...Bachelor's Degree in Computer Science, Data Processing/Analytics/Science...Full timeH1b- ...seeking a tech-driven analyst to join our data and technology team, reporting directly to... ...pipelines, and leading the integration of AI tools to optimize processes across our 4,5... ...graduate student, or recent graduate in Data Science, Business Analytics, Computer Science or...Full timeInternshipFlexible hours
Castle Park Investments, LLC
New York, NYmore than 2 months ago $76.6k - $126.3k
..., particularly the increasing data burden, by enhancing efficiency... ...), artificial intelligence (AI), and blockchain.Additionally,... ...haveA bachelor's degree in data science, information system, tax... ...enhance our clients' ability to evaluate risk and capitalize on opportunitiesWillingness...Summer holidayFlexible hours- ...An environmental consulting firm is looking for Environmental Scientists and Specialists to provide expertise in refining AI systems. This role involves analyzing various environmental datasets, contributing to risk assessments, and collaborating with technical teams....Remote workFlexible hours
Crossing Hurdles
New York, NY1 day ago $118.7k - $218.6k
...Summary Cyber Data Protection and PKI Specialist... ...as a subject matter expert and trusted advisor to... ..., assisting them to evaluate strategic and practical... ...Engineering, Computer Science, Information Technology... ...:Quantum readiness and AI protection experience a...Work experience placementLocal areaVisa sponsorship$135k - $170k
The Blackstone Group L.P. is seeking a Data Analyst, Associate specializing in Private Equity and Real Estate Technology based in New York. This role focuses on developing analytical tools to enhance asset management operations and involves collaboration with cross-functional...$124k - $177k
...landscape. Leverage cutting-edge technologies like Generative AI to increase productivity, streamline processes, and create seamless... ...team operates at the intersection of artificial intelligence, data science, and enterprise strategy. We are redefining how insights and...Local area3 days per week- OCHIN is a national nonprofit health IT organization enabling data-driven evaluation across programs. The Program Evaluation Analyst partners with leaders to design, implement, and report rapid cycle evaluations that guide strategic decisions and improve outcomes. The...Remote job
- Acorns is seeking a Data Analyst II to own measurement, analyses, and experimentation across the customer journey, measuring campaign... ...for communication touchpoints that drive engagement, and evaluating what’s working across product and lifecycle marketing. Every experiment...
- A leading AI company is looking for a Remote Business Analyst who excels in analytical tasks and is fluent in both English and Japanese. This role involves answering analytical questions and working with large language models to improve performance. Candidates should possess...Remote job
$90k - $112k
Jobright.ai is seeking a Junior Data Analyst to support translational research at Columbia University Irving Medical Center. The role involves designing assays, performing statistical analysis, and providing bioinformatic support. Candidates should have a Bachelor’s degree...- ...Audit teams. This role is a pipeline to a possible full-time position. The program emphasizes risk assessment, control testing, and presenting findings to management, with opportunities to apply AI/data analytics concepts and hands-on AI tools. #J-18808-Ljbffr Socket.devFull timeSummer work
- Garner Health, a fast-growing healthcare technology company in New York, seeks an AI-Native Data Analyst II to join the Reporting team. You will own client-facing analysis end-to-end, build scalable report automation, and partner with client teams to answer complex questions...Work at office
- ...Services) seeks an experienced JT&E analyst to join the Joint Test & Evaluation Support Cell. You will conduct quantitative and qualitative... ...survey methods to improve decision-making. The role emphasizes data-driven solutions for improving mission effectiveness, readiness...
- Garner Health is seeking an AI‑Native Data Analyst in New York City. You will own KPIs and dashboards end‑to‑end, partnering with stakeholders to define success metrics and build scalable data pipelines that power reliable reporting. You’ll develop domain expertise, identify...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Science Expert - AI Evaluation. Be the first to apply!
Related searches
- data specialist New York, NY
- data management specialist New York, NY
- provider data specialist New York, NY
- research data associate New York, NY
- data associate New York, NY
- data entry associate New York, NY
- data visualization specialist New York, NY
- data integration specialist New York, NY
- data management associate New York, NY
- data operations specialist New York, NY