Expert Talent Pool, Evals and Benchmarks
$75 - $175 per hourClera
ABOUT THE ROLE Join a network of senior evaluation experts supporting AI research projects. When projects match your experience, you may lead benchmark design, validation, and grading operations to help assess model performance rigorously. WHAT YOU'LL DO - Design benchmarks from the ground up, defining scope, question types, difficulty levels, and scoring methods. - Write and review rubrics to promote consistent grading and strong agreement across evaluators. - Develop gold-standard answers and test sets that can withstand expert review. - Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes. - Analyze model failures, identify root causes, and recommend improvements. - Design and conduct rigorous model comparison studies. - Validate benchmarks for contamination, ambiguous answers, and saturation. - Design user studies on how people rate and interact with model outputs. WHAT WE'RE LOOKING FOR - At least 5 years of experience in evaluation, testing, or assessment, or 5 years of domain expertise combined with hands-on evaluation work.
- Experience designing a benchmark, test plan, exam, or assessment used by others. - Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers. - Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo. - Experience analyzing model or test failures and recommending fixes. - Background in machine learning evaluation, QA, test engineering, psychometrics, assessment design, UX research, data science, or subject matter expertise. - Clear written English communication. Experience validating benchmarks, measuring grader agreement, running model comparisons, or conducting user studies is relevant. COMPENSATION & BENEFITS Compensation is $75 to $175 per hour, depending on project, field, and experience level. Payment is based on hours worked. LOCATION Fully remote. Projects may last from one week to several months.
$30 - $60 per hour
...Analyze incorrect answers, group error patterns, and document findings. - Compare multiple models using the same prompts. - Validate benchmarks and flag questions that may no longer provide reliable measurements. - Support small studies of how people evaluate or interact...SuggestedHourly payFull timeRemote work- ...About this opportunity This is a talent pool application, not a role with an immediate start date. We are building a vetted network... ...other legal professionals against those rubrics. This is expert, task-based work performed remotely and asynchronously. Requirements...SuggestedPart timeImmediate startRemote work
- Talent Pool: Financial Well-being Expert Remote Are you interested in joining us as a Freelance Financial Well-being expert, but don't see a specific role currently listed on our website? We are always looking to connect with talented professionals, so we've created a...SuggestedPermanent employmentContract workFreelanceLocal areaRemote work
$25 - $50 per hour
...ABOUT THE ROLE Join a flexible, project-based expert talent pool contributing to AI training work. You will write, edit, and evaluate examples that help AI models reason and respond effectively. WHAT YOU'LL DO - Write prompts and ideal responses for conversational AI...SuggestedHourly payFull timeRemote workFlexible hours$83.2k - $156k
...About the Role Join an expert annotation network supporting advanced AI research projects. You will work on complex, multimodal data labeling and help maintain high quality across projects. What You'll Do Label difficult or ambiguous examples across images, video...SuggestedHourly payFull timeRemote work$75 - $175 per hour
...ABOUT THE ROLE Join an expert pool supporting AI safety testing projects. You will lead evaluations across security, fairness, and domain-specific safety, helping identify and address risks before AI systems are deployed. WHAT YOU'LL DO - Design and lead red teaming campaigns...Hourly payFull timeRemote work$62.4k - $124.8k
...About the Role Join an expert network supporting AI model evaluation projects as a safety tester. You will identify and document ways AI systems can fail, helping improve model safety across projects matched to your background. What You'll Do Write prompts and...Hourly payFull timeRemote work- ...About this opportunity This is a talent pool application, not a role with an immediate start date. Our open pools cover the areas... ...produced by other professionals against those rubrics. This is expert, task-based work performed remotely and asynchronously. What...Part timeImmediate startRemote work
$20 - $45 per hour
...ABOUT THE ROLE Join a talent pool supporting AI model evaluation and improvement through careful, structured feedback. This flexible, project-based work helps improve the quality, accuracy, and safety of AI systems. WHAT YOU'LL DO - Compare and rank model responses to...Hourly payFull timeRemote workFlexible hours$25 - $50 per hour
...ABOUT THE ROLE Join a flexible, project-based expert talent pool supporting AI training work. You will create, edit, and evaluate examples that help AI models reason and respond more effectively, drawing on your writing, technical, or subject-matter expertise. WHAT YOU...Hourly payFull timeRemote workFlexible hours$100 - $140 per hour
...improve how well AI systems handle real questions of law across multiple jurisdictions. We keep an ongoing talent pool of qualified, practising lawyers who serve as expert reviewers on these projects. This is a rolling listing: we review applications on a continuous basis...Contract workRemote workFlexible hours$100 - $140 per hour
...Lawyers & Legal Experts — Global Talent Pool (UK, Germany, France, Spain, Australia... is a remote review track for evaluating AI outputs in legal review workflows. Reviewers grade citation accuracy, statutory reasoning, and policy adherence; flag risk; and document the...Remote jobFor contractors10 hours per week$150 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered... ...San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,... ...Position: General Clinician (MD/DO) - HLS Expert Pool Type: Contract Compensation:...Contract workSummer workRemote work$35 - $70 per hour
...ABOUT THE ROLE Join a paid talent pool supporting AI projects focused on agents and reinforcement learning. You will help build training environments and evaluate agent behavior, contributing to AI systems that interact with software and real-world workflows. WHAT YOU'...Hourly payFull timeRemote work- ...Project: Life Sciences Research & Benchmarking Study Role: Subject Matter Expert (SME) – Ecological, Evolutionary & Environmental Sciences Qualification: PhD in a relevant Life Sciences discipline Engagement: Project-Based / Research Assignment About the...Temporary work
- ...Position: Life Sciences Research & Benchmarking Expert Qualification: PhD in a relevant Life Sciences discipline Engagement: Project-Based / Research Evaluation About the Project We are seeking highly qualified PhD professionals in Life Sciences with strong...Temporary work
$61 - $77 per hour
...Mercor connects elite creative and technical talent with leading AI research labs.... ...in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,... ...undergraduate), Hard (advanced undergraduate), or Expert (post-graduate and above). Provide 1...Contract workSummer workRemote work$40 - $90 per hour
ABOUT THE ROLE Lead physical AI data capture programs that help robots learn from human movement and manipulation. You will set up capture systems, guide demonstrations, and uphold data quality across projects ranging from one week to several months. WHAT YOU'LL DO - Set...Hourly payFull timeRemote work$61 - $77 per hour
...Mercor connects elite creative and technical talent with leading AI research labs.... ...in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,... ...question's difficulty as Medium, Hard, or Expert, providing 1 correct answer and 9 plausible...Contract workSummer workRemote work$60 - $150 per hour
...ABOUT THE ROLE Join a remote expert contributor pool supporting AI training projects. You will create and review high-quality examples that help AI models reason and respond accurately, while contributing expertise across writing, language, engineering, and specialized...Hourly payFull timeRemote work$50 - $120 per hour
...ABOUT THE ROLE Join a remote expert network as a senior rater and reviewer, evaluating complex AI model outputs and helping calibrate other graders. Your work supports the development of high-quality, trustworthy AI systems across specialized fields. WHAT YOU'LL DO -...Hourly payFull timeRemote work$80 per hour
Responsibilities Design reinforcement learning environments with Python, Docker, state spaces, tool integrations, and scoring mechanisms. Build robust verifiers and reward functions that resist agent gaming. Translate ERP, CRM, finance, and operations workflows...Hourly payFull timeRemote work$160 per hour
...Job Description About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack...Remote jobContract workSummer work- Sun Recruiting, Inc. is seeking a Talent Acquisition Advisor to drive high-touch candidate experiences and strong partnerships with hiring... ...across technical and manufacturing roles. You will build talent pools, execute on campus recruitment, and attend events to brand our...Remote job
- Mercor is seeking an HR Expert for a remote contract role. You will lead HR scenarios across talent acquisition, workforce planning, employee relations, and compensation design, using tools like Workday and SAP SuccessFactors. You will own HR programs, ensure compliance...Remote jobContract work
- Prolific is seeking HR and Talent Professionals in San Jose, California to join their Expert Network. Responsibilities include evaluating AI-generated content and creating HR scenarios. Ideal candidates will have professional HR experience, strong compliance expertise,...Remote jobHourly payWork from homeFlexible hours
$15 - $35 per hour
...ABOUT THE ROLE Join an expert network supporting AI training projects as an annotation specialist. You will label a variety of data types to help improve how AI systems learn, with project matching based on your background. WHAT YOU'LL DO - Classify images, video, audio...Hourly payFull timeRemote work$45 - $90 per hour
...ABOUT THE ROLE Lead data operations for projects supporting advanced AI development. As part of a flexible expert network, you will be matched with projects that fit your background and work directly with project teams to help ensure the quality of datasets used to train...Hourly payFull timeRemote workFlexible hours$85 per hour
...Mercor connects elite creative and technical talent with leading AI research labs.... ...in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,... ...Position: Fab Safety and Emergency Response Expert Type: Contract Compensation: $...Contract workSummer workRemote work- ...Microsocopy team as the scientific authority on ground truth and to design tasks and datasets behind AI benchmarks in cell biology, tissue, and disease. You will determine how expert reasoning is defined and ensure AI agents can distinguish true biology from artifacts in...Remote job
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Expert Talent Pool, Evals and Benchmarks. Be the first to apply!



