Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Expert Talent Pool, Evals and Benchmarks (Associate)

$30 - $60 per hour
Full-time

Clera

ABOUT THE ROLE Join a specialist talent pool supporting human-data projects that evaluate AI models. You will help design and run assessments that measure model capabilities and contribute to AI safety and capability research. WHAT YOU'LL DO - Write questions and tasks that isolate and test specific model capabilities. - Develop rubrics that define strong, partial, and poor answers. - Create reference answers and edge cases to challenge AI models. - Grade model outputs against rubrics and explain scoring decisions. - Analyze failed answers, group error patterns, and document findings. - Compare multiple models using identical prompts. - Validate benchmarks and flag compromised or ineffective questions. - Participate in small user studies evaluating ratings or interactions with model outputs.

WHAT WE'RE LOOKING FOR - Zero to five years of professional experience. - Experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or relevant subject-matter work involving question writing. - Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria. - Comfort using spreadsheets or Python to organize and analyze results, and identifying and documenting error patterns. - Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo is useful. - Clear written communication and the ability to compare AI model outputs or participate in user and rating studies. COMPENSATION & BENEFITS Compensation is $30 to $60 per hour, depending on project, field, and experience. Payment is for hours worked. LOCATION Fully remote. Projects typically last from one week to several months.

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Expert Talent Pool, Evals and Benchmarks (Associate) in Remote vacancy
  • $75 - $175 per hour

     ...ABOUT THE ROLE Join a network of senior evaluation experts supporting AI research projects. You will lead benchmark design, validation, and grading operations when projects align with your expertise, helping produce rigorous and reliable assessments of AI models. WHAT... 
    Suggested
    Hourly pay
    Remote work

    Clera

    Remote
    2 days ago
  • $62.4k - $124.8k

     ...About the Role Join an expert network supporting AI model evaluation projects as a safety tester. You will identify and document ways AI systems can fail, helping improve model safety across projects matched to your background. What You'll Do Write prompts and... 
    Suggested
    Hourly pay
    Full time
    Remote work

    Clera

    Remote
    1 day ago
  • $25 - $50 per hour

     ...ABOUT THE ROLE Join a flexible, project-based expert talent pool contributing to AI training work. You will write, edit, and evaluate examples that help AI models reason and respond effectively. WHAT YOU'LL DO - Write prompts and ideal responses for conversational AI... 
    Suggested
    Hourly pay
    Remote work
    Flexible hours

    Clera

    Remote
    2 days ago
  • $35 - $70 per hour

     ...ABOUT THE ROLE Join a paid talent pool supporting projects in AI agents and reinforcement learning. You will help build practice environments and evaluate agent behavior, supporting the development of AI systems that interact with software and workflows. WHAT YOU'LL DO... 
    Suggested
    Hourly pay
    Remote work

    Clera

    Remote
    2 days ago
  • $20 - $45 per hour

     ...ABOUT THE ROLE Join a project-based expert network supporting AI model evaluation and improvement. You will provide structured feedback that helps make model responses more accurate, helpful, and safe. WHAT YOU'LL DO - Rank and compare model responses to identify the... 
    Suggested
    Hourly pay
    Remote work

    Clera

    Remote
    2 days ago
  • $15 - $35 per hour

     ...ABOUT THE ROLE Join an expert network supporting AI training projects as an annotation specialist. You will label diverse data types to help improve how AI systems learn, with project assignments matched to your background. WHAT YOU'LL DO - Classify images, video, audio... 
    Hourly pay
    Remote work

    Clera

    Remote
    2 days ago
  • $17 - $22 per hour

     ...Associate Patient Access Specialist Help patients get the care they need with patience, clarity, and compassion. As an Associate Patient...  ...same hours, this job won't be the best match. About Our Talent Pool Hummingbird is growing fast, and we interview year-round... 
    Hourly pay
    Full time
    Work experience placement
    Remote work
    Monday to Friday

    Hummingbird

    United States
    1 day ago
  • Talent Pool: Financial Well-being Expert Remote Are you interested in joining us as a Freelance Financial Well-being expert, but don't see a specific role currently listed on our website? We are always looking to connect with talented professionals, so we've created a... 
    Permanent employment
    Contract work
    Freelance
    Local area
    Remote work

    OpenUp

    New York, NY
    3 days ago
  •  ...Job Description Job Description Are you a talented professional with experience in medical devices , diagnostics , or biotech ?...  ...of advancing health through innovation. By joining our Talent Pool , you’ll be among the first we contact when new roles open up in... 
    Work at office
    Remote work
    1 day per week

    T45 Labs

    Santa Clara, CA
    a month ago
  • $83.2k - $156k

     ...About the Role Join an expert annotation network supporting advanced AI research projects. You will work on complex, multimodal data labeling and help maintain high quality across projects. What You'll Do Label difficult or ambiguous examples across images, video... 
    Hourly pay
    Full time
    Remote work

    Clera

    Remote
    1 day ago
  • $61 - $77 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs....  ...in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,...  ...question's difficulty as Medium, Hard, or Expert, providing 1 correct answer and 9 plausible... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    26 days ago
  • $61 - $77 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs....  ...in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,...  ...undergraduate), Hard (advanced undergraduate), or Expert (post-graduate and above). Provide 1... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    5 days ago
  • $150 per hour

     ...connects elite creative and technical talent with leading AI research labs. Headquartered...  ...San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,...  ...Position: General Clinician (MD/DO) - HLS Expert Pool Type: Contract Compensation:... 
    Contract work
    Summer work
    Remote work

    Mercor

    Nashville, TN
    4 days ago
  •  ...About this opportunity This is a talent pool application, not a role with an immediate start date. Our open pools cover the areas...  ...produced by other professionals against those rubrics. This is expert, task-based work performed remotely and asynchronously. What... 
    Part time
    Immediate start
    Remote work

    Vetto

    Menlo Park, CA
    a month ago
  •  ...About this opportunity This is a talent pool application, not a role with an immediate start date. We are building a vetted network...  ...other legal professionals against those rubrics. This is expert, task-based work performed remotely and asynchronously. Requirements... 
    Part time
    Immediate start
    Remote work

    Vetto

    Menlo Park, CA
    a month ago
  • $61 - $77 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs....  ...in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,...  ...question's difficulty as Medium, Hard, or Expert, providing 1 correct answer and 9 plausible... 
    Contract work
    Summer work
    Remote work

    Mercor

    Houston, TX
    26 days ago
  • $100 - $140 per hour

     ...improve how well AI systems handle real questions of law across multiple jurisdictions. We keep an ongoing talent pool of qualified, practising lawyers who serve as expert reviewers on these projects. This is a rolling listing: we review applications on a continuous basis... 
    Contract work
    Remote work
    Flexible hours

    Pearson

    Remote
    8 days ago
  • $100 - $140 per hour

     ...Lawyers & Legal Experts — Global Talent Pool (UK, Germany, France, Spain, Australia... is a remote review track for evaluating AI outputs in legal...  ...policy reviewers to grade AI outputs the way a senior associate would — with citations, severity tags, and the corrected... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    7 days ago
  •  ...Associate Vice President, Talent About the Company Highly rated community bank offering personalized financial services in North Alabama...  ...hiring, maintains job descriptions, coordinates compensation benchmarking and incentive program design, and analyzes workforce... 
    Work at office

    Confidential

    Muscle Shoals, AL
    4 days ago
  • $160 per hour

     ...Job Description About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack... 
    Remote job
    Contract work
    Summer work

    Mercor

    New York, NY
    14 days ago
  •  ...Project: Life Sciences Research & Benchmarking Study Role: Subject Matter Expert (SME) – Ecological, Evolutionary & Environmental Sciences Qualification: PhD in a relevant Life Sciences discipline Engagement: Project-Based / Research Assignment About the... 
    Temporary work

    Truelancer AI

    Remote
    10 days ago
  •  ...Position: Life Sciences Research & Benchmarking Expert Qualification: PhD in a relevant Life Sciences discipline Engagement: Project-Based / Research Evaluation About the Project We are seeking highly qualified PhD professionals in Life Sciences with strong... 
    Temporary work

    Truelancer AI

    Remote
    10 days ago
  • $45 - $90 per hour

     ...planning, policy and employee lifecycle, that test AI systems. You also judge how well models handle them. What you will do Create HR benchmark tasks with expected answers Review AI outputs on HR policy and operations Write clear rationales Keep tasks realistic and current... 
    Hourly pay
    Weekly pay
    Contract work
    Part time
    For contractors
    Remote work
    Flexible hours

    Vettedlance

    New York, NY
    4 days ago
  • $71.93k

    Online Talent Development Expert Recruitment Firm: Naukri Mitra Job Title: Online Talent Development Expert Location: Remote Annual Salary: $71,927 About the Role: Naukri Mitra is excited to announce an opportunity for an experienced and innovative Online Talent Development... 
    Remote work
    Worldwide

    Naukri Mitra

    New York, NY
    4 days ago
  •  ...personalized outreach to senior professionals. You will identify suitable candidates on LinkedIn, XING and company sites, build talent pools, and coordinate interviews while leveraging AI tools like ChatGPT to improve search methods and messaging. Fluency in German (C1... 
    Remote job

    Team Up

    New York, NY
    2 days ago
  • Prolific Academic Ltd invites HR and Talent Acquisition professionals to join its Expert Network to train and evaluate AI models with real-world HR expertise. You will assess AI-generated HR content, create realistic HR scenarios, and annotate data to improve model accuracy... 
    Remote job
    Work from home
    Flexible hours

    Prolific Academic Ltd

    New York, NY
    2 days ago
  •  ...Tuesdays, Wednesdays and a third day unique to each team or employee). The Impact you will have in this role: Being a member of the Talent Acquisition team, you will be a steward of DTCC’s greatest asset - our people. The Sr Sourcer partners closely with Recruiters to... 
    Full time
    Work at office
    Remote work
    Flexible hours

    DTCC

    Jersey City, NJ
    4 days ago
  • Rockstar is a US-based remote staffing company seeking a Talent Acquisition Specialist with hands-on medical billing experience to join our internal team. You will source and screen billing professionals, build a steady candidate pipeline, and support partner practices... 
    Remote job

    Rockstar

    Phoenix, AZ
    15 hours ago
  • Prolific is seeking HR and Talent Professionals in San Jose, California to join their Expert Network. Responsibilities include evaluating AI-generated content and creating HR scenarios. Ideal candidates will have professional HR experience, strong compliance expertise,... 
    Remote job
    Hourly pay
    Work from home
    Flexible hours

    Prolific

    San Jose, CA
    15 hours ago
  • Full-time remote role blending medical billing expertise with talent acquisition. Supports U.S. therapy practices by hiring skilled RCM professionals. Key Responsibilities: Source and screen candidates using hands-on billing knowledge. Assess proficiency via live calls... 
    Remote job
    Full time

    Jobgether

    New York, NY
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Expert Talent Pool, Evals and Benchmarks (Associate). Be the first to apply!