Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Expert - Rubric Specialist

$40 - $75 per hour

Mercor

Job Description

Job Description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: AI Evaluation Rubric Reviewer
Type: Contract
Compensation: $40–$75/hour
Location: Remote

Role Responsibilities

  • Review AI-drafted tasks and evaluate model attempts for fairness and accuracy.
  • Assess if the grading rubric aligns with the prompt and correct any unfair criteria.
  • Flag tasks that cannot be saved and provide concise feedback on each evaluation.
  • Analyze long agent transcripts to identify errors and improve AI model performance .
  • Ensure rubric structures are correct per guidelines and enhance training data quality .

Qualifications

Must-Have

  • Substantial experience writing or reviewing rubrics for LLM evaluations .
  • Ability to read long agent transcripts and identify errors.
  • Precise writing skills and a commitment to data verification.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Evaluation Expert - Rubric Specialist in San Francisco, CA vacancy
  • $24 per hour

     ...and QuickBooks experience to help improve AI responses to real small-business and...  ...In this remote freelance project, you'll evaluate accuracy, identify gaps and help build reliable...  .... Apply and help refine evaluation rubrics, flag recurring issues and participate in... 
    Suggested
    Part time
    Freelance
    Remote work

    Braintrust

    San Francisco, CA
    19 hours ago
  • $50 - $175 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Jack Dorsey . Position: JSON-fluent & Rubrics Expert Type: Contract Compensation:...  ...hours/week Role Responsibilities Evaluate JSON-formatted AI model outputs against... 
    Suggested
    Contract work
    Summer work
    Start working today
    Remote work

    Mercor

    San Francisco, CA
    7 days ago
  • $80 - $120 per hour

     ...technical talent with leading AI research labs. Headquartered in...  .../ public administration Evaluator Type: Contract Compensation...  ...domain-specific quality rubrics. Identify factual, aesthetic...  ...Collaborate with subject matter experts to ensure consistency and quality... 
    Suggested
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    San Francisco, CA
    29 days ago
  •  ...Create and evaluate board game tasks to enhance AI reasoning skills. About Turing Turing is one of the...  ...We are seeking Board Game Reasoning Experts to help train and evaluate next-generation...  ...quality. Develop prompts, rubrics, and evaluation guidelines for... 
    Suggested
    Contract work
    Part time
    For contractors
    Freelance
    Remote work

    Turing

    San Francisco, CA
    a month ago
  • $150 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Position: General Clinician (MD/DO) - HLS Expert Pool Type: Contract Compensation...  ...for consistent model response grading. Evaluate clinical dialogues for accuracy, safety... 
    Suggested
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    10 days ago
  • $60 - $70 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Location: Remote Role Responsibilities Evaluate AI-generated responses for safety,...  ...domains. Apply and refine evaluation rubrics for RLHF , SFT , and AI safety benchmarking... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    24 days ago
  • $90 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...genuinely correct replications from those that merely look correct. Evaluate responsive behavior and semantic quality, ensuring proper use of... 
    Contract work
    Summer work
    Local area
    Remote work

    Mercor

    San Francisco, CA
    a month ago
  • $80 - $120 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...customer research and feedback synthesis Evaluator Type: Contract Compensation: $...  ...artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation... 
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    San Francisco, CA
    4 days ago
  • $70 - $120 per hour

     ...USA Join a leading AI lab's cutting-edge GenAI...  ...Interior Visualization Specialist, an architect or senior...  ...and material boards. Evaluate AI-generated plans,...  ...Develop guidelines, rubrics and quality checks for...  ...with other subject matter experts to ensure consistency and... 
    Full time
    Contract work
    Afternoon shift
    Weekday work

    Mercor

    San Francisco, CA
    12 days ago
  • $100 - $150 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Dorsey . Position: Software Engineer Expert Type: Contract Compensation: $...  ...Location: Remote Role Responsibilities Evaluate AI-generated artifacts for usability in... 
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    San Francisco, CA
    26 days ago
  • $70 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Summers , and Jack Dorsey . Position: Mathematics PhD Coding Experts Type: Contract Compensation: $70/hour Location:... 
    Contract work
    Summer work
    Immediate start
    Remote work

    Mercor

    San Francisco, CA
    25 days ago
  • $145 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Position: ML Research PhD Experts (ICML / NeurIPS / ICLR Publications...  ...Role Responsibilities Evaluate the accuracy and depth of AI-...  ...performance. Develop evaluation rubrics and benchmarks for assessing... 
    Remote job
    Contract work
    Summer work

    Mercor

    San Francisco, CA
    9 days ago
  • $150 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Responsibilities Create and critique rubrics that analyze commercial drugs and drug-development...  ...to meet deadlines while improving AI evaluation . Qualifications Must-Have US-... 
    Hourly pay
    Contract work
    For contractors
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    2 days ago
  • $60 - $85 per hour

     ...Frontier AI models can draft a contract clause or summarise...  ...Frontier Lab is bringing in EU-law specialists to read the legal tasks its...  ...those models are trained and evaluated. It is about 10 hours a week...  ...glance Openings: 12 legal experts. Time commitment:... 
    Hourly pay
    Full time
    Contract work
    Remote work
    10 hours per week
    Flexible hours

    Wintermeyer Ventures

    San Francisco, CA
    10 hours ago
  • $50 - $150 per hour

     ...hours / week Experience: 3 - 5 years Location: United States | Canada + 6 more We're hiring PhD-level experts to help train and evaluate advanced AI models. This is remote, flexible contract work where your academic expertise directly shapes how cutting-edge... 
    Contract work
    Temporary work
    Remote work
    Flexible hours

    Braintrust

    San Francisco, CA
    a month ago
  •  ...Opportunity Our client, a leading AI research organization, is looking...  ...architects and architectural visualization specialists to support AI training and evaluation. You’ll develop plan-to-...  ...and building-code evaluation rubrics. Your expertise will help create useful... 
    Afternoon shift

    Braintrust

    San Francisco, CA
    17 hours ago
  • $70 - $120 per hour

     ...USA Join a leading AI lab's cutting-edge GenAI...  ...Producer and Audio Post Specialist, a commercial composer...  ...editable session formats. Evaluate AI-generated music,...  ...Develop guidelines, rubrics and quality checks for...  ...with other subject matter experts to ensure consistency... 
    Full time
    Contract work
    Weekday work

    Mercor

    San Francisco, CA
    12 days ago
  •  ...Job Application for Italian Audio Evaluations Specialist - Freelance AI Trainer Project at Meridial Back to...  ...accuracy relies entirely on robust, expert-crafted training data. The objective...  ...structured prompts, complex evaluation rubrics, and technical guidelines.... 
    Hourly pay
    Part time
    For contractors
    Freelance
    Remote work

    Invisible Technologies

    San Francisco, CA
    a month ago
  • $90 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...justification. Preferred Prior RLHF , preference-labeling, model-evaluation, or structured code-review work. Pixel-perfect design-to-code... 
    Contract work
    Summer work
    Local area
    Remote work

    Mercor

    San Francisco, CA
    1 day ago
  • $20 per hour

     ...Job Application for Search Engine Evaluation Specialist - Freelance AI Trainer Project at Meridial Back to...  ...accuracy relies entirely on robust, expert-crafted training data. The objective...  ...prompts, and clear, well-structured rubrics that generate clean, reliable data for... 
    Hourly pay
    Part time
    For contractors
    Freelance
    Remote work

    Invisible Technologies

    San Francisco, CA
    a month ago
  • $80 - $120 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Dorsey . Position: Healthcare / clinical Evaluator Type: Contract Compensation: $...  ...against domain-specific quality rubrics. Identify factual, aesthetic, and presentation... 
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    San Francisco, CA
    12 days ago
  • $70 - $100 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Computational Astrophysics & Cosmology Expert Type: Contract Compensation: $...  ...Design challenging computational problems to evaluate AI capabilities in scientific software... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    a month ago
  • $45 - $55 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Role Responsibilities Evaluate the accuracy and depth of AI-...  ...performance. Develop evaluation rubrics and benchmarks for assessing...  ...with subject matter experts to ensure consistency, relevance... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    5 days ago
  • $90 - $140 per hour

     ...technical talent with leading AI research labs....  ...Responsible AI / Security Specialist Type: Contract...  ...responsible AI problems. Evaluate security and...  ...violations. Develop rubrics to assess the safety, accuracy...  ...other subject matter experts to ensure consistency... 
    Contract work
    Summer work
    Remote work
    Weekday work

    Mercor

    San Francisco, CA
    8 days ago
  • $60 - $80 per hour

     ...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco,...  ...Remote Role Responsibilities Review expert-written tasks, including prompts, golden outputs, and rubrics. Evaluate the accuracy and depth of AI-generated content... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    22 days ago
  • $125 - $175 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Position: Cybersecurity Practitioner: Paid Expert Interviews (SOC, Incident Response, Detection...  ...in enterprise cyber defense work. Evaluate how AI assistance currently impacts your... 
    Hourly pay
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    26 days ago
  • $60 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...clear ground-truth outputs and objective rubrics. Evaluate AI models on visual document...  .... Collaborate with subject matter experts to ensure task relevance and accuracy.... 
    Contract work
    Summer work
    Remote work

    Mercor

    San Francisco, CA
    1 day ago
  • $120 per hour

     ...Prolific, located in San Francisco, is looking for Licensed Pharmacists to train and evaluate AI models. You will guide these systems by reviewing AI-generated pharmaceutical scenarios and providing your expertise. With competitive pay rates of up to $120 USD per hour... 
    Hourly pay
    Remote work
    Work from home
    Flexible hours

    Prolific

    San Francisco, CA
    4 days ago
  • $1,350 per month

     ...years Location: United States | Canada + 23 more Help evaluate how AI systems approach scientific and clinical work. Use your subject...  ...per week for 1–6 months. Company: Senior Science & Clinical Experts - AI Benchmarking Location: Any time Apply here:... 
    Temporary work

    Braintrust

    San Francisco, CA
    7 days ago
  •  ...We're building a research cohort of senior domain experts across seven specialist tracks — software/systems engineering, formal methods and...  ..., hardware design, and creative technology — to help evaluate how frontier AI models reason about hard, real-world expert-level... 
    Temporary work

    AfterQuery

    San Francisco, CA
    a month ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Expert - Rubric Specialist. Be the first to apply!