AI Evaluation Expert - Rubric Specialist
$40 - $75 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: AI Evaluation Rubric Reviewer
Type: Contract
Compensation: $40–$75/hour
Location: Remote
Role Responsibilities
- Review AI-drafted tasks and evaluate model attempts for fairness and accuracy.
- Assess if the grading rubric aligns with the prompt and correct any unfair criteria.
- Flag tasks that cannot be saved and provide concise feedback on each evaluation.
- Analyze long agent transcripts to identify errors and improve AI model performance .
- Ensure rubric structures are correct per guidelines and enhance training data quality .
Qualifications
Must-Have
- Substantial experience writing or reviewing rubrics for LLM evaluations .
- Ability to read long agent transcripts and identify errors.
- Precise writing skills and a commitment to data verification.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
$24 per hour
...and QuickBooks experience to help improve AI responses to real small-business and... ...In this remote freelance project, you'll evaluate accuracy, identify gaps and help build reliable... .... Apply and help refine evaluation rubrics, flag recurring issues and participate in...SuggestedPart timeFreelanceRemote work$50 - $175 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Jack Dorsey . Position: JSON-fluent & Rubrics Expert Type: Contract Compensation:... ...hours/week Role Responsibilities Evaluate JSON-formatted AI model outputs against...SuggestedContract workSummer workStart working todayRemote work$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... .../ public administration Evaluator Type: Contract Compensation... ...domain-specific quality rubrics. Identify factual, aesthetic... ...Collaborate with subject matter experts to ensure consistency and quality...SuggestedContract workSummer workWork at officeRemote work- ...Create and evaluate board game tasks to enhance AI reasoning skills. About Turing Turing is one of the... ...We are seeking Board Game Reasoning Experts to help train and evaluate next-generation... ...quality. Develop prompts, rubrics, and evaluation guidelines for...SuggestedContract workPart timeFor contractorsFreelanceRemote work
$150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: General Clinician (MD/DO) - HLS Expert Pool Type: Contract Compensation... ...for consistent model response grading. Evaluate clinical dialogues for accuracy, safety...SuggestedContract workSummer workRemote work$60 - $70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Location: Remote Role Responsibilities Evaluate AI-generated responses for safety,... ...domains. Apply and refine evaluation rubrics for RLHF , SFT , and AI safety benchmarking...Contract workSummer workRemote work$90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...genuinely correct replications from those that merely look correct. Evaluate responsive behavior and semantic quality, ensuring proper use of...Contract workSummer workLocal areaRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...customer research and feedback synthesis Evaluator Type: Contract Compensation: $... ...artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation...Contract workSummer workWork at officeRemote work$70 - $120 per hour
...USA Join a leading AI lab's cutting-edge GenAI... ...Interior Visualization Specialist, an architect or senior... ...and material boards. Evaluate AI-generated plans,... ...Develop guidelines, rubrics and quality checks for... ...with other subject matter experts to ensure consistency and...Full timeContract workAfternoon shiftWeekday work$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Software Engineer Expert Type: Contract Compensation: $... ...Location: Remote Role Responsibilities Evaluate AI-generated artifacts for usability in...Contract workSummer workWork at officeRemote work$70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Summers , and Jack Dorsey . Position: Mathematics PhD Coding Experts Type: Contract Compensation: $70/hour Location:...Contract workSummer workImmediate startRemote work$145 per hour
...technical talent with leading AI research labs. Headquartered in... ...Position: ML Research PhD Experts (ICML / NeurIPS / ICLR Publications... ...Role Responsibilities Evaluate the accuracy and depth of AI-... ...performance. Develop evaluation rubrics and benchmarks for assessing...Remote jobContract workSummer work$150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Responsibilities Create and critique rubrics that analyze commercial drugs and drug-development... ...to meet deadlines while improving AI evaluation . Qualifications Must-Have US-...Hourly payContract workFor contractorsSummer workRemote work$60 - $85 per hour
...Frontier AI models can draft a contract clause or summarise... ...Frontier Lab is bringing in EU-law specialists to read the legal tasks its... ...those models are trained and evaluated. It is about 10 hours a week... ...glance Openings: 12 legal experts. Time commitment:...Hourly payFull timeContract workRemote work10 hours per weekFlexible hours$50 - $150 per hour
...hours / week Experience: 3 - 5 years Location: United States | Canada + 6 more We're hiring PhD-level experts to help train and evaluate advanced AI models. This is remote, flexible contract work where your academic expertise directly shapes how cutting-edge...Contract workTemporary workRemote workFlexible hours- ...Opportunity Our client, a leading AI research organization, is looking... ...architects and architectural visualization specialists to support AI training and evaluation. You’ll develop plan-to-... ...and building-code evaluation rubrics. Your expertise will help create useful...Afternoon shift
$70 - $120 per hour
...USA Join a leading AI lab's cutting-edge GenAI... ...Producer and Audio Post Specialist, a commercial composer... ...editable session formats. Evaluate AI-generated music,... ...Develop guidelines, rubrics and quality checks for... ...with other subject matter experts to ensure consistency...Full timeContract workWeekday work- ...Job Application for Italian Audio Evaluations Specialist - Freelance AI Trainer Project at Meridial Back to... ...accuracy relies entirely on robust, expert-crafted training data. The objective... ...structured prompts, complex evaluation rubrics, and technical guidelines....Hourly payPart timeFor contractorsFreelanceRemote work
$90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...justification. Preferred Prior RLHF , preference-labeling, model-evaluation, or structured code-review work. Pixel-perfect design-to-code...Contract workSummer workLocal areaRemote work$20 per hour
...Job Application for Search Engine Evaluation Specialist - Freelance AI Trainer Project at Meridial Back to... ...accuracy relies entirely on robust, expert-crafted training data. The objective... ...prompts, and clear, well-structured rubrics that generate clean, reliable data for...Hourly payPart timeFor contractorsFreelanceRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Healthcare / clinical Evaluator Type: Contract Compensation: $... ...against domain-specific quality rubrics. Identify factual, aesthetic, and presentation...Contract workSummer workWork at officeRemote work$70 - $100 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Computational Astrophysics & Cosmology Expert Type: Contract Compensation: $... ...Design challenging computational problems to evaluate AI capabilities in scientific software...Contract workSummer workRemote work$45 - $55 per hour
...technical talent with leading AI research labs. Headquartered in... ...Role Responsibilities Evaluate the accuracy and depth of AI-... ...performance. Develop evaluation rubrics and benchmarks for assessing... ...with subject matter experts to ensure consistency, relevance...Contract workSummer workRemote work$90 - $140 per hour
...technical talent with leading AI research labs.... ...Responsible AI / Security Specialist Type: Contract... ...responsible AI problems. Evaluate security and... ...violations. Develop rubrics to assess the safety, accuracy... ...other subject matter experts to ensure consistency...Contract workSummer workRemote workWeekday work$60 - $80 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco,... ...Remote Role Responsibilities Review expert-written tasks, including prompts, golden outputs, and rubrics. Evaluate the accuracy and depth of AI-generated content...Contract workSummer workRemote work$125 - $175 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Cybersecurity Practitioner: Paid Expert Interviews (SOC, Incident Response, Detection... ...in enterprise cyber defense work. Evaluate how AI assistance currently impacts your...Hourly payContract workSummer workRemote work$60 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...clear ground-truth outputs and objective rubrics. Evaluate AI models on visual document... .... Collaborate with subject matter experts to ensure task relevance and accuracy....Contract workSummer workRemote work$120 per hour
...Prolific, located in San Francisco, is looking for Licensed Pharmacists to train and evaluate AI models. You will guide these systems by reviewing AI-generated pharmaceutical scenarios and providing your expertise. With competitive pay rates of up to $120 USD per hour...Hourly payRemote workWork from homeFlexible hours$1,350 per month
...years Location: United States | Canada + 23 more Help evaluate how AI systems approach scientific and clinical work. Use your subject... ...per week for 1–6 months. Company: Senior Science & Clinical Experts - AI Benchmarking Location: Any time Apply here:...Temporary work- ...We're building a research cohort of senior domain experts across seven specialist tracks — software/systems engineering, formal methods and... ..., hardware design, and creative technology — to help evaluate how frontier AI models reason about hard, real-world expert-level...Temporary work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Expert - Rubric Specialist. Be the first to apply!
- technology expert San Francisco, CA
- fulfillment expert San Francisco, CA
- guest service support expert San Francisco, CA
- subject matter expert San Francisco, CA
- ai data scientist San Francisco, CA
- ai scientist San Francisco, CA
- subject matter expert physics
- water expert
- microsoft word expert
- samsung experience expert








