Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Rubric Reviewer - Fully Remote | Up to $75/hr

$75 per hour

Mercor

New York, NY
  • Remote job

Job Description

Job Description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: AI Evaluation Rubric Reviewer
Type: Contract
Compensation: $40–$75/hour
Location: Remote

Role Responsibilities

  • Review AI-drafted tasks and evaluate model attempts for fairness and accuracy.
  • Assess if the grading rubric aligns with the prompt and correct any unfair criteria.
  • Flag tasks that cannot be saved and provide concise feedback on each evaluation.
  • Analyze long agent transcripts to identify errors and improve AI model performance .
  • Ensure rubric structures are correct per guidelines and enhance training data quality .

Qualifications

Must-Have

  • Substantial experience writing or reviewing rubrics for LLM evaluations .
  • Ability to read long agent transcripts and identify errors.
  • Precise writing skills and a commitment to data verification.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Rubric Reviewer - Fully Remote | Up to $75/hr in New York, NY vacancy
  • $40 - $75 per hour

     ...AI Evaluation Rubric Reviewer is a remote evaluation track for reviewing ai evaluation rubric evaluation prompts and responses against AuraOne's quality...  ...be authorized to work from US. Compensation $40–$75 / hr Application process Apply through AuraOne's specialist... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    2 days ago
  • AuraOne, Inc. is seeking an AI Agent Interaction Specialist to remotely review ai agent interaction evaluation prompts and responses against AuraOne's quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for the modeling team... 
    Remote work
    10 hours per week

    AuraOne Human Data

    New York, NY
    2 days ago
  • $50 per hour

    prolificacademicltd is seeking a UX design reviewer to contribute professional product design and UX expertise to paid tasks that train and evaluate AI models. After a short skills assessment, successful applicants may join a network of independent participants and review... 
    Remote job
    Flexible hours

    prolificacademicltd

    New York, NY
    3 days ago
  • $18 per hour

    Classifieds24.info is seeking remote workers to help review AI-generated content, including voice models, images and videos. You will provide feedback to improve AI performance and ensure content meets project requirements. Strong written and verbal communication, attention... 
    Remote job
    Weekly pay
    Flexible hours

    Classifieds24.info

    Toms River, NJ
    2 days ago
  • $18 per hour

    Adver.info is seeking remote assistants to review and help improve AI systems. In this role you will interact with AI voice models and review AI-generated images and videos, providing feedback to enhance accuracy and usefulness. This is a flexible, remote, gig-style opportunity... 
    Remote job
    Hourly pay
    Weekly pay
    Flexible hours

    Adver.info

    Detroit, MI
    2 days ago
  • $55 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Wholesale and Technical Sales Reviewer Type: Contract...  ...$50–$55/hour Location: Remote Role Responsibilities Evaluate...  ...performance. Develop evaluation rubrics and benchmarks for assessing... 
    Remote work
    Contract work
    Summer work

    Mercor

    New York, NY
    24 days ago
  • $18 per hour

    Globalclassifieds.info is seeking individuals to review AI content remotely. You will interact with AI voice models, review images and videos, and provide feedback to improve AI performance. The role offers remote work with flexible hours and weekly payments. Compensation... 
    Remote job
    Hourly pay
    Weekly pay
    Flexible hours

    Globalclassifieds.info

    Lakewood, CO
    1 day ago
  • $18 per hour

    Globalclassifieds is offering remote gigs to review AI content, including voice models and generated images/videos. Earn up to $18 per hour, with flexible scheduling and weekly payments. Candidates should demonstrate strong communication and attention to detail, and adapt... 
    Remote job
    Hourly pay
    Weekly pay
    Flexible hours

    Globalclassifieds

    Lakewood, CO
    3 days ago
  • $20 - $45 per hour

    Quant STEM Bachelor AI Training Reviewer (English) Review homework...  ...accuracy and reasoning. This remote, part-time contractor...  ...Hourly · $20-$45/hr $20-$45/hr Contract,...  ...-user perspective in a fully remote, 16-week US project...  ...remote, flexible role pays $75 per hour and suits... 
    Remote work
    Hourly pay
    Contract work
    Part time
    For contractors
    10 hours per week
    Flexible hours

    OpenTrain AI, Inc.

    New York, NY
    1 day ago
  •  ...HR Policy Workflow AI Reviewer is a remote review track for evaluating AI outputs in policy review workflows. Reviewers grade citation accuracy, statutory...  ...on real matters. Comfort applying multi-page rubrics consistently across long batches. Clear written reasoning... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    8 days ago
  • $90 per hour

     ...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco...  .... Position: Finance & Insurance Practice Reviewer Type: Contract Compensation: $75–$90/hour Location: Remote Role Responsibilities Evaluate AI-... 
    Remote job
    Contract work
    Summer work
    Flexible hours

    Mercor

    New York, NY
    26 days ago
  • $80 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Position: Vision Verticals Reviewer Type: Contract Compensation...  ...$60–$80/hour Location: Remote Role Responsibilities...  ...prompts, golden outputs, and rubrics. Evaluate the accuracy and... 
    Remote job
    Contract work
    Summer work

    Mercor

    San Francisco, CA
    26 days ago
  • $120 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...$80–$120/hour Location: Remote Role Responsibilities...  ...against domain-specific quality rubrics. Identify factual, aesthetic...  ...improve AI-generated content. Review documents, spreadsheets, and... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    New York, NY
    11 days ago
  • $120 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...$80–$120/hour Location: Remote Role Responsibilities...  ...against domain-specific quality rubrics. Identify factual, aesthetic...  ...****@*****.*** PS: Our team reviews applications daily. Please... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    San Francisco, CA
    26 days ago
  • $18 per hour

    Globalclassifieds.info is hiring to help improve AI by reviewing voice models and AI-generated images and videos from a remote location. You’ll provide feedback to improve accuracy and quality. This flexible, part-time role pays up to $18/hour with weekly payments and requires... 
    Remote job
    Weekly pay
    Part time
    Flexible hours

    Globalclassifieds.info

    Toms River, NJ
    2 days ago
  • $18 per hour

    Realtor.pm is seeking individuals to test and review AI content remotely. Earn up to $18 per hour by talking to AI, evaluating AI-generated visuals, and providing feedback to improve AI performance. This talent gig offers flexible hours, remote work, and weekly payments... 
    Remote job
    Hourly pay
    Weekly pay
    Flexible hours

    Realtor.pm

    Lakewood, NJ
    1 day ago
  •  ...engagements, and advisory opportunities. We are seeking experienced HR professionals to help train a language model on professional...  ...and slide decks, while advancing core HR workflows. This flexible, remote role offers 5-20 hours per week with ASAP start and a competitive... 
    Remote job
    Hourly pay
    Immediate start
    Flexible hours

    Ethos

    New York, NY
    16 hours ago
  • $25 per hour

     ...the gig Join our distributed team as a Remote AI Task Reviewer and help improve AI systems by...  ...when needed Perks & Benefits ~$25/hr + bonuses ~ Work from home on your...  ...If you’re looking for a well-paid, fully remote side gig you can do from your own... 
    Remote job
    Second job
    Immediate start
    Work from home
    Flexible hours

    RLDG

    Honey Brook, PA
    12 days ago
  •  ...Assessment Rubric Learning Content Reviewer is a remote review track for evaluating AI outputs across assessment rubric learning content specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk;... 
    Remote job
    Hourly pay
    For contractors
    Work experience placement
    10 hours per week

    AuraOne Human Data

    Remote
    a month ago
  • $90 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Position: Civil Engineer Reviewer Type: Contract Compensation...  ...: $90/hour Location: Remote Role Responsibilities...  ...omissions. Evaluate grading rubrics for clarity and specificity, ensuring... 
    Remote work
    Contract work
    Summer work

    Mercor

    New York, NY
    13 days ago
  • Uplinq is seeking a Tax Reviewer to serve as the final technical check before returns go...  ...review federal and state returns, including AI-prepared work, and use our Claude-based...  ..., and proactive problem solving. This fully remote US role offers growth as the company scales... 
    Remote job

    Uplinq

    New York, NY
    2 days ago
  • AuraOne is seeking reviewers for its remote Logistics & Planning Professionals track. You will assess AI outputs against supply chain workflows and firm policies, grading tone...  ..., and stakeholder fit using a structured rubric. The role emphasizes clear written reasoning... 
    Remote job
    For contractors

    Auraone

    New York, NY
    1 day ago
  •  ...seeking computer science specialists to evaluate AI-generated graphical abstracts derived from academic papers. This fully remote engagement focuses on expert judgement to...  ...faithfully reflect the underlying research. You will review CS papers alongside AI-generated graphics,... 
    Remote job

    prolificacademicltd

    New York, NY
    3 days ago
  • AuraOne, Inc. is seeking a remote Differential Diagnosis AI Reviewer to evaluate evaluation prompts and responses against the company's quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for model retraining. The role is an... 
    Remote job
    For contractors
    10 hours per week

    AuraOne, Inc.

    New York, NY
    2 days ago
  • AI Trainer Jobs in the United States seeks a remote Business Document Expert (German Speaker) to evaluate German generalist prompts and responses against AuraOne's rubric. You will label edge cases, provide structured feedback for retraining, and compare outputs to select... 
    Remote work
    Hourly pay
    For contractors

    Auraone

    New York, NY
    3 days ago
  • AuraOne seeks experienced reviewers to evaluate car mounted 360 video recorder outputs against a versioned rubric, tagging severity and highlighting edge cases. You will compare...  ...failure modes for model improvement. This remote contractor role requires clear reasoning... 
    Remote job
    For contractors

    Auraone

    New York, NY
    2 days ago
  • Uplinq Financial Technologies is seeking a Tax Reviewer who will be the final technical check before a return goes...  ...estates, and nonprofits, including those prepared with AI, using Claude and our platform. Fully remote from the US. Ideal candidates have tax review... 
    Remote job

    Uplinq Financial Technologies

    New York, NY
    2 days ago
  •  ...seeking a Federal Performance and Financial Audit Expert to remotely review AI outputs across audit workflows. You will grade calculations,...  ...credentials (CPA/CFA/ACA or equivalent), and the ability to apply rubrics consistently. 10+ hours weekly async availability is... 
    Remote job

    AuraOne Human Data

    New York, NY
    3 days ago
  •  ...seeking an experienced forestry and land management expert to review AI-generated scenarios for accuracy and practicality. You will...  ...providing structured feedback to improve AI reasoning. This is a fully remote, task-based contract offering 10-40 hours per week and... 
    Remote job
    Contract work
    10 hours per week
    Flexible hours

    Alignerr

    New York, NY
    1 day ago
  •  ...Ltd. is seeking a qualified neuroscience expert for a fully remote verification role. You will compare AI-generated graphical abstracts to source papers,...  ...deep knowledge of current methods, and a track record of meticulous review. #J-18808-Ljbffr prolificacademicltd
    Remote job
    Flexible hours

    prolificacademicltd

    New York, NY
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Rubric Reviewer - Fully Remote | Up to $75/hr. Be the first to apply!