Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Rubric Reviewer Fully Remote

$40 - $75 per hour
Full-time

Mercor

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include Benchmark General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey .

Position: AI Evaluation Rubric Reviewer

Type: Contract

Compensation: $40$75/hour

Location: Remote

Role Responsibilities

  • Review AI-drafted tasks and evaluate model attempts for fairness and accuracy.
  • Assess if the grading rubric aligns with the prompt and correct any unfair criteria.
  • Flag tasks that cannot be saved and provide concise feedback on each evaluation.
  • Analyze long agent transcripts to identify errors and improve AI model performance .
  • Ensure rubric structures are correct per guidelines and enhance training data quality .

Qualifications

Must-Have

  • Substantial experience writing or reviewing rubrics for LLM evaluations .
  • Ability to read long agent transcripts and identify errors.
  • Precise writing skills and a commitment to data verification.

Application Process (Takes 2030 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

  • For details about the interview process and platform information please check:
  • For any help or support reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the AI Rubric Reviewer Fully Remote in New York, NY vacancy
  • $40 - $75 per hour

     ...AI Evaluation Rubric Reviewer is a remote evaluation track for reviewing ai evaluation rubric evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    5 days ago
  • $25 per hour

     ...Description About the gig Join our distributed team as a Remote AI Task Reviewer and help improve AI systems by completing short, style...  ...onboarding right away. If you’re looking for a well-paid, fully remote side gig you can do from your own laptop or tablet, apply... 
    Remote job
    Second job
    Immediate start
    Work from home
    Flexible hours

    RLDG

    Honey Brook, PA
    15 days ago
  •  ...Assessment Rubric Learning Content Reviewer is a remote review track for evaluating AI outputs across assessment rubric learning content specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk;... 
    Remote job
    Hourly pay
    For contractors
    Work experience placement
    10 hours per week

    AuraOne Human Data

    Remote
    a month ago
  •  ...a Wholesale and Technical Sales Reviewer to join our distributed team. This...  ...managing pipelines. You will evaluate AI-generated sales content, develop evaluation rubrics, and collaborate with subject...  ...ensure accuracy across datasets. Remote work is supported and English... 
    Remote job
    Contract work

    Mercor

    San Francisco, CA
    4 days ago
  •  ...management professionals to help shape how AI understands sustainable forestry,...  ...ecosystem dynamics, and land-use practices. This fully remote, flexible hourly contract lets you...  ...development on your own schedule. You will review AI-generated scenarios, assess content accuracy... 
    Remote job
    Hourly pay
    Contract work
    10 hours per week
    Flexible hours

    Alignerr Corp.

    Miami, FL
    3 days ago
  • $30 per hour

     ...Freelance Physics QA Reviewer - AI Trainer 2 weeks ago - Be among the first 25 applicants Get AI-powered advice on this job and...  ...work with challenging, complex guidelines. Your role is fully remote, so you just need a laptop, internet connection, available time... 
    Remote work
    Part time
    Freelance

    Mind Rift

    New York State
    1 day ago
  •  ...A leading recruitment firm is seeking a Veterinary Technician - AI Content Reviewer for a fully remote position on a part-time basis. This 12-month contract role involves evaluating AI-produced content related to veterinary medicine and providing expert feedback. Applicants... 
    Remote work
    Contract work
    Part time

    Motion Recruitment

    United States
    2 days ago
  • $40 - $120 per hour

     ...Content Reviewer $40-120/hr Remote Freelance GENERAL About the Role What if your sharp eye for quality...  ...how millions of people experience AI? We're looking for Content Reviewers...  ...accurate, and genuinely useful. This is a fully remote, flexible contract role open to... 
    Remote work
    Hourly pay
    Ongoing contract
    Contract work
    Freelance
    Flexible hours

    Alignerr

    United States
    2 days ago
  • $20 - $30 per hour

     ...Type: Contract, hourly Location: Remote Hours: 20–30 per week Pay: $20–30/hour, based...  ...THE ROLE We evaluate how well advanced AI coding agents solve real engineering...  ...looked at yet and judge it step by step. • Review — Take an existing annotation, written by... 
    Remote work
    Hourly pay
    Full time
    Contract work
    Immediate start

    Bespoke Labs

    Parkersburg, WV
    13 hours ago
  • An AI research accelerator is seeking a candidate skilled in internal or emergency medicine to enhance AI systems. This remote position demands collaboration on clinical scenarios and evaluation of AI responses, requiring strong communication and diagnostic skills. An MD... 
    Remote work
    Contract work
    Flexible hours

    Turing

    San Francisco, CA
    1 day ago
  • $70 - $100 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...$70–$100/hour Location: Remote Commitment: 10–20+ hours/...  ...outputs against structured rubrics and provide feedback to improve...  ...~ Immediately ; applications reviewed on a rolling basis. Application... 
    Remote work
    Contract work
    Summer work
    Work at office
    Immediate start

    Mercor

    Dallas, TX
    13 days ago
  • $70 - $100 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...$70–$100/hour Location: Remote Commitment: 10–20+ hours/...  ...outputs against structured rubrics. Provide written feedback to...  ...professional experience producing and reviewing office documents. ~ Strong... 
    Remote work
    Contract work
    Summer work
    Work at office
    Immediate start

    Mercor

    Dallas, TX
    7 days ago
  • $55 - $120 per hour

     ...Job Description Cardiologist – AI Medical Content Reviewer (Remote, Flexible, Non-Clinical) Are you a trained Cardiologist (or a physician with...  ...prescribing, or clinical responsibility . Why This Role Fully remote and highly flexible — work anytime, including... 
    Remote work
    Hourly pay
    Long term contract
    Full time
    Part time
    Flexible hours
    Afternoon shift

    Brain Trust Inc

    United States
    2 days ago
  •  ...an experienced Gynecologist to help train AI models by evaluating obstetrics and gynecology responses. The role involves reviewing clinical questions, rating answers, and logging...  ...licensed in OB/GYN are welcome to apply. Remote-friendly, with emphasis on English fluency,... 
    Remote job

    DataAnnotation

    Frisco, TX
    2 days ago
  • DataAnnotation is seeking an experienced Gynecologist to train AI models in obstetrics and gynecology, evaluating...  ...and their reasoning quality. This role is remote and open to qualified clinicians worldwide. You will review AI answers, rate accuracy, compare independent... 
    Remote job
    Worldwide

    DataAnnotation

    Newport News, VA
    2 days ago
  • $125 per hour

     ...seeking an experienced gynecologist or licensed clinician to train AI models in obstetrics and gynecology. You will assess AI-generated...  ..., NPs, and PAs in qualifying regions are welcome to apply. This remote role offers flexible engagement with compensation up to $125+ per... 
    Remote job
    Hourly pay
    Flexible hours

    DataAnnotation

    Katy, TX
    1 day ago
  • DataAnnotation is seeking an experienced OB/GYN professional to help train AI models by evaluating responses to obstetrics and gynecology...  .../GYNs or nurses, NPs, and PAs licensed in obstetrics/gynecology. Remote, project-based work available to qualified applicants in eligible... 
    Remote job

    DataAnnotation

    Katy, TX
    1 day ago
  • DataAnnotation is seeking an experienced obstetrician-gynecologist or licensed nurse/NP/PA to help train AI models in obstetrics and gynecology. You will review AI-generated clinical answers, rate accuracy, safety, and usefulness, and compare competing responses. You’ll... 
    Remote job
    Flexible hours

    DataAnnotation

    Orlando, FL
    2 days ago
  • $125 per hour

    DataAnnotation seeks an experienced clinician to help train AI models in obstetrics and gynecology. You will judge AI responses to clinical...  ...GYN experience, precise writing, and collaboration across tasks. Remote, project-based work with compensation up to $125+/hr is available... 
    Remote job

    DataAnnotation

    Orlando, FL
    2 days ago
  • $100 - $120 per hour

    OpenTrain AI, Inc. is seeking a Family Medicine physician to review Japanese patient histories and evaluate AI-generated care plans for Type 2 diabetes. This remote contractor role requires an active medical license in Japan and ongoing clinical practice. The project is... 
    Remote job
    Hourly pay
    Part time
    For contractors

    OpenTrain AI, Inc.

    Annapolis, MD
    4 days ago
  • $55 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Wholesale and Technical Sales Reviewer Type: Contract Compensation: $50-$55/hour Location: Remote Role Responsibilities Evaluate...  ...performance. Develop evaluation rubrics and benchmarks for assessing AI... 
    Remote job
    Contract work
    Summer work

    Mercor

    San Francisco, CA
    4 days ago
  •  ...gynecologist or clinically licensed professional to train and validate AI models in obstetrics and gynecology. You will assess AI-generated...  ...and to nurses, NPs, and PAs licensed in obstetrics and gynecology, with remote, flexible arrangements. #J-18808-Ljbffr DataAnnotation
    Remote job
    Flexible hours

    DataAnnotation

    Cincinnati, OH
    2 days ago
  • DataAnnotation is seeking an experienced Gynecologist to help train AI models by evaluating obstetrics and gynecology questions. You...  ...and to nurses, NPs, and PAs licensed in obstetrics and gynecology. Remote, flexible project work with diverse healthcare tasks and... 
    Remote job
    Flexible hours

    DataAnnotation

    Dunwoody, GA
    2 days ago
  •  ...Emergency Medicine AI Reviewer is a remote clinical-review track for evaluating AI outputs that touch emergency medicine. Reviewers grade differential...  ...reasoning, dosing, and red-flag handling on a structured rubric. Flag patient-safety issues with severity tags that... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    11 days ago
  •  ...Compliance Workflow Workflow AI Reviewer is a remote review track for evaluating AI outputs in compliance workflows. Reviewers grade citation...  ...or advising on real matters. Comfort applying multi-page rubrics consistently across long batches. Clear written reasoning... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    11 days ago
  •  ...Valuation AI Reviewer is a remote review track for evaluating AI outputs across valuation workflows. Reviewers grade calculations, narrative...  ...applied experience also works. Comfort applying multi-page rubrics consistently across long batches. Clear written... 
    Remote job
    Hourly pay
    For contractors
    Work experience placement
    10 hours per week

    AuraOne Human Data

    Remote
    a month ago
  •  ...Computational Biology AI Reviewer is a remote review track for evaluating AI outputs across biology reasoning, calculations, and research workflows...  ...at a professional level. Comfort applying multi-page rubrics consistently across long batches. Clear written... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    a month ago
  • $75 per hour

     ...Aerospace Engineer, AI Technical Reasoning Reviewer is a remote engineering review track for evaluating production code, debugging traces, and developer...  ...solutions and rank them with a written rationale tied to the rubric. Tag failure modes (compile error, runtime crash,... 
    Remote job
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    12 hours ago
  • $90 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ...Position: Civil Engineer Reviewer Type: Contract Compensation...  ...: $90/hour Location: Remote Role Responsibilities...  ...omissions. Evaluate grading rubrics for clarity and specificity, ensuring... 
    Remote work
    Contract work
    Summer work

    Mercor

    New York, NY
    16 days ago
  • $55 per hour

     ...A leading staffing agency is seeking a Dental Hygienist Subject Matter Expert to review and evaluate AI-generated educational content for dental hygiene learners. This part-time remote position requires an active RDH license and a minimum of two years of clinical experience... 
    Remote work
    Hourly pay
    Part time

    Motion Recruitment

    New York, NY
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Rubric Reviewer Fully Remote. Be the first to apply!