AI Rubric Reviewer Fully Remote
$40 - $75 per hourMercor
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include Benchmark General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey .
Position: AI Evaluation Rubric Reviewer
Type: Contract
Compensation: $40$75/hour
Location: Remote
Role Responsibilities
- Review AI-drafted tasks and evaluate model attempts for fairness and accuracy.
- Assess if the grading rubric aligns with the prompt and correct any unfair criteria.
- Flag tasks that cannot be saved and provide concise feedback on each evaluation.
- Analyze long agent transcripts to identify errors and improve AI model performance .
- Ensure rubric structures are correct per guidelines and enhance training data quality .
Qualifications
Must-Have
- Substantial experience writing or reviewing rubrics for LLM evaluations .
- Ability to read long agent transcripts and identify errors.
- Precise writing skills and a commitment to data verification.
Application Process (Takes 2030 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information please check:
- For any help or support reach out to:
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
$40 - $75 per hour
...AI Evaluation Rubric Reviewer is a remote evaluation track for reviewing ai evaluation rubric evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team...Remote jobFor contractors10 hours per week$25 per hour
...Description About the gig Join our distributed team as a Remote AI Task Reviewer and help improve AI systems by completing short, style... ...onboarding right away. If you’re looking for a well-paid, fully remote side gig you can do from your own laptop or tablet, apply...Remote jobSecond jobImmediate startWork from homeFlexible hours- ...Assessment Rubric Learning Content Reviewer is a remote review track for evaluating AI outputs across assessment rubric learning content specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk;...Remote jobHourly payFor contractorsWork experience placement10 hours per week
- ...a Wholesale and Technical Sales Reviewer to join our distributed team. This... ...managing pipelines. You will evaluate AI-generated sales content, develop evaluation rubrics, and collaborate with subject... ...ensure accuracy across datasets. Remote work is supported and English...Remote jobContract work
- ...management professionals to help shape how AI understands sustainable forestry,... ...ecosystem dynamics, and land-use practices. This fully remote, flexible hourly contract lets you... ...development on your own schedule. You will review AI-generated scenarios, assess content accuracy...Remote jobHourly payContract work10 hours per weekFlexible hours
$30 per hour
...Freelance Physics QA Reviewer - AI Trainer 2 weeks ago - Be among the first 25 applicants Get AI-powered advice on this job and... ...work with challenging, complex guidelines. Your role is fully remote, so you just need a laptop, internet connection, available time...Remote workPart timeFreelance- ...A leading recruitment firm is seeking a Veterinary Technician - AI Content Reviewer for a fully remote position on a part-time basis. This 12-month contract role involves evaluating AI-produced content related to veterinary medicine and providing expert feedback. Applicants...Remote workContract workPart time
$40 - $120 per hour
...Content Reviewer $40-120/hr Remote Freelance GENERAL About the Role What if your sharp eye for quality... ...how millions of people experience AI? We're looking for Content Reviewers... ...accurate, and genuinely useful. This is a fully remote, flexible contract role open to...Remote workHourly payOngoing contractContract workFreelanceFlexible hours$20 - $30 per hour
...Type: Contract, hourly Location: Remote Hours: 20–30 per week Pay: $20–30/hour, based... ...THE ROLE We evaluate how well advanced AI coding agents solve real engineering... ...looked at yet and judge it step by step. • Review — Take an existing annotation, written by...Remote workHourly payFull timeContract workImmediate start- An AI research accelerator is seeking a candidate skilled in internal or emergency medicine to enhance AI systems. This remote position demands collaboration on clinical scenarios and evaluation of AI responses, requiring strong communication and diagnostic skills. An MD...Remote workContract workFlexible hours
$70 - $100 per hour
...technical talent with leading AI research labs. Headquartered in... ...$70–$100/hour Location: Remote Commitment: 10–20+ hours/... ...outputs against structured rubrics and provide feedback to improve... ...~ Immediately ; applications reviewed on a rolling basis. Application...Remote workContract workSummer workWork at officeImmediate start$70 - $100 per hour
...technical talent with leading AI research labs. Headquartered in... ...$70–$100/hour Location: Remote Commitment: 10–20+ hours/... ...outputs against structured rubrics. Provide written feedback to... ...professional experience producing and reviewing office documents. ~ Strong...Remote workContract workSummer workWork at officeImmediate start$55 - $120 per hour
...Job Description Cardiologist – AI Medical Content Reviewer (Remote, Flexible, Non-Clinical) Are you a trained Cardiologist (or a physician with... ...prescribing, or clinical responsibility . Why This Role Fully remote and highly flexible — work anytime, including...Remote workHourly payLong term contractFull timePart timeFlexible hoursAfternoon shift- ...an experienced Gynecologist to help train AI models by evaluating obstetrics and gynecology responses. The role involves reviewing clinical questions, rating answers, and logging... ...licensed in OB/GYN are welcome to apply. Remote-friendly, with emphasis on English fluency,...Remote job
- DataAnnotation is seeking an experienced Gynecologist to train AI models in obstetrics and gynecology, evaluating... ...and their reasoning quality. This role is remote and open to qualified clinicians worldwide. You will review AI answers, rate accuracy, compare independent...Remote jobWorldwide
$125 per hour
...seeking an experienced gynecologist or licensed clinician to train AI models in obstetrics and gynecology. You will assess AI-generated... ..., NPs, and PAs in qualifying regions are welcome to apply. This remote role offers flexible engagement with compensation up to $125+ per...Remote jobHourly payFlexible hours- DataAnnotation is seeking an experienced OB/GYN professional to help train AI models by evaluating responses to obstetrics and gynecology... .../GYNs or nurses, NPs, and PAs licensed in obstetrics/gynecology. Remote, project-based work available to qualified applicants in eligible...Remote job
- DataAnnotation is seeking an experienced obstetrician-gynecologist or licensed nurse/NP/PA to help train AI models in obstetrics and gynecology. You will review AI-generated clinical answers, rate accuracy, safety, and usefulness, and compare competing responses. You’ll...Remote jobFlexible hours
$125 per hour
DataAnnotation seeks an experienced clinician to help train AI models in obstetrics and gynecology. You will judge AI responses to clinical... ...GYN experience, precise writing, and collaboration across tasks. Remote, project-based work with compensation up to $125+/hr is available...Remote job$100 - $120 per hour
OpenTrain AI, Inc. is seeking a Family Medicine physician to review Japanese patient histories and evaluate AI-generated care plans for Type 2 diabetes. This remote contractor role requires an active medical license in Japan and ongoing clinical practice. The project is...Remote jobHourly payPart timeFor contractors$55 per hour
...technical talent with leading AI research labs. Headquartered in... ...Wholesale and Technical Sales Reviewer Type: Contract Compensation: $50-$55/hour Location: Remote Role Responsibilities Evaluate... ...performance. Develop evaluation rubrics and benchmarks for assessing AI...Remote jobContract workSummer work- ...gynecologist or clinically licensed professional to train and validate AI models in obstetrics and gynecology. You will assess AI-generated... ...and to nurses, NPs, and PAs licensed in obstetrics and gynecology, with remote, flexible arrangements. #J-18808-Ljbffr DataAnnotationRemote jobFlexible hours
- DataAnnotation is seeking an experienced Gynecologist to help train AI models by evaluating obstetrics and gynecology questions. You... ...and to nurses, NPs, and PAs licensed in obstetrics and gynecology. Remote, flexible project work with diverse healthcare tasks and...Remote jobFlexible hours
- ...Emergency Medicine AI Reviewer is a remote clinical-review track for evaluating AI outputs that touch emergency medicine. Reviewers grade differential... ...reasoning, dosing, and red-flag handling on a structured rubric. Flag patient-safety issues with severity tags that...Remote jobHourly payFor contractors10 hours per week
- ...Compliance Workflow Workflow AI Reviewer is a remote review track for evaluating AI outputs in compliance workflows. Reviewers grade citation... ...or advising on real matters. Comfort applying multi-page rubrics consistently across long batches. Clear written reasoning...Remote jobHourly payFor contractors10 hours per week
- ...Valuation AI Reviewer is a remote review track for evaluating AI outputs across valuation workflows. Reviewers grade calculations, narrative... ...applied experience also works. Comfort applying multi-page rubrics consistently across long batches. Clear written...Remote jobHourly payFor contractorsWork experience placement10 hours per week
- ...Computational Biology AI Reviewer is a remote review track for evaluating AI outputs across biology reasoning, calculations, and research workflows... ...at a professional level. Comfort applying multi-page rubrics consistently across long batches. Clear written...Remote jobHourly payFor contractors10 hours per week
$75 per hour
...Aerospace Engineer, AI Technical Reasoning Reviewer is a remote engineering review track for evaluating production code, debugging traces, and developer... ...solutions and rank them with a written rationale tied to the rubric. Tag failure modes (compile error, runtime crash,...Remote jobFor contractors10 hours per week$90 per hour
...technical talent with leading AI research labs. Headquartered in... ...Position: Civil Engineer Reviewer Type: Contract Compensation... ...: $90/hour Location: Remote Role Responsibilities... ...omissions. Evaluate grading rubrics for clarity and specificity, ensuring...Remote workContract workSummer work$55 per hour
...A leading staffing agency is seeking a Dental Hygienist Subject Matter Expert to review and evaluate AI-generated educational content for dental hygiene learners. This part-time remote position requires an active RDH license and a minimum of two years of clinical experience...Remote workHourly payPart time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Rubric Reviewer Fully Remote. Be the first to apply!
- remote data architect New York, NY
- remote data manager New York, NY
- remote property manager New York, NY
- office manager remote New York, NY
- it manager remote New York, NY
- claims specialist remote New York, NY
- project manager remote europe New York, NY
- remote video game New York, NY
- remote office assistant New York, NY
- node js remote New York, NY



