Assessment Rubric AI Evaluator [Remote]
AuraOne Human Data
Remote
- Remote job
Assessment Rubric AI Evaluator is a remote review track for evaluating AI outputs across assessment rubric ai specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next step so the modeling team can train on it.
Why this role matters
Assessment Rubric AI specialist operations AI has to fit into an actual day at work. AuraOne uses experienced operators to grade outputs the way a senior peer would — checking workflow, policy, and the unwritten rules that decide whether a task actually gets done.
Responsibilities
- Review AI outputs against current assessment rubric ai specialist operations workflows, playbooks, and firm policy for Assessment Rubric AI Evaluator assignments.
- Grade tone, escalation logic, and stakeholder fit on a structured rubric.
- Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
- Capture the right next step so the modeling team can train on it.
- Adjudicate disputed treatments against published playbooks or firm guidance.
- Maintain reviewer-quality scores in inter-rater calibration cycles.
Qualifications
- Direct working experience in assessment rubric ai specialist operations on real teams for Assessment Rubric AI Evaluator work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the policy or workflow being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Grade a model's response to a real assessment rubric ai specialist operations ticket and rate workflow, tone, and escalation.
- Flag a missed escalation with the right severity tag and corrected next step.
- Adjudicate a disputed playbook call between two reviewers using firm guidance.
- Audit a 25-row batch for rubric consistency and report drift to the program lead.
Nice to have
- Prior experience training, calibrating, or QA-ing operations teams.
- Familiarity with AI-assisted workflow tooling and its failure modes.
- Bilingual experience for cross-region operations.
Skills
- Operational review
- Policy adherence
- Workflow judgment
- Stakeholder communication
- Assessment Rubric AI specialist operations
- Learning design
- Assessment review
- Pedagogy
- Assessment
- Rubric
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Assessment Rubric AI Evaluator [Remote] in Remote vacancy
- 24-MAG LLC seeks experienced Educator/ Assessor Writers for remote evaluation of AI-generated written responses. You will apply structured rubrics, assess clarity and relevance, and provide detailed rationales to ensure consistent quality judgments across tasks. Ideal candidates...SuggestedRemote job
- AI Trainer Jobs is seeking an Independent contractor for a remote JavaScript and TypeScript AI Evaluator role. You will review prompts and responses against a rubric, label issues, and provide structured feedback to help retrain the model. Responsibilities include evaluating...SuggestedRemote jobFor contractors
- AuraOne is seeking a Fluid Dynamics AI Reviewer to remotely evaluate prompts and responses against its quality rubric. You will compare paired outputs, label edge cases, and provide structured feedback to retrain the model. This contractor role requires careful reasoning...SuggestedRemote jobFor contractors10 hours per week
- AI Trainer Jobs is seeking a remote Conversational AI Evaluator to review prompts and model outputs against a evolving rubric. You will compare paired responses, label errors, and provide structured feedback to support retraining efforts. The role emphasizes clear reasoning...SuggestedRemote jobHourly payFor contractors10 hours per week
- ...About the role We are hiring expert Evaluators in Data analysis / quantitative readouts to review and assess AI-generated work products (documents, spreadsheets, and... ...generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and...SuggestedHourly payWork at officeRemote work
Mercor Inc
New York, NY1 day ago $30 per hour
...hours/week Role Responsibilities Evaluate outputs from large language... ...autonomous agent systems using defined rubrics and quality standards. Review... ...and reasoning traces, to assess accuracy and completeness.... ...experience in LLM evaluation, AI output analysis, QA/testing, UX...Remote jobHourly payContract work- YO AI Labs is seeking an AI Evaluation Specialist to remotely assess enterprise AI outputs against detailed rubrics, measuring accuracy, relevance and quality. You will provide concise, actionable feedback to drive improvements. Responsibilities include identifying reasoning...Remote jobFor contractors
- AI Trainer Jobs seeks a remote independent contractor to review AI outputs for fintech operations evaluation. You will assess workflow adherence, tone, and escalation logic, assigning severity tags... ...the ability to apply multi-page rubrics consistently. This contractor...Remote jobHourly payFor contractorsFlexible hours
- YO AI Labs is seeking an experienced AI Evaluation Specialist to remotely assess AI assistant outputs for an enterprise training project. The contractor will evaluate responses against detailed rubrics, identify issues, and provide actionable feedback to improve AI systems...Remote jobFor contractors
- ...& Marketing to remotely review evaluation prompts and responses against the company's quality rubric. You will compare paired outputs... .... As a contractor, you will assess frontier model outputs, judge benchmark... ...reasoning, and reliable async #J-18808-Ljbffr AI Trainer JobsRemote jobFor contractors
$37.5 per hour
...create a golden data set for AI generated ad copy for various... ...role you will be responsible for evaluating advertising copy generated by... ...against an existing Amazon-provided rubric, while also providing feedback... ...candidates. AI helps assess applications and qualifications...Contract workTemporary workRemote workTEKsystems
New York, NY23 hours ago- ...About the role Swahili Evaluation AI Evaluator is a remote evaluation track for reviewing swahili generalist evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback...Hourly payFor contractorsRemote work10 hours per week
AI Trainer Jobs
New York, NY7 hours ago $80 - $120 per hour
...technical talent with leading AI research labs. Headquartered... .../ trademark / copyright law Evaluator Type: Contract Compensation... ...domain-specific quality rubrics. Identify factual,... ...asynchronously to ensure high-quality assessments. Utilize Microsoft Office...Contract workSummer workWork at officeRemote workMercor
Chicago, IL25 days ago- ...Seeking a full-time Remote AI Research Evaluator with a PhD in Quantitative Finance to assess and enhance AI models' capabilities in financial reasoning and quantitative analysis through flexible, contract-based work. Key responsibilities Assessing the factuality and...Full timeContract workRemote workFlexible hours
Virtual Vocations Inc
United States4 days ago $80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... ...inventory / capacity planning Evaluator Type: Contract... ...against domain-specific quality rubrics. Ensure accuracy and rigor in... ...tools, especially Slides , to assess and refine work products. Qualifications...Contract workSummer workWork at officeRemote workMercor
Dallas, TX20 days ago- Mercor is seeking expert Evaluators in Clinical / biomedical / pharma to review and assess AI-generated work products (documents, spreadsheets, and slide decks) for accuracy, rigor, and domain quality. This is a remote, hourly engagement. You will apply deep subject-matter...Remote jobHourly pay
- About the role Conversational AI Evaluator is a remote evaluation track for reviewing conversational ai evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling...Hourly payFor contractorsRemote work10 hours per week
- Obsidian is seeking expert Evaluators for a remote, hourly role focused on reviewing AI-generated work products for quality and accuracy. The successful candidates... ...will leverage their subject matter expertise to assess various documents, spreadsheets, and slide decks....Remote jobHourly payWork at office
$20 - $30 per hour
A remote-focused technology firm is looking for an individual proficient in evaluating large language model outputs. The role involves assessing AI systems, reviewing workflows, and providing actionable feedback to enhance product quality. Ideal candidates will have strong...Remote jobHourly pay- AI Trainer Jobs is seeking licensed clinicians to remotely evaluate AI outputs in internal medicine clinical reviews. You will assess differential diagnoses, dosing logic, and guideline adherence, and document corrected reasoning for model training. Ideal candidates hold...Remote jobHourly pay10 hours per week
- Prolific is seeking fluent Norwegian speakers to act as evaluators for AI training, performing side-by-side assessments of text and voice snippets to judge naturalness and authenticity. You will listen to audio clips and rate how naturally the AI speaks, providing detailed...Remote jobWork from homeFlexible hours
$65 - $70 per hour
Trust and Safety Policy AI Evaluator is a remote red-team track for stress-testing AI systems against adversarial prompts. Reviewers craft... ...the failure mode, and pair each successful jailbreak with the rubric clause it violated so the safety team can patch the gap....Hourly payFor contractorsRemote work10 hours per week$24 per hour
Prolific is seeking Fluent Tamil speakers to serve as AI evaluators on a remote, freelance basis. You will assess how well AI models capture emotions and cultural nuance in Tamil, with pay up to $24/hr for one-hour tasks, and the option for shorter work sessions. Requirements...Remote jobFreelance- AuraOne is seeking a Children Safety Content AI Evaluator for a remote, contractor-based role. You will evaluate model outputs, compare responses... ...label content with structured tags according to our quality rubric. Ideal candidates have prior evaluation/annotation experience...Remote jobFor contractors10 hours per week
$40 - $100 per hour
About OpenTrain OpenTrain AI is the hiring and contracting organization for this opportunity... ...work About AI Training and Scientific Evaluation AI training is the human side of building... ...range of scientific topics. You will assess scientific accuracy, logical soundness, completeness...Hourly payContract workPart timeFor contractorsRemote work- ...Healthcare Compliance AI Evaluator is a remote clinical-review track for evaluating AI outputs... ...and red-flag handling on a structured rubric. Flag patient-safety issues with... ...reading patient cases and writing structured assessments. Comfort applying multi-page rubrics...Remote jobHourly payFor contractors10 hours per week
- AuraOne is seeking a Manipulation Risk Risk Evaluator for a remote, contractor role focused on stress-testing AI systems against adversarial prompts. Reviewers craft... ...failures, and pair each jailbreak with the violated rubric clause so the safety team can patch gaps....For contractorsRemote work
- AI Trainer Jobs is seeking a Bilingual AI Response Evaluator for a remote, contract-based role. You will review bilingual prompts and responses, compare outputs,... ...judgment, clear written rationales, and consistent rubric application. You'll work asynchronously for at...Remote jobHourly payContract work10 hours per week
$20 per hour
Prolific is seeking fluent Norwegian speakers to act as evaluators for AI training. You will perform side-by-side evaluations of text and voice snippets, assessing naturalness and authenticity to help models understand Norwegian language nuances. Pay is up to $20/hr with...Remote jobFlexible hours- About the role PhD Mathematics AI Evaluator is a remote review track for evaluating AI outputs across mathematics reasoning, calculations... ...the topic at a professional level. Comfort applying multi-page rubrics consistently across long batches. Clear written reasoning that...Hourly payFor contractorsRemote work10 hours per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Assessment Rubric AI Evaluator [Remote]. Be the first to apply!
