Assessment Rubric AI Evaluator [Remote]
AuraOne Human Data
Remote
- Remote job
Assessment Rubric AI Evaluator is a remote review track for evaluating AI outputs across assessment rubric ai specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next step so the modeling team can train on it.
Why this role matters
Assessment Rubric AI specialist operations AI has to fit into an actual day at work. AuraOne uses experienced operators to grade outputs the way a senior peer would — checking workflow, policy, and the unwritten rules that decide whether a task actually gets done.
Responsibilities
- Review AI outputs against current assessment rubric ai specialist operations workflows, playbooks, and firm policy for Assessment Rubric AI Evaluator assignments.
- Grade tone, escalation logic, and stakeholder fit on a structured rubric.
- Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
- Capture the right next step so the modeling team can train on it.
- Adjudicate disputed treatments against published playbooks or firm guidance.
- Maintain reviewer-quality scores in inter-rater calibration cycles.
Qualifications
- Direct working experience in assessment rubric ai specialist operations on real teams for Assessment Rubric AI Evaluator work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the policy or workflow being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Grade a model's response to a real assessment rubric ai specialist operations ticket and rate workflow, tone, and escalation.
- Flag a missed escalation with the right severity tag and corrected next step.
- Adjudicate a disputed playbook call between two reviewers using firm guidance.
- Audit a 25-row batch for rubric consistency and report drift to the program lead.
Nice to have
- Prior experience training, calibrating, or QA-ing operations teams.
- Familiarity with AI-assisted workflow tooling and its failure modes.
- Bilingual experience for cross-region operations.
Skills
- Operational review
- Policy adherence
- Workflow judgment
- Stakeholder communication
- Assessment Rubric AI specialist operations
- Learning design
- Assessment review
- Pedagogy
- Assessment
- Rubric
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
Vacancy posted 26 days ago
Similar jobs that could be interesting for youBased on the Assessment Rubric AI Evaluator [Remote] in Remote vacancy
- Rex.zone is seeking a Senior AI data annotator to perform data labeling and evaluation for NLP tasks, RLHF assessments, and prompt QA to improve training data quality and model... ...Miami talent demand. You will apply strict rubrics, document edge cases, and collaborate asynchronously...SuggestedRemote jobFull time
$80 - $120 per hour
Mercor is looking for a Biology / environmental science Evaluator to assess AI-generated artifacts based on quality rubrics. This position requires evaluating documents for errors and collaborating with AI teams to improve model performance. The ideal candidate should have...SuggestedRemote jobHourly payContract workWork at office$80 - $120 per hour
...NY, is looking for a Legal contracts / diligence / redlines Evaluator to assess AI-generated artifacts. This role requires 5+ years of... ...fluency in English. The ideal candidate will evaluate quality rubrics, provide structured feedback, and work independently to improve...SuggestedRemote job- About the role We are hiring expert Evaluators in Compliance / regulatory response with financial-services AI to review and assess AI-generated work products (documents, spreadsheets... ...against domain-specific quality rubrics. Identify factual, aesthetic, and presentation...SuggestedHourly payWork at officeRemote work
$30 per hour
...hours/week Role Responsibilities Evaluate outputs from large language... ...autonomous agent systems using defined rubrics and quality standards. Review... ...and reasoning traces, to assess accuracy and completeness.... ...experience in LLM evaluation, AI output analysis, QA/testing, UX...SuggestedRemote jobHourly payContract work- ...Based in Poland, you will contribute to AI-powered visual evaluation systems by rating design samples and... ...high evaluation quality. You will assess typography, color, hierarchy, and visual... ...sessions and a 1–5 rating rubric to ensure consistency across large datasets...Contract workPart timeRemote work
Jobgether
Poland, NY3 days ago - ...Graphic Design Expert for a remote, part-time contract to evaluate AI-powered visual datasets. You will assess typography, hierarchy, color, and overall polish across large volumes of samples, using established rubrics and rating scales. The role requires formal training...Contract workPart timeRemote work
Jobgether
Cedar Grove, WI3 days ago - AuraOne is seeking a Brand Design Quality Evaluator to remotely assess design quality prompts and model outputs. You will compare paired responses, apply a versioned rubric, and provide structured feedback for retraining the modeling team. Responsibilities include labeling...Remote jobFor contractors
$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered... ...success / support operations Evaluator Type: Contract Compensation... ...domain-specific quality rubrics. Identify factual,... ...while ensuring high-quality assessments. Collaborate with subject...Contract workSummer workWork at officeRemote workMercor
Houston, TX3 days ago$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... ...inventory / capacity planning Evaluator Type: Contract... ...against domain-specific quality rubrics. Ensure accuracy and rigor in... ...tools, especially Slides , to assess and refine work products. Qualifications...Contract workSummer workWork at officeRemote workMercor
Dallas, TX3 days ago- ...Hiring PhD-level quantitative finance experts, the full-time Remote AI Research Evaluator will assess AI-generated financial content, craft relevant questions, and evaluate model responses in a flexible contract role. Key responsibilities Assessing the factuality and...Full timeContract workRemote workFlexible hours
Virtual Vocations Inc
United States2 days ago $50 - $175 per hour
...technical talent with leading AI research labs. Headquartered in... .... Position: JSON-fluent & Rubrics Expert Type: Contract... ...week Role Responsibilities Evaluate JSON-formatted AI model... ...rubrics. Parse, read, and assess JSON-based outputs, grading them...Contract workSummer workStart working todayRemote workMercor
Seattle, WA11 days ago- ...Alignerr is seeking a Search Quality Evaluator to assess search engine results, AI-generated answers, and content recommendations. This fully remote, flexible contract role values strong critical thinking and quality instincts, with no technical background required beyond...Contract workRemote workFlexible hours
Alignerr
Peru, IL1 day ago $14.5 per hour
...diverse, and scalable to supercharge their AI models. As a Welocalize brand, Welo Data... ...? Join our team as a Web Search Evaluator and help shape the future of search engines... ...Web Search Evaluator, you will: Review and assess internet search results, ensuring users receive...Bi-weekly payHourly payPart timeImmediate startRemote workWork from homeFlexible hoursJobs for Humanity
Jacksonville, FL4 days ago$80 - $120 per hour
...materials, fundraising, and pitchbook expertise to assess AI-generated documents, spreadsheets, and slide... ...quality. Role Overview You will evaluate AI-generated work products using domain-specific quality rubrics and deliver well-reasoned assessments grounded...Hourly payWork at officeRemote workSaidGig
United States8 days ago$120 - $175 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Contribute to text-only, objective, verifiable rubrics for grading CNC machining solutions across... .... ~ Strong production judgment to assess safe operations on real machines. ~ Written...Contract workSummer workRemote workMercor
Detroit, MI11 days ago$100 per hour
...looking for a highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as... ...whether the model thinks like a great engineer. You will assess how AI coding agents behave in real-world scenarios — focusing on:...Contract workImmediate startG2i Inc.
Remote2 days ago- Obsidian is seeking expert Evaluators for a remote, hourly role focused on reviewing AI-generated work products for quality and accuracy. The successful candidates... ...will leverage their subject matter expertise to assess various documents, spreadsheets, and slide decks....Remote jobHourly payWork at office
$20 - $30 per hour
A remote-focused technology firm is looking for an individual proficient in evaluating large language model outputs. The role involves assessing AI systems, reviewing workflows, and providing actionable feedback to enhance product quality. Ideal candidates will have strong...Remote jobHourly pay- ...Mental Health Clinical AI Evaluator is a remote clinical-review track for evaluating AI outputs... ...and red-flag handling on a structured rubric. Flag patient-safety issues with... ...reading patient cases and writing structured assessments. Comfort applying multi-page rubrics...Remote jobHourly payFor contractors10 hours per week
$9 per hour
Productive Playhouse is forming a pool of Vietnamese-speaking evaluators for testing AI chatbots. This independent contractor role lets you choose... .... Responsibilities include interacting with AI models to assess capabilities, safety, and usefulness; reporting outcomes and...Remote jobHourly payFor contractorsFreelanceFlexible hours- Join Acentra Health as a PASRR Evaluator in Kansas. In this role, you will conduct PASRR Level II Pre-Admission Screening and Resident Review assessments in Kansas City and the surrounding areas, make level-of-care recommendations, and help ensure the delivery of quality...Remote jobReliefWork from homeHome office
- TELUS Digital AI Community invites freelance content evaluators in the United States to join a remote independent contractor role. You will help improve AI... ...on project guidelines. Responsibilities include assessing relevance, quality, accuracy, usefulness, and user intent...Remote jobFor contractorsFreelance
- ...is building a talent pool of Armenian speakers to test and evaluate leading AI chatbots. This freelance, project-based opportunity lets you... ...specifications in each SOW. You will interact directly with AI models to assess capabilities, safety, and helpfulness, and deliver...Remote jobFreelanceFlexible hours
$190 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...RemoteCommitment:20+ hours/week Role Responsibilities Evaluate AI systems on complex personal workflows,... ...to evaluate AI outputs and write against rubrics. Preferred Extensive rubric experience...Remote jobHourly payFor contractorsSummer workTrial period$24 per hour
Prolific is seeking an AI Trainer with advanced Tamil fluency to evaluate AI models' understanding of the Tamil language's emotional and cultural nuances. Responsibilities include assessing audio clips, reviewing tones for cultural relevancy, and ensuring quality control...Remote jobFlexible hours- Mercor is seeking senior insurance professionals to build evaluation tasks for AI systems operating in Fortune 500 enterprise insurance and risk... ...insurance scenarios, draft reference outputs, and write rubrics that capture how senior F500 insurance operators think. #J...Remote job
- A talent marketplace is seeking soccer experts to evaluate live soccer games. The role involves assessing AI-generated commentary, scoring performance, and providing feedback. Qualified candidates will have deep expertise in soccer, strong analytical and communication...Contract work
- ...Playhouse is building a pool of Marathi speakers to test and evaluate leading AI chatbots. You will work as an independent contractor on a... ...as needed. Responsibilities include testing conversations, assessing model capabilities and safety, and submitting deliverables in...Remote jobFor contractorsFlexible hours
$80 - $120 per hour
...Role Overview Assess, grade, and provide structured written feedback on AI-generated data quality and CRM operations deliverables... .... Key Responsibilities Evaluate AI-generated artifacts against domain-specific quality rubrics for Data quality and CRM operations...Hourly payWork at officeRemote workSaidGig
United States1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Assessment Rubric AI Evaluator [Remote]. Be the first to apply!
