Learning Progression AI Evaluator [Remote]
AuraOne Human Data
Remote
- Remote job
Learning Progression AI Evaluator is a remote review track for evaluating AI outputs across learning progression ai specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next step so the modeling team can train on it.
Why this role matters
Learning Progression AI specialist operations AI has to fit into an actual day at work. AuraOne uses experienced operators to grade outputs the way a senior peer would — checking workflow, policy, and the unwritten rules that decide whether a task actually gets done.
Responsibilities
- Review AI outputs against current learning progression ai specialist operations workflows, playbooks, and firm policy for Learning Progression AI Evaluator assignments.
- Grade tone, escalation logic, and stakeholder fit on a structured rubric.
- Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
- Capture the right next step so the modeling team can train on it.
- Adjudicate disputed treatments against published playbooks or firm guidance.
- Maintain reviewer-quality scores in inter-rater calibration cycles.
Qualifications
- Direct working experience in learning progression ai specialist operations on real teams for Learning Progression AI Evaluator work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the policy or workflow being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Grade a model's response to a real learning progression ai specialist operations ticket and rate workflow, tone, and escalation.
- Flag a missed escalation with the right severity tag and corrected next step.
- Adjudicate a disputed playbook call between two reviewers using firm guidance.
- Audit a 25-row batch for rubric consistency and report drift to the program lead.
Nice to have
- Prior experience training, calibrating, or QA-ing operations teams.
- Familiarity with AI-assisted workflow tooling and its failure modes.
- Bilingual experience for cross-region operations.
Skills
- Operational review
- Policy adherence
- Workflow judgment
- Stakeholder communication
- Learning Progression AI specialist operations
- Learning design
- Assessment review
- Pedagogy
- Learning
- Progression
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
Vacancy posted 7 days ago
Similar jobs that could be interesting for youBased on the Learning Progression AI Evaluator [Remote] in Remote vacancy
$18 - $22 per hour
...improve the safety of advanced AI models by applying Thai... ...fluency and cultural judgment to evaluate how models respond to sensitive... ...provided, and prior AI or machine learning experience is not required.... ...degree completed or currently in progress. Strong written reasoning...SuggestedRemote jobHourly payImmediate startSaidGig
Remote3 days ago- ...Supporting diverse AI data and language projects, the hourly contractor AI Trainer and Evaluator will work remotely to generate content, annotate data, and evaluate AI... ...model responses, or record voice samples for AI learning Label, classify, and structure documents to...SuggestedHourly payFor contractorsRemote work
Virtual Vocations Inc
United States3 days ago $14.5 per hour
A technology company is seeking an AI Web Search Evaluator to enhance the quality of search engine results. This flexible, remote, part-time role focuses on analyzing search performance and providing feedback to improve algorithms. Ideal candidates will have strong analytical...SuggestedHourly payPart timeRemote workFlexible hoursWelo Data
United States1 day ago- ...MERIT Beauty is seeking Turkish-speaking annotators for a contract role evaluating AI-generated content. You will review content for coherence, consistency, and alignment with real-world expectations in Turkish. The ideal candidates are native or fluent Turkish speakers...SuggestedContract workTemporary workImmediate startRemote work
MERIT Beauty
New York, NY5 days ago $20 per hour
A tech company specializing in AI is hiring a Digital Web Designer. In this remote role, you will evaluate AI-generated designs and provide feedback to enhance the model’s understanding of aesthetics. An ideal candidate will have a strong background in UI/UX design and...SuggestedRemote workFlexible hoursDataAnnotation
United States1 day ago- ...Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Work at officeRemote work
Obsidian
New York, NY3 days ago $11.5 per hour
...position as an Online Task Contributor. In this role, you will evaluate and provide feedback on content to enhance search engine results... ...at $11.50 hourly, based on task completion, with a supportive community of contributors involved in AI advancements. #J-18808-LjbffrHourly payPart timeRemote workUniversity of Delaware
United States1 day ago- ...Collects, analyzes, and reports on data to evaluate effectiveness of program services.Core... ...social systems in an effort to actively progress across the antiracist multicultural institution... ...prepared.Able to receive feedback and to learn from it.Has insight about self (i.e.,...Full timeSummer workLocal area
Family Connections
East Orange, NJ2 days ago $14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology companies to provide datasets that are high-quality, ethically sourced, relevant, diverse, and scalable to supercharge their AI models. As a Welocalize brand, Welo Data...Hourly payPart timeImmediate startRemote workWork from home10 hours per weekFlexible hoursWelo Data
New York, NY4 days ago- ...About the role We are hiring expert Evaluators in Data analysis / quantitative readouts to review and assess AI-generated work products (documents, spreadsheets, and slide decks) for accuracy, rigor, and domain quality. You will apply deep subject-matter expertise...Hourly payWork at officeRemote work
Mercor Inc
New York, NY2 days ago - ...Classroom Scenario AI Evaluator is a remote evaluation track for reviewing classroom scenario ai evaluation prompts and responses against... ...Inter-rater calibration Classroom Scenario AI evaluation Learning design Assessment review Pedagogy Classroom...Remote jobHourly payFor contractors10 hours per week
$14.5 per hour
...AI Web Search Evaluator Welo Data works with technology companies to provide datasets that are high-quality, ethically sourced, relevant, diverse, and scalable to supercharge their AI models. As a Welocalize brand, Welo Data leverages over 25 years of experience in...Bi-weekly payHourly payPart timeImmediate startRemote workWork from homeFlexible hours$90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San Francisco... ...Location: Remote Role Responsibilities Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks for training and evaluating models....Remote jobContract workSummer work$140 per hour
...Role Overview Apply your subject matter expertise to improve how next-generation AI systems learn, reason, and perform. As an AI Domain Expert you will evaluate AI outputs, create challenging prompts, annotate real-world data, and provide concise, actionable feedback...Hourly payPart timeFor contractorsRemote workVisa sponsorshipWork visaFree visaSaidGig
United States23 days ago$20 per hour
A leading AI development company in the United States is seeking detail-oriented individuals for remote opportunities in training AI... ...include developing prompts, writing high-quality responses, and evaluating AI outputs. The ideal candidates are fluent in English with...Hourly payRemote workFlexible hoursSupportFinity
United States4 days ago$15 - $30 per hour
...Help improve AI assistants by evaluating how they handle the everyday coordination people rely on, including email, scheduling, bookings, and shared... ...communication skills for clear reporting and timely progress updates. Strong time-management and self-organization skills...Hourly payContract workFor contractorsRemote workSaidGig
United States2 days ago- ...Prolific is seeking talented Product Designers and UX Specialists to join our Expert Network for training and evaluation of cutting-edge AI models. Candidates should have a relevant educational background and at least one year of experience in design-related fields. Responsibilities...Remote workWork from homeFlexible hours
Prolific
Arizona City, AZ4 days ago - ...Prolific is seeking Product Designers and UX Specialists to join our Expert Network, contributing to the training and evaluation of cutting-edge AI models. This role requires expertise in usability, design systems, and user research, providing essential feedback on design...Remote workWork from homeFlexible hours
Prolific
Tucson, AZ3 days ago $20 per hour
A leading AI technology firm is seeking a Graphic Design Expert to evaluate graphic design elements for AI model training. This part-time contract role offers remote work flexibility and requires expertise in design principles. Ideal candidates should have formal training...Contract workPart timeRemote workHumanSignal
Austin, TX4 days ago$166.4k - $197.6k
...! Position Summary As an Autism Evaluator at Blackbird Health, you will play a central... ...Manage Evaluation Timelines: Complete progress notes within 24 hours of service and... ...and belonging. We are able to grow and learn better together with a diverse team of employees...Contract workRemote workFlexible hoursWeekend work2 days per weekWeekday work$20 - $80 per hour
...Role Overview Help improve next-generation AI systems by supplying precise, real-world evaluation, annotation, and feedback. This remote contractor role focuses on how AI models learn, reason, and perform across diverse subject areas. Key Responsibilities Evaluate...Hourly payContract workFor contractorsRemote workSaidGig
United Statesmore than 2 months ago- ...remote, hourly contractor role supporting AI data and language projects on a project-... ...high-quality voice samples, to guide AI learning across diverse topics. Data annotation... ...to support AI training datasets. LLM evaluation: reviewing AI-generated responses for accuracy...Hourly payTemporary workFor contractorsRemote workFlexible hours
$50 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...0 hours/week Role Responsibilities Evaluate the effectiveness of AI systems in handling... ...food, health, productivity, careers, and learning. Provide detailed feedback to enhance...Contract workSummer workRemote work$20 - $80 per hour
...role, you''ll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. Key Responsibilities: Evaluate and score AI-generated responses using well-defined rubrics...Hourly payContract workRemote workSaidGig
United States3 days ago- ...Mental Health Evaluator The Mental Health Evaluator is a critical role dedicated to delivering comprehensive, clinically appropriate... ...exercise universal precautions, use personal protective equipment and devices, and learn the policies concerning infection control...
Maria Fareri Children's Hospital, a member of the WMCHealth ...
Valhalla, NY1 day ago $120 - $175 per hour
...Job Description Job Description About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers...Contract workSummer workRemote workMercor
Detroit, MI13 days ago$75 per hour
...Develop and deliver sociology content through AI training initiatives for postsecondary education. Evaluate and review AI-generated coursework, assignments,... ...and simulated academic papers. Create lectures, learning modules, and case studies across core sociological...Hourly payContract workRemote workCrossing Hurdles
New York, NY1 day ago- Description The Early Childhood Evaluator is responsible for conducting comprehensive developmental... ...eligibility and provides ongoing progress monitoring for early intervention... ...Actively seek professional development and learning opportunities and provide documentation...Full timeContract workPart timeReliefFlexible hours
- ...Materials Science AI Evaluator is a remote review track for evaluating AI outputs across materials science reasoning, calculations, and research workflows. Reviewers grade derivations and assumptions, reproduce key results, and document the correct method so the modeling...Remote jobHourly payFor contractors10 hours per week
- ...Insurance Policy AI Evaluator is a remote review track for evaluating AI outputs across insurance workflows. Reviewers grade calculations, narrative reasoning, and policy adherence; flag compliance and reconciliation issues; and document the correct treatment so the modeling...Remote jobHourly payFor contractorsWork experience placement10 hours per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Learning Progression AI Evaluator [Remote]. Be the first to apply!
