Retrieval Relevance Task Evaluator [Remote]
AuraOne Human Data
- Remote job
Retrieval Relevance Task Evaluator is a remote evaluation track for reviewing retrieval relevance task evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.
Why this role matters
AI data reviewers help turn retrieval relevance task evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data.
Responsibilities
- Evaluate retrieval relevance task evaluation model outputs against a versioned rubric and assign severity tags for Retrieval Relevance Task Evaluator assignments.
- Compare paired responses and pick the stronger answer with a written rationale.
- Label hallucinations, instruction-following failures, and unsafe content with structured tags.
- Capture ambiguous prompts and route them back to the program team for rubric updates.
- Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
- Document recurring failure modes so the modeling team can target them in the next training run.
Qualifications
- Prior evaluation, annotation, or human-rater experience on retrieval relevance task evaluation or adjacent content for Retrieval Relevance Task Evaluator work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the issue and the rubric clause being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Compare two retrieval relevance task evaluation model responses to the same prompt and pick the stronger one with rationale.
- Tag an unsafe response with the correct policy category and severity.
- Audit a 50-row batch for rubric consistency and report drift to the program lead.
- Propose a rubric clarification after spotting a recurring failure mode.
Nice to have
- Background in linguistics, content moderation, or trust & safety review.
- Experience with inter-rater agreement metrics and calibration cycles.
- Domain expertise that lets you spot subject-matter errors automated checks miss.
Skills
- Model output evaluation
- Rubric-based annotation
- Severity tagging
- Inter-rater calibration
- Retrieval Relevance Task evaluation
- Web research
- Source grounding
- Browser automation
- Retrieval
- Relevance
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
- A leading digital solutions provider is seeking a Personalized Ads Evaluator for a part-time position. This entry-level role involves reviewing online advertisements to assess their relevance to search terms, providing detailed feedback on content and cultural context....SuggestedRemote jobPart time
- ...Collects, analyzes, and reports on data to evaluate effectiveness of program services.Core... ...Quality to develop, maintain and report on relevant process and outcome data in a timely... ...and self-direction to complete assigned tasks and identify tools and approaches for analysis...SuggestedFull timeSummer workLocal area
$40 - $100 per hour
...language work About AI Training and Scientific Evaluation AI training is the human side of... ...calculations, and research-style reasoning tasks. Focus on scientific evaluation of AI... ...science, or space physics Deep knowledge of relevant fundamentals, including radiative...SuggestedHourly payContract workPart timeFor contractorsRemote work- ...Physical Affordance Task Evaluator is a remote review track for evaluating AI outputs across physics reasoning, calculations, and research workflows. Reviewers grade derivations and assumptions, reproduce key results, and document the correct method so the modeling team...SuggestedRemote jobHourly payFor contractors10 hours per week
- ...Surgical Robotics Task Evaluator is a remote review track for evaluating robotics review outputs against real-world physical-AI constraints. Reviewers grade trajectory choices, perception assumptions, and safety envelopes; flag failure modes; and document the corrected...SuggestedRemote jobHourly payFor contractors10 hours per week
$50 - $60 per hour
...Contributors review examples, test model behavior, evaluate responses, and identify errors so AI... ...directly with cutting-edge AI evaluation tasks Use practical analytics expertise to... ...commitments. OpenTrain helps you find relevant projects, apply efficiently, and develop...Hourly payPart timeFor contractorsRemote workWorldwideFlexible hours$24 per hour
...Trainer with advanced Tamil fluency to evaluate AI models' understanding of the Tamil language... ...clips, reviewing tones for cultural relevancy, and ensuring quality control in audio... ...compensation is offered at up to $24/hr for tasks, with flexible hours and remote work...Remote jobFlexible hours- ...Robot Policy Compliance Task Evaluator is a remote red-team track for stress-testing AI systems against adversarial prompts. Reviewers craft attack scenarios, document the failure mode, and pair each successful jailbreak with the rubric clause it violated so the safety...Remote jobHourly payFor contractors10 hours per week
- ...context to support a range of AI training and evaluation projects . In this role, you will work across diverse task types, including evaluating prompts and AI-... ...assess AI-generated outputs. Deliver culturally relevant and appropriate content by applying local...Extra incomeFull timeFor contractorsLocal area10 hours per week
- ...work.Work You’ll DoAs a Project - Program Analytics & Insights Evaluator IIon the project, you will be responsible for:Supporting the design... ...relationships Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environmentStrong interpersonal skills...Local area
- ...individual differences and with psychological regulatory processes affecting individual and interpersonal functioning.• Familiarity with relevant ethical and professional standards of practice in an inpatient setting• Good oral communication skills and an ability to deliver...Work experience placementWork at officeLocal areaRemote work
$55k
...projects Compiles and analyzes data, taking into consideration all relevant factors according to appraisal industry standards (e.g.,... ...skills with the ability to make progress on multiple duties and tasks simultaneously and work in high-pressure situations...Full timeTemporary workWork experience placementRemote workFlexible hours- ...TechnicalPer DiemAll ShiftsvariedJob SummaryThe Mental Health Evaluator (MHE) is a critical role dedicated to delivering comprehensive... ...Program Manager for the Crisis Care Division.RESPONSIBILITIESGather relevant psychosocial and diagnostic information using available...
$14.5 per hour
...AI Web Search Evaluator Unlock the Power of the Internet! Are you curious, tech-savvy, and passionate about improving online search... ...search results, ensuring users receive the most accurate and relevant information. Provide feedback to improve search engine algorithms...Bi-weekly payHourly payPart timeImmediate startRemote workWork from homeFlexible hours$11.5 per hour
...A leading educational institution is offering a part-time remote position as an Online Task Contributor. In this role, you will evaluate and provide feedback on content to enhance search engine results and quality. No prior experience is needed, but proficiency in English...Hourly payPart timeRemote work- ...infrastructure engineering backgrounds to test and evaluate AI-assisted workflows across modern... ...coding or productivity agents is highly relevant. Responsibilities Test and validate... ...and virtual discussions to clarify task requirements. Evaluate whether AI-generated...Temporary work
- ...Career-Service Social ServicesProfessional (Substance Use Disorder Evaluator) to provide substance use disorder diagnosis, assessment, and... ...440-22-240. Two (2) years of experience in the last 5 years, relevant work experience with people involved in the criminal legal...Contract workTemporary workWork experience placementWork at officeRemote workTrial periodMonday to FridayFlexible hoursNight shift
- A leading research accelerator is seeking a contractor to evaluate North American teen humor. The role involves reviewing short-form content, rating based on cultural relevance, and explaining humor dynamics clearly. Ideal candidates are 18 to 19 years old, familiar with...Remote jobContract workFor contractorsFreelanceFlexible hours
- About the role We are hiring expert Evaluators in Brand / creative direction / marketing collateral to review and assess AI-generated work... ..., hourly engagement. Requirements (must have) 5+ years of relevant professional experience in Brand / creative direction / marketing...Hourly payWork at officeRemote work
- ...by our partnership with EQT. Website: Linkedin Job Title: Evaluator - Political Science Location: Remote (USA) Job Type: Contract... ...processing and comply with data protection protocols and other relevant laws and guidelines. Requirements PhD or Masters in Political...Contract workRemote workWorldwide
- ...Government Policy and Legislative Research Evaluator is a remote review track for evaluating... ...at least 10 hours per week. Example tasks Verify the citations in a model's... ...admission or legal-equivalent credential in a relevant jurisdiction. Prior experience...Remote jobHourly payFor contractors10 hours per week
- ...Patent Claim AI Evaluator is a remote review track for evaluating AI outputs in patent law... ...at least 10 hours per week. Example tasks Verify the citations in a model's patent... ...or legal-equivalent credential in a relevant jurisdiction. Prior experience reviewing...Remote jobHourly payFor contractors10 hours per week
- Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality... .... This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of...Remote jobWork at office
$14.5 per hour
Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology companies to provide datasets that are high-quality, ethically sourced, relevant, diverse, and scalable to supercharge their AI models. As a Welocalize brand, Welo Data leverages...Hourly payPart timeImmediate startRemote workWork from home10 hours per weekFlexible hours- Obsidian is hiring expert Evaluators in Healthcare operations for a remote, hourly engagement. You will review AI-generated work products... ...in the field. The ideal candidate has over 5 years of relevant experience, outstanding proficiency in Microsoft Office and Google...Remote jobHourly payWork at office
$15 - $16 per hour
...partner is looking for a German Language Data Evaluator based in United States. This is a... ...level of accuracy, consistency, and relevance. Apply strong linguistic judgment when... ...project performance. Manage assigned tasks efficiently while maintaining accuracy and...Remote jobHourly payFull timeWork from homeMonday to FridayFlexible hours$80 - $120 per hour
...Incident Management / Reliability / SRE Evaluator based in the United States. This remote... ..., incident response, and SRE-related tasks. The engagement is flexible and fully... ...institution. Requirements ~5+ years of relevant professional experience in incident...Remote jobHourly payWeekly payContract workFor contractorsH1bWork at officeFlexible hours$14.5 per hour
...wondered how search engines decide what's relevant and what isn't? At Welo Data, we're... ...comfort of your home.WHAT YOU'LL DO: • Evaluate search results and rate how relevant and... ...about content quality • Complete online tasks using clear, structured guidelines — no guesswork...Remote jobBi-weekly payPart time$20 per hour
...is hiring a Digital Web Designer. In this remote role, you will evaluate AI-generated designs and provide feedback to enhance the model’... ...offers flexible hours and pays $20+ USD/hr for general projects and $40+ for design-focused tasks. #J-18808-Ljbffr DataAnnotationRemote jobFlexible hours- Mercor is seeking expert Evaluators in Clinical / biomedical / pharma to review and assess AI-generated work products (documents, spreadsheets... ...written feedback. Ideal candidates have 5+ years of relevant experience, native or professional English fluency, and strong...Remote jobHourly pay
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Retrieval Relevance Task Evaluator [Remote]. Be the first to apply!



