Repository Navigation Evaluation Specialist [Remote]
AuraOne Human Data
Remote
- Remote job
Repository Navigation Evaluation Specialist is a remote review track for evaluating AI outputs across repository navigation evaluation specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next step so the modeling team can train on it.
Why this role matters
Repository Navigation Evaluation specialist operations AI has to fit into an actual day at work. AuraOne uses experienced operators to grade outputs the way a senior peer would — checking workflow, policy, and the unwritten rules that decide whether a task actually gets done.
Responsibilities
- Review AI outputs against current repository navigation evaluation specialist operations workflows, playbooks, and firm policy for Repository Navigation Evaluation Specialist assignments.
- Grade tone, escalation logic, and stakeholder fit on a structured rubric.
- Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
- Capture the right next step so the modeling team can train on it.
- Adjudicate disputed treatments against published playbooks or firm guidance.
- Maintain reviewer-quality scores in inter-rater calibration cycles.
Qualifications
- Direct working experience in repository navigation evaluation specialist operations on real teams for Repository Navigation Evaluation Specialist work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the policy or workflow being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Grade a model's response to a real repository navigation evaluation specialist operations ticket and rate workflow, tone, and escalation.
- Flag a missed escalation with the right severity tag and corrected next step.
- Adjudicate a disputed playbook call between two reviewers using firm guidance.
- Audit a 25-row batch for rubric consistency and report drift to the program lead.
Nice to have
- Prior experience training, calibrating, or QA-ing operations teams.
- Familiarity with AI-assisted workflow tooling and its failure modes.
- Bilingual experience for cross-region operations.
Skills
- Operational review
- Policy adherence
- Workflow judgment
- Stakeholder communication
- Repository Navigation Evaluation specialist operations
- Code review
- Software testing
- Agent evaluation
- Repository
- Navigation
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
Vacancy posted 13 days ago
Similar jobs that could be interesting for youBased on the Repository Navigation Evaluation Specialist [Remote] in Remote vacancy
$90.58k - $114.59k
Position InformationNY HELP: YesAgency: Mental Health, Office ofTitle: Mental Hygiene Program Evaluation Specialist 3 (NY HELPS), Kingsboro Psychiatric Center, P27987Occupational Category: Health Care, Human/Social ServicesSalary Grade: 23Bargaining Unit: PS&T - Professional...SuggestedPermanent employmentFull timeWork at officeRemote workState of New York, USA
Albany, NY1 day ago$5,540 - $5,780 per month
...Reporting to the Director of Graduate Admissions and Program Evaluations (GAPE) in the College of Graduate Studies, this position is responsible... ...are completed efficiently and on schedule. The Evaluation Specialist is responsible for interpreting and applying complex policies...SuggestedPermanent employmentFull timeWork experience placementInternshipWork at officeVisa sponsorshipThe California State University
California, MO1 day ago- ...Program Evaluation SpecialistLocation: Nashville, Tennessee - 37243 Contract: 12+ months from the projected start Position Description: The Program Evaluation Lead is responsible for working with the management team to develop and implement evaluation and data collection...SuggestedContract workWork experience placementWork at office
Software Technology Inc
Nashville, TN1 day ago - ...Evaluation Specialist UN - United Nations Remote | Multiple locations CON level Speaks English, French Application deadline: December 31, 2026 (3 months) The Evaluation Specialist will conduct evaluations of various projects and programs for the UN Secretariat. This...SuggestedWork at officeRemote workShift work
UN - United Nations
United States3 days ago $70k - $140k
...Program Evaluation Specialist - Department of State The Position: We are seeking driven, thoughtful candidates with experience in program evaluation to support impactful initiatives for our Federal clients at the U.S. Department of State (DOS). As a Program Evaluation...SuggestedFull timeWork at officeRemote workWork from homeFlexible hoursNight shiftCenseo Consulting Group
Washington DC3 days ago$70k - $80k
...Program Evaluation SpecialistFully Remote • AFT Headquarters - WASHINGTON, DC 20006OverviewSalary Range $70,000.00 - $80,000.00 Salary... ...Position Type Full TimeDescriptionJob purposeThe Program Evaluation Specialist will help lead program impact evaluation efforts at AFT. The...Full timeCasual workLocal areaRemote workWeekend workAfternoon shiftAmerican Farmland Trust
Washington DC1 day ago- AuraOne is seeking a remote evaluation specialist to review application-user outputs for the DaVinci Resolve on macOS Generalist track. You will compare paired responses, label edge cases, and provide structured feedback the modeling team can use to retrain. As a contractor...Remote jobFor contractors
$5,540 - $5,780 per month
...Reporting to the Director of Graduate Admissions and Program Evaluations (GAPE) in the College of Graduate Studies, this position is responsible... ...are completed efficiently and on schedule. The Evaluation Specialist is responsible for interpreting and applying complex policies...Permanent employmentFull timeWork experience placementInternshipWork at officeImmediate startVisa sponsorshipSan Jose State University
San Jose, CA3 days ago$80k - $130k
...given the means and mentorship needed to succeed, and your creativity will be rewarded.About the PositionThe Monitoring and Evaluation (M&E) Specialist will provide technical support to the Office of Monitoring and Evaluation (OME), Division of Global Health Protection (...Full timeTemporary workWork at officeRemote workDexis Consulting Group
Washington DC4 days ago- ...Government Agencies. Learn More About ProSidian Consulting at .Job DescriptionProSidian Seeks a Infrastructure Project Evaluation and Coordination Specialist [DOE0139138] for Program Support on a Exempt W2: No Overtime Pay Basis Contract Contingent generally located across...Full timeContract workTemporary workFor contractorsFor subcontractorWork at officeRemote workFlexible hours
Prosidian Consultng
Richland, WA2 days ago $80k - $117.5k
...scientific discovery, and reach those in need. For more information, please visit Position Summary & Role The Senior Specialist, Technical Evaluation and Proposals role is an integral member of the Commercial Operations team responsible for generating proposals across...Full timeContract workLocal areaWork visaFlexible hours$45 - $70 per hour
...Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Behavioral Health Expert, AI Safety and Model Evaluation Type: Contract Compensation: $45–$70/hour Location: Remote Commitment: 20–40 hours/week Role...Contract workSummer workRemote workWeekday workMercor
New York, NY15 days ago$90 per hour
...DOM impact. ~ Professional written English for task justification. Preferred Prior RLHF , preference-labeling, model-evaluation, or structured code-review work. Pixel-perfect design-to-code experience. Accessibility expertise ( ARIA , semantic landmarks...Contract workSummer workLocal areaRemote workMercor
San Francisco, CA26 days ago- ...Financial Navigator Ensemble is a leading provider of technology-enabled revenue cycle management solutions for health systems, including... .../Medicaid advocate/eligibility, state case/eligibility specialist or other applicable roles Prior Patient Access or Physician...Local areaImmediate startRemote work
- ...who truly fit their needs. About the Role We are seeking experienced Pediatric Hospitalists to support the training and evaluation of next-generation AI systems focused on inpatient pediatric care and clinical documentation. You will review medical records, structured...Contract work
- Spectraforce Technologies seeks a Patient Support Specialist (Therapy Navigator) for a remote, contract role of 5+ months. The role focuses on building patient relationships, educating on chronic therapies, and driving satisfaction through proactive support. Collaboration...Remote jobContract work
- AuraOne is seeking a Kannada Audio Recording Expert on a remote contractor basis to evaluate audio annotation model outputs against a rubric and provide structured feedback. You will compare responses, label issues, and help identify edge cases for retraining. The role...For contractorsRemote work10 hours per week
- AI Trainer Jobs is seeking an Albanian Bilingual Expert to remotely evaluate prompts and responses against AuraOne's quality rubric. Reviewers label edge cases, compare outputs, and write structured feedback to aid model retraining. You will work as a contractor, reviewing...Remote jobFor contractors
$28.85 per hour
...throughout our revenue cycle. As our Specialist III, you'll review contract... ...build our team’s contract repository into an organized,... ...how to effectively source and navigate these documents to extract what... ...how it works and what it evaluates, and obtain consent beforehand...Hourly payFull timeContract workRemote workHome officeFlexible hoursDay shift- Supervisory Responsibility: Research and Evaluation Team Application Closes: 25 January 2025 Resumption Date: 1 February 2025 Location: Hybrid Title: Evaluation Specialist Reports Directly To: Evaluation Lead Supervisory Responsibility: Research and Evaluation Team Application...Work at officeRemote work
$65k - $75k
...Job Posting Title: Evaluation Coordinator ---- Hiring Department: Special Education ----... ...management activities in coordination with data specialists. Communicate with project sites,... ...employee, log-in to Workday, navigate to your Worker Profile, click the Career...For contractorsWork at officeImmediate startThe University of Texas at Austin
Austin, TX3 days ago$29.43 - $45.39 per hour
...Description: The Cardiac/Pulmonary Rehab Specialist plays a vital role in helping patients... ...on designing, implementing, and evaluating individualized exercise and lifestyle management... ...heavy items. Often required to navigate crowded and busy rooms (full of equipment...Hourly payImmediate startRemote workIntermountain Health
Logan, UT23 hours ago- ...talent with leading AI research labs. Headquartered in San Francisco, this remote contract position seeks a Generalist Expert to evaluate AI-generated responses, strengthening reasoning and rigor in model outputs. Responsibilities include providing structured written...Contract workRemote work
- ...consulting firm in Colby, Kansas, is seeking a Nurse Practitioner or Physician Assistant to conduct examinations for veteran compensation evaluations. Responsibilities include conducting assessments, reviewing medical histories, and completing reports, all while collaborating...Remote jobWork from homeFlexible hours
- AuraOne is seeking a remote evaluation contractor to assess Korean generalist outputs against a quality rubric. You will compare paired responses, label edge cases, and provide structured feedback to aid retraining of the model. Ideal candidates have prior evaluation or...Remote jobFor contractors10 hours per week
- Xperteez Technology Pvt Ltd is seeking Humanities Evaluation Specialists for a remote contractor role. You will develop high-difficulty Q&A pairs to train AI models, using your humanities expertise to ensure rigorous reasoning and accuracy with strong source citations....For contractorsImmediate startRemote workFlexible hours
- AuraOne, Inc. is seeking a remote Video Editing Professional contractor to review evaluation prompts and responses against our quality rubric. You will compare paired outputs, assign severity tags, and provide structured feedback for retraining. This role supports the...Remote jobFor contractors
- AuraOne is seeking a Voice Actor: CX Agent Voice Cloning (Indonesian Bahasa - Female) for a remote evaluation track reviewing prompts and responses against our quality rubric. Reviewers label edge cases and provide structured feedback for retraining. You will compare paired...Remote jobFor contractors10 hours per week
$70k - $95k
...DayWhen you join the growing BILH team, you're not just taking a job, you’re making a difference in people’s lives.The Patient Navigation Specialist supports Beth Israel Lahey Health (BILH) and its Community Hospital Division by providing coordinated, responsive assistance...Full timeRemote work3 days per week- AI Trainer Jobs is seeking a Tamil Bilingual Expert for a remote evaluation track. Reviewers assess bilingual prompts and model outputs against AuraOne's rubric, provide structured feedback, and help retrain the system through auditable labels and rationales. You will...Remote jobFor contractorsFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Repository Navigation Evaluation Specialist [Remote]. Be the first to apply!
Related searches
- sewing specialist Remote
- graduation specialist Remote
- technology integration specialist Remote
- consumer insights specialist Remote
- vulnerability management specialist Remote
- programmatic specialist Remote
- evaluation specialist Remote
- escalation specialist Remote
- outsourcing specialist Remote
- educational technology specialist Remote