Remote AI Agent Evaluator & Rubric Specialist
AuraOne, Inc.
New York, NY
- Remote job
AuraOne, Inc. is seeking a remote AI data reviewer to evaluate agent evaluation outputs against a versioned rubric. You will label edge cases, provide rationales, and help calibrate review quality across long batches. Ideal candidates have prior evaluation or annotation experience, strong attention to detail, and reliable weekly availability for async work. This contractor role offers flexible remote work and requires US eligibility. #J-18808-Ljbffr AuraOne, Inc.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Remote AI Agent Evaluator & Rubric Specialist in New York, NY vacancy
- YO AI Labs seeks a former U.S. Consular Officer or Passport Specialist to evaluate AI-generated guidance on passport and citizenship matters. This remote, contractor role asks for 20-30 hours per week and... ...reviewing questions, developing rubrics for complex cases, and...Remote jobFor contractors
$80 - $120 per hour
...technical talent with leading AI research labs. Headquartered in... .../ community programs Evaluator Type: Contract Compensation... ...$80–$120/hour Location: Remote Role Responsibilities Evaluate... ...domain-specific quality rubrics. Identify factual, aesthetic...Remote workContract workSummer workWork at officeMercor
Atlanta, GA9 days ago- AI Trainer Jobs is seeking a remote Database and SQL AI Evaluator to review prompts and responses against AuraOne's quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for retraining. Role details: freelance contractor, hourly...Remote jobHourly payFor contractorsFreelance
- ...Job Description Job Description Role Title: AI Agent Evaluator Role Type: Contractor Location: United States We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. You'll test two AI agents side by side, assess their...Remote jobFor contractorsWork at office
YO AI Labs
New York, NY3 days ago - ...Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Required Skills: ChatGPT Muse AI agents Attention to Detail Following Instructions Screen Recording About the role: AI agents are starting to handle real-life errands...Remote jobContract workFor contractorsWork at officeWeekend work
$45 per hour
# AI Agent Evaluator$45per hourContractor Remote Last verified 21 Jul 2026Listed on micro1 micro1.aiApplyAffiliate disclosureWork Expert (WE) is an independent publishing and referral website. We are not a recruiter, hiring manager, agent or employer, and we are not affiliated...Remote work- RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction...Remote jobFor contractorsStart working today
- OpenTrain AI, Inc. is seeking a remote, part-time contractor to test two AI agents on a smartphone, completing realistic errands and recording outcomes. You will evaluate task completion, quality, control issues, user satisfaction, and time, then provide clear feedback...Remote jobPart timeFor contractorsFlexible hours
- ...Dover is seeking an AI Agent Evaluator contractor based in the United States to test two AI agents by performing real-life tasks and recording... ...steps, note any issues, and rate performance. This role is remote and offers flexible scheduling. #J-18808-Ljbffr Jobleads-USRemote workFor contractorsFlexible hours
- 24-MAG LLC seeks experienced Educator/ Assessor Writers for remote evaluation of AI-generated written responses. You will apply structured rubrics, assess clarity and relevance, and provide detailed rationales to ensure consistent quality judgments across tasks. Ideal...Remote job
- ...IT Helpdesk Agent Evaluator is a remote evaluation track for reviewing it helpdesk... ...against AuraOne's quality rubric. Reviewers compare paired outputs... ...Why this role matters AI data reviewers help turn it... .... Remote · Independent specialist contractor. Employment type...Remote jobHourly payFor contractors10 hours per week
- AuraOne seeks an experienced Senior Journalist / Writer for a remote evaluation track. You will review prompts and responses, compare paired outputs, label edge cases, and provide structured feedback for retraining. You’ll help turn outputs into auditable labels, rationales...Remote jobHourly payFor contractors
$50 per hour
...Inc. seeks a Video Editing Professional — AI Video & Post-Production Review to remotely assess model outputs against the company rubric. You will compare paired responses, label... ...50 per hour. Ideal candidates have prior evaluation or annotation experience in video editing...Remote jobHourly payFor contractors- ...connects elite creative and technical talent with leading AI research labs. The company is headquartered in San... ...a AI Safety Practitioner for a contract role with remote work options. The role focuses on evaluating AI-generated responses for safety, accuracy, and policy...Remote jobContract workSummer work
$80 per hour
...technology company is seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong analytical thinking,... ...expected behaviors for AI agents. This flexible remote opportunity allows you to work on your own schedule while...Remote jobPart timeFlexible hours- ...Assessment Rubric AI Evaluator is a remote review track for evaluating AI outputs across assessment rubric ai specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next...Remote jobHourly payFor contractorsWork experience placement10 hours per week
$25 - $45 per hour
...last step is done. If you do not see it check your spam folder. This is a fully remote position open to candidates based in United States. Pay: $25$45/hr Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot Pay...Remote workFull timeFor contractorsWork at officeWeekend work$80 - $120 per hour
AuraOne is seeking a remote Pricing / ROI / revenue economics Evaluator to review prompts and model outputs against our quality rubric. You will compare paired responses, label edge cases, and provide structured feedback to retrain the model. As an independent contractor...Remote jobHourly payFor contractors10 hours per week- AI Trainer Jobs seeks a remote independent contractor to review AI outputs for fintech operations evaluation. You will assess workflow adherence, tone, and escalation logic, assigning severity... ...and the ability to apply multi-page rubrics consistently. This contractor position...Remote jobHourly payFor contractorsFlexible hours
- YO AI Labs is seeking an AI Evaluation Specialist to remotely assess enterprise AI outputs against detailed rubrics, measuring accuracy, relevance and quality. You will provide concise, actionable feedback to drive improvements. Responsibilities include identifying reasoning...Remote jobFor contractors
- Rex.zone is seeking a Senior AI data annotator to perform data labeling and evaluation for NLP tasks, RLHF assessments, and... ...performance. This is a US-based remote, full-time role aligned with... ...talent demand. You will apply strict rubrics, document edge cases, and...Remote jobFull time
- AuraOne is seeking a Primary Care Physician (MD) for a remote Clinical Documentation & AI Evaluation track. Review AI outputs, assess differential... ...clinical writing, and the ability to apply multi-page rubrics to batches. You will work asynchronously for at least...Remote jobFor contractors
- ...About the role Registered Nurse AI Evaluator is a remote clinical-review track for evaluating AI outputs... ...red-flag handling on a structured rubric. Flag patient-safety issues with... ...review Work model Remote · Independent specialist contractor Compensation Hourly rate confirmed...Remote workHourly payFor contractors10 hours per week
AI Trainer Jobs
New York, NY21 hours ago - ...Visual Document Understanding contractor to support AI model evaluation in the real estate domain. This remote, 15-20 hours/week engagement emphasizes rigorous... ...You will review complex tasks, develop evaluation rubrics, and collaborate with subject matter experts to ensure...Remote jobFor contractors
- Mercor is seeking a contract Evaluator in the humanities/arts domain to assess AI-generated artifacts against domain rubrics. Remote work arrangement allows asynchronous collaboration with researchers. You will identify factual, aesthetic, and presentation errors in documents...Remote jobContract work
- Mercor seeks a contract evaluator for general business strategy to assess AI-generated artifacts against domain-specific rubrics. You will work remotely, evaluate quality, and provide structured written feedback to help improve AI model performance. You will collaborate...Remote jobContract work
$30 - $40 per hour
OpenTrain AI, Inc. is seeking a Passport Application AI Evaluation Expert for a remote, short-term contract. You will review passport questions for realism, create rubrics for accurate answers, and define evaluation standards for identity documents, renewals, and minors...Remote jobHourly payTemporary work- Mercor, a San Francisco-based firm, is hiring a remote Evaluator to review AI-generated artifacts in media, journalism, and communications. You will assess quality against domain rubrics, identify factual, aesthetic, and presentation errors, and deliver clear structured...Remote job
- YO AI Labs is seeking an experienced AI Evaluation Specialist to remotely assess AI assistant outputs for an enterprise training project. The contractor will evaluate responses against detailed rubrics, identify issues, and provide actionable feedback to improve AI systems...Remote jobFor contractors
- Mercor is seeking expert Evaluators in Procurement / vendor management to review AI-generated documents, spreadsheets, and... ...decks for accuracy and rigor. This remote, hourly engagement requires deep... ...written feedback aligned with quality rubrics. #J-18808-Ljbffr MercorRemote jobHourly pay
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote AI Agent Evaluator & Rubric Specialist. Be the first to apply!
Related searches