Remote AI Agent QA & Workflow Evaluator
AuraOne
New York, NY
- Remote job
AuraOne is seeking an Everyday AI Agent Tester specialist operations contractor for remote US-eligible work. You will review AI outputs against workflows, policy, and playbooks, grade tone and escalation, and document next steps for model training. Ideal candidates have hands-on experience with real teams performing everyday AI agent tester operations, can apply multi-page rubrics consistently, and communicate clear reasoning about policy or workflow. #J-18808-Ljbffr AuraOne
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Remote AI Agent QA & Workflow Evaluator in New York, NY vacancy
$80 per hour
A technology company is seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong analytical thinking, attention... ...expected behaviors for AI agents. This flexible remote opportunity allows you to work on your own...Remote jobPart timeFlexible hours- AI Trainer Jobs seeks a remote independent contractor to review AI outputs for fintech operations evaluation. You will assess workflow adherence, tone, and escalation logic, assigning severity tags and documenting next steps for model training. The role requires experience...Remote jobHourly payFor contractorsFlexible hours
- ...is seeking a seasoned sales professional to help evaluate and improve AI-generated sales content and workflows for technical products. You will review conversations... ...manufacturing, and scientific markets. This is a remote, hourly engagement with flexible hours. #J-18808-...Remote jobHourly payFlexible hours
$50 - $60 per hour
...and building careers in AI training and data... ..., test model behavior, evaluate responses, and identify... ...realistic data-analysis workflows. Your analytics judgment... ...and logically sound. Remote work available worldwide... ...rubric-based evaluation and QA Understanding of KPI...Remote workHourly payPart timeFor contractorsWorldwideFlexible hours$190 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...: $50–$190/hour Location: Remote Commitment: 20+ hours/week Role Responsibilities Evaluate AI systems on complex personal workflows, including personal health ,...Remote jobHourly payContract workFor contractorsSummer workTrial periodMercor
New York, NY13 days ago$45 - $65 per hour
...Group is seeking an experienced pharmacy technician to evaluate and improve AI medication workflows in healthcare settings. You will create realistic... ...safety, applying real-world workflow knowledge. This fully remote, asynchronous contract offers hourly compensation ($45...Remote jobHourly payContract work10 hours per weekFlexible hours- ...Job Description Job Description Role Title: AI Agent Evaluator Role Type: Contractor Location: United States We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. You'll test two AI agents side by side, assess their...Remote jobFor contractorsWork at office
YO AI Labs
New York, NY7 days ago - RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction...Remote jobFor contractorsStart working today
- Progressive Leasing is seeking a Forward Deployed Agent Builder to collaborate with business and product teams to design, build, and deploy AI agents that solve real-world workflows. This role sits at the intersection of agent engineering, product thinking, and stakeholder...Remote job
- AuraOne, Inc. is seeking a remote AI data reviewer to evaluate agent evaluation outputs against a versioned rubric. You will label edge cases, provide rationales, and help calibrate review quality across long batches. Ideal candidates have prior evaluation or annotation...Remote jobFor contractorsFlexible hours
- ...Compliance Workflow Agent Evaluator is a remote review track for evaluating AI outputs in compliance workflows. Reviewers grade citation accuracy, statutory reasoning, and policy adherence; flag risk; and document the corrected analysis so the modeling team can train on...Remote jobHourly payFor contractors10 hours per week
- ...IT Helpdesk Agent Evaluator is a remote evaluation track for reviewing it helpdesk agent evaluation prompts... ...to retrain. Why this role matters AI data reviewers help turn it helpdesk... ...IT Helpdesk Agent evaluation Workflow evaluation Tool-use review Operational...Remote jobHourly payFor contractors10 hours per week
- ...Dover is seeking an AI Agent Evaluator contractor based in the United States to test two AI agents by performing real-life tasks and recording... ...steps, note any issues, and rate performance. This role is remote and offers flexible scheduling. #J-18808-Ljbffr Jobleads-USRemote workFor contractorsFlexible hours
- Enterprise AI Agent Users is a remote review track for evaluating AI outputs across enterprise ai agent users specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next...Remote jobWork experience placement
- Invoca is seeking an AI Operations Lead to own internal agentic workflows and build agents using Claude and Gumloop. This hands-on role interacts with executives and... ...security, data governance, and practical adoption. Remote-first with US locations including Seattle area....Remote job
- AuraOne is seeking a remote VFX Artist / 3D Animator reviewer to evaluate AI outputs for 3D design workflows. You will assess craft, constraints, and tooling adherence, flag production-readiness issues, and document corrections to train the modeling team. This is a contractor...Remote jobFor contractors10 hours per week
- Benture is seeking individuals to evaluate AI assistants live during MLB postseason games. Work remotely from the United States, on scheduled game days, asking questions and scoring AI responses for accuracy, usefulness, and clarity. Each task lasts about 2 to 2.5 hours...Remote job
- Rex.zone is seeking a Senior AI data annotator to perform data labeling and evaluation for NLP tasks, RLHF assessments, and prompt QA to improve training data quality and model performance. This is a US-based remote, full-time role aligned with Miami talent demand. You...Remote jobFull time
- Mercor is seeking expert Evaluators in Procurement / vendor management to review AI-generated documents, spreadsheets, and slide decks for accuracy and rigor. This remote, hourly engagement requires deep domain expertise and fluency in English to assess quality and provide...Remote jobHourly pay
- Mercor is seeking experts in Spreadsheet QA and workbook maintenance to review AI-generated documents, spreadsheets, and slide decks for accuracy and quality. This is a remote, hourly engagement. Ideal candidates have 5+ years in Spreadsheet QA, fluent English, and strong...Remote jobHourly payWork at office
- Volga Partners is seeking an AI Language Quality Evaluator fluent in Greek and English to assess localization accuracy across UI texts and app screens. This freelance, task-based role offers flexible remote engagement with on-demand tasks and no guaranteed hours. As part...Remote jobFreelanceFlexible hours
- RemoteLeads seeks an experienced contractor to evaluate AI-generated artifacts and provide structured feedback. You will review documents... ...and SRE best practices to grade outputs. The role is remote, hourly, with a flexible schedule and weekly payments. Candidates...Remote jobHourly payWeekly payFor contractorsWork at officeFlexible hours
$180k - $200k
Arbiter is the AI-powered care orchestration... ...complex healthcare workflows that interface with... ...As a QA Engineer reporting... ...dropped calls, wrong agent responses, failed handoffs... ...analysis. Exposure to LLM evaluation, prompt testing, or... ...This role will be remote or onsite in our...Remote workWork experience placementWork at office- ...Job Title: Agentic AI / Multi-Agent Workflow Engineer Forward Deployed Engineer (FDE) Location... ...three days per week, with two days remote. Travel to Amazon customer locations... ...injection. Develop observability, evaluation, security guardrails, and reliability...Remote workFull timeFor contractors3 days per week
SoftPathTechnologies
Mountain View, CA3 days ago $50 - $70 per hour
Mercor is seeking a Generalist Expert to review and assess documents and AI-generated outputs. This contract-remote role requires a US/Canada base and strong written English, with flexible hours and independence across docs, slides, and sheets. Compensation ranges from...Remote jobHourly payContract workFlexible hours$15 - $30 per hour
Crossing Hurdles is seeking a Contractor (Remote) to perform realistic, everyday digital tasks using AI assistant tools under standardized conditions. Compensation... ...for accuracy, apply grading guidelines to evaluate responses, and test scenarios involving email, calendar...Remote jobHourly payFor contractors10 hours per week$60 per hour
...Missouri is seeking contributors for a part-time QA project focused on autonomous AI agents. This flexible remote opportunity requires strong analytical and... ...structured data formats. Candidates will review evaluation tasks, identify inconsistencies, and help define...Remote jobPart timeFlexible hours$25 - $45 per hour
...last step is done. If you do not see it check your spam folder. This is a fully remote position open to candidates based in United States. Pay: $25$45/hr Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot Pay...Remote workFull timeFor contractorsWork at officeWeekend work- Mercor is hiring experienced lawyers to help evaluate and improve cutting-edge AI systems. In this remote, full-time role, you will review AI-generated legal content and workflows to ensure accuracy and compliance. We seek lawyers with strong analytical skills and interest...Remote jobFull time
$90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...: $70-$90/hour Location: Remote Role Responsibilities Evaluate the quality and correctness of AI-assisted... ...developer tools for correctness, workflow soundness, and reasoning. Provide clear...Remote jobContract workSummer work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote AI Agent QA & Workflow Evaluator. Be the first to apply!
Related searches
- work from home chat agent New York, NY
- gate agent New York, NY
- telemarketer - state farm agent team member New York, NY
- executive protection agent New York, NY
- agent assistant New York, NY
- commissioning agent New York, NY
- registered agent New York, NY
- transfer agent New York, NY
- sourcing agent New York, NY
- state farm agent work from home New York, NY