AI Agent Evaluator - Remote
YO AI Labs
- Remote job
Job Description
Job Description
Role Title: AI Agent Evaluator
Role Type: Contractor
Location: United States
We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. You'll test two AI agents side by side, assess their performance, and provide feedback to improve their reliability and user experience.
Key Responsibilities- Test AI agents on everyday tasks, including shopping, travel, bookings, bill payments, email follow-ups, and forms.
- Approve or stop agents at key steps.
- Screen-record each testing session.
- Score task completion, quality, control issues, user satisfaction, and time.
- Provide clear written feedback on agent performance and usability.
- Based in the United States with a bachelor's degree.
- Regularly use AI tools in daily life.
- 2–3 years of experience, preferably in engineering, business management, or office operations.
- Strong attention to detail and ability to follow instructions.
- Excellent written communication and documentation skills.
- An active personal AI account (Free, Go, or Plus; not Pro) and a smartphone capable of screen recording.
Preferred: Relevant domain expertise. No coding or AI development experience is required.
Selection Process- Submit an application and complete screening questions.
- Attend a 15–30-minute interview.
- Complete profile review and onboarding.
- Task duration: Approximately 1.5 hours per run, including evidence and scoring.
- Kickoff: Monday, October 12.
- Onboarding: Before the project begins, subject to scheduling.
- AuraOne is seeking an Everyday AI Agent Tester specialist operations contractor for remote US-eligible work. You will review AI outputs against workflows, policy, and playbooks, grade tone and escalation, and document next steps for model training. Ideal candidates have...Remote jobFor contractors
$80 per hour
A leading AI consultancy is searching for QA analysts to validate AI agents in a project focused on improving complex tasks. This opportunity is a flexible, remote, part-time role ideal for curious thinkers... ...identify inconsistencies in AI evaluation. Competitive compensation...Remote jobPart timeFlexible hours$20 - $40 per hour
Put AI agents through realistic multi-step jobs and judge whether they actually finish them. Your scenarios, ratings and notes show developers... ...to follow detailed guidelines and work independently Contract Remote, hourly contract (independent contractor) Part-time, flexible:...Remote jobHourly payWeekly payContract workPart timeFor contractorsFlexible hours- AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining. The role focuses on auditing...Remote job
- OpenTrain AI, Inc. is seeking individuals to remotely test two AI agents on everyday tasks using a smartphone. As a part-time contractor in the United States, you will guide agents, screen record sessions, and provide detailed feedback on completion, control, and user...Remote jobPart timeFor contractors
$60 per hour
A leading AI company is seeking QAs for autonomous AI agents to ensure quality assurance in complex task evaluations. This remote project-based role is ideal for individuals who excel in analytical thinking and have experience in evaluating system logic. The opportunity...Remote jobFlexible hours$20 - $40 per hour
...seeking detail-oriented contributors to craft realistic AI task scenarios and evaluate agent performance. You will describe steps, judge outcomes, and... ...to help developers improve automation. The role is remote, part-time, hourly, 10-40 hours per week, paid weekly in...Remote jobHourly payWeekly payPart time10 hours per week$80 per hour
...technology company is seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong analytical thinking,... ...expected behaviors for AI agents. This flexible remote opportunity allows you to work on your own schedule while...Remote jobPart timeFlexible hours- ...AI Agent Evaluator is a remote evaluation track for reviewing ai agent evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain...Remote jobHourly payFor contractors10 hours per week
$25 - $45 per hour
...as bills, returns, bookings, groceries and travel through two AI agents side by side, using their own accounts on their phone. They approve... ..., step by step. Listed pay: $25 to $45 per hour. Where: Remote, US-based. Time: About 1.5 hours per run, on your own schedule....Remote workHourly payContract workWork at office$14.5 per hour
A technology company is seeking an AI Web Search Evaluator to enhance the quality of search engine results. This flexible, remote, part-time role focuses on analyzing search performance and providing feedback to improve algorithms. Ideal candidates will have strong analytical...Remote workHourly payPart timeFlexible hours$14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology companies to provide datasets that are... ...optimization or data analysis is beneficial. Seeks a flexible, remote job that can easily fit around other commitments. Pay...Remote workHourly payPart timeImmediate startWork from home10 hours per weekFlexible hours- ...Alignerr Corp. is seeking Creative Writing Evaluators to shape AI training by judging AI-generated writing across fiction, poetry, essays, and... ..., voice, coherence, and reader engagement. Work is fully remote on an hourly contract, with 10–40 hours per week available....Remote workHourly payContract work10 hours per week
$20 per hour
...A tech company specializing in AI is hiring a Digital Web Designer. In this remote role, you will evaluate AI-generated designs and provide feedback to enhance the model’s understanding of aesthetics. An ideal candidate will have a strong background in UI/UX design and...Remote workFlexible hours$14.5 per hour
...AI Web Search Evaluator Welo Data works with technology companies to provide datasets that are high-quality, ethically sourced, relevant,... ...suits you—choose your hours and achieve work-life balance. Remote Work: Enjoy the freedom of working from home anywhere in...Remote workBi-weekly payHourly payPart timeImmediate startWork from homeFlexible hours$25 - $45 per hour
micro1 Inc. is seeking an AI Agent Evaluator contractor based in the United States. You will run real-life scenarios using two AI agents, screen record, and score results, providing detailed written feedback to help improve next‑gen AI agents. Onboarding is available today...Hourly payFor contractorsImmediate start$25 - $45 per hour
Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot Pay: $25-45/hour Time per task: ~1.5 hours per run (evidence and scoring included) Project kickoff: Monday, October 12 Onboarding: Today (Fri), Saturday, or Sunday, so you're...For contractorsWork at officeWeekend work- Welo Data is seeking Mandarin-speaking experts for a 4-week remote language evaluation project. You will evaluate AI-generated Mandarin content, write and rewrite prompts, and refine responses to improve dataset quality. This is a freelance contract with flexible weekly...Remote jobContract workFreelanceWork from homeFlexible hours
- OpenTrain AI, Inc. seeks an experienced 211 Specialist to evaluate AI answers for government services. This part-time, remote role focuses on reviewing questions, writing rubrics, and giving precise feedback to improve accuracy and usefulness. You will assess AI responses...Remote jobPart timeFlexible hours
- Prolific Academic Ltd. is seeking expert UX evaluators to assess AI-generated design work. You will judge usefulness, accessibility, and coherence... ...and user journeys, with clear reasoning. The engagement is remote and paid, subject to a brief skills assessment and...Remote job
- Prolific Academic Ltd is seeking a remote, on-demand Physics expert to evaluate AI-generated graphical abstracts of scientific papers. You will review, compare, and judge the accuracy of the summaries, applying deep physics knowledge to ensure fidelity to the underlying...Remote jobHourly payFlexible hours
- Prolific Academic Ltd is seeking individuals to help train and assess AI systems through paid, task-based work. Responsibilities include reviewing support conversations, drafting realistic customer tickets, and improving customer-facing guidance. A short skills assessment...Remote job
- prolificacademicltd is offering a remote, flexible-role focused on evaluating and improving AI-generated customer support content. You will assess model replies, create test tickets and dialogue flows, and review FAQs to ensure clarity and accuracy. The position requires...Remote jobFlexible hours
$40 per hour
...Ltd. is seeking a part-time Materials Science specialist to audit scientific papers and machine-generated graphical abstracts. This remote role in the US involves opt-in assignments based on client demand, with typical engagement at $40+ per hour and tasks usually under...Remote jobHourly payPart time$80 per hour
Develop and map analytic cases to defined taxonomies across multiple evaluation areas. Author complex test cases to evaluate model performance on source handling and analytic rigor. Assess and grade AI-generated responses based on reasoning, source use, and practical...Remote job- AuraOne, Inc. is seeking a remote Deception Behavior Risk Evaluator to stress-test AI systems with adversarial prompts. You will design jailbreak-like prompts, document failures with reproduction steps, and map each incident to the applicable policy clause to enable rapid...Remote jobFor contractors
- A tech-focused company is seeking a qualified LLM - AI Quality Analyst Personalization, fluent in Italian, to evaluate and improve AI model responses. In this remote role, you will analyze AI outputs, design multi-turn prompts, and provide structured feedback to enhance...Remote jobFlexible hours
- Outlier is seeking detail-oriented English speakers to review AI-generated media from home on a flexible schedule. You will roleplay live conversations with AI voice models, score exchanges, and explain in writing what looked or sounded off. You will compare image and...Remote jobFlexible hours
$50 per hour
prolificacademicltd is seeking a product design and UX expert to help train and evaluate AI models through paid, task-based work. You’ll review generated... .... A short skills assessment is part of the selection process. Remote, flexible work with payments per task, rates up to $50/hour...Remote jobFlexible hours- Turing is seeking detail-oriented AI Analysts based in the United States for a Google Wallet evaluation project. This role allows you to engage with advanced AI tools while contributing to the future of AI. You will evaluate model responses, review output quality, and provide...Remote jobFull timeContract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluator - Remote. Be the first to apply!
- tsa agent New York, NY
- import export agent New York, NY
- cruise agent New York, NY
- agent assistant New York, NY
- telemarketer - state farm agent team member New York, NY
- agent New York, NY
- showing agent New York, NY
- state farm agent work from home New York, NY
- right of way agent New York, NY
- enrolled agent New York, NY

