AI Agent Evaluator Remote
Full-time
YO AI Labs
Role Title: AI Agent Evaluator
Role Type: Contractor
Location: United States
We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. Youll test two AI agents side by side assess their performance and provide feedback to improve their reliability and user experience.
Key Responsibilities
- Test AI agents on everyday tasks including shopping travel bookings bill payments email follow-ups and forms.
- Approve or stop agents at key steps.
- Screen-record each testing session.
- Score task completion quality control issues user satisfaction and time.
- Provide clear written feedback on agent performance and usability.
Required Qualifications
- Based in the United States with a bachelors degree.
- Regularly use AI tools in daily life.
- 23 years of experience preferably in engineering business management or office operations.
- Strong attention to detail and ability to follow instructions.
- Excellent written communication and documentation skills.
- An active personal AI account (Free Go or Plus; not Pro) and a smartphone capable of screen recording.
Preferred: Relevant domain expertise. No coding or AI development experience is required.
Selection Process
- Submit an application and complete screening questions.
- Attend a 1530-minute interview.
- Complete profile review and onboarding.
Project Details
- Task duration: Approximately 1.5 hours per run including evidence and scoring.
- Kickoff: Monday October 12.
- Onboarding: Before the project begins subject to scheduling.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluator Remote in New York, NY vacancy
- ...Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Required Skills: ChatGPT Muse AI agents Attention to Detail Following Instructions Screen Recording About the role: AI agents are starting to handle real-life errands...Remote jobContract workFor contractorsWork at officeWeekend work
$45 per hour
# AI Agent Evaluator$45per hourContractor Remote Last verified 21 Jul 2026Listed on micro1 micro1.aiApplyAffiliate disclosureWork Expert (WE) is an independent publishing and referral website. We are not a recruiter, hiring manager, agent or employer, and we are not affiliated...Remote work- ...Job Description Job Description Role Title: AI Agent Evaluator Role Type: Contractor Location: United States We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. You'll test two AI agents side by side, assess their...Remote jobFor contractorsWork at office
YO AI Labs
New York, NY5 days ago - RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction...Remote jobFor contractorsStart working today
- AuraOne, Inc. is seeking a remote AI data reviewer to evaluate agent evaluation outputs against a versioned rubric. You will label edge cases, provide rationales, and help calibrate review quality across long batches. Ideal candidates have prior evaluation or annotation...Remote jobFor contractorsFlexible hours
- ...Dover is seeking an AI Agent Evaluator contractor based in the United States to test two AI agents by performing real-life tasks and recording... ...steps, note any issues, and rate performance. This role is remote and offers flexible scheduling. #J-18808-Ljbffr Jobleads-USRemote workFor contractorsFlexible hours
$80 per hour
...technology company is seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong analytical thinking,... ...expected behaviors for AI agents. This flexible remote opportunity allows you to work on your own schedule while...Remote jobPart timeFlexible hours- ...IT Helpdesk Agent Evaluator is a remote evaluation track for reviewing it helpdesk agent evaluation prompts and responses against AuraOne's quality... ...team can use to retrain. Why this role matters AI data reviewers help turn it helpdesk agent evaluation outputs...Remote jobHourly payFor contractors10 hours per week
$25 - $45 per hour
...last step is done. If you do not see it check your spam folder. This is a fully remote position open to candidates based in United States. Pay: $25$45/hr Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot Pay...Remote workFull timeFor contractorsWork at officeWeekend work- ...YO AI Labs is seeking an Everyday AI Agent Tester based in the United States to evaluate AI agents performing common tasks. You will compare two agents side by side, document results, and provide actionable feedback to improve user experience and reliability. Responsibilities...
- ...MERIT Beauty is seeking Turkish-speaking annotators for a contract role evaluating AI-generated content. You will review content for coherence, consistency, and alignment with real-world expectations in Turkish. The ideal candidates are native or fluent Turkish speakers...Remote workContract workTemporary workImmediate start
MERIT Beauty
United States5 days ago $14.5 per hour
...diverse, and scalable to supercharge their AI models. As a Welocalize brand, Welo Data... ...insights on relevance and quality. Evaluate and rate the effectiveness of search engine... ...analysis is beneficial. Seeks a flexible, remote job that can easily fit around other commitments...Remote workHourly payPart timeImmediate startWork from home10 hours per weekFlexible hoursKanz.us
Dallas, TX2 days ago$14.5 per hour
A technology company is seeking an AI Web Search Evaluator to enhance the quality of search engine results. This flexible, remote, part-time role focuses on analyzing search performance and providing feedback to improve algorithms. Ideal candidates will have strong analytical...Remote workHourly payPart timeFlexible hoursWelo Data
United States5 days ago- RWS Group is seeking a remote evaluator to help improve AI search results by providing detailed feedback on text, audio, images, and videos. You’ll work from home, simulating everyday searches to influence how major search engines respond to real users. Ideal candidates...Remote jobWork from home
$20 - $26 per hour
Prolific is seeking a fluent Punjabi speaker to act as an AI trainer/evaluator. You will listen to audio and compare it with text, rating naturalness and authenticity. The role requires strong linguistic intuition and the ability to provide precise feedback to help models...Remote jobHourly payWork from homeFlexible hours- RWS Group is seeking a Speech AI Evaluation Specialist to support the improvement of AI-generated content in Chinese Simplified (USA). This freelance, part-time position offers remote work from home in the USA, with a flexible schedule at about 10+ hours per week and immediate...Remote jobPart timeFreelanceImmediate startWork from home10 hours per weekFlexible hours
$50 per hour
Prolific seeks Product Designers and UX Specialists to help train AI models using your expertise. You'll evaluate AI-generated designs and ensure usability and accessibility while working from home. Ideal candidates hold a BS, MS, or PhD in a relevant field and have at...Remote jobWork from homeFlexible hours- RWS TrainAI seeks part-time, remote Search Quality Raters to test and refine how client search engines respond to everyday queries. Work... ...it ideal for freelancers, students, or flexible workers seeking to contribute to AI model improvements. #J-18808-Ljbffr RWS TrainAIRemote jobHourly payPart timeFreelanceWork from homeFlexible hours
$60 per hour
Prolific is seeking Chemistry Experts and Chemical Engineers to join our Expert Network to train and evaluate AI models using real-world chemical knowledge. A quick 10- to 15-minute test will assess your skills; successful candidates join Prolific as participants and are...Remote jobHourly payWork from homeFlexible hours- RWS TrainAI is seeking a flexible part-time Search Quality Evaluator to join our remote team. You will assess how everyday searches are handled and provide feedback to improve AI-powered results for global users. Ideal for students or freelancers in Poland, with a strong...Remote jobHourly payPart timeFreelanceWork from homeFlexible hours
- RWS is seeking a part-time Search Quality Evaluator to help improve AI-driven search results. You will test how search engines respond to everyday queries from regular users, provide feedback on text, audio, images, and videos, and help refine data used by major platforms...Remote jobPart timeWork from homeFlexible hours
- ...Prolific is seeking Product Designers and UX Specialists to join our Expert Network, contributing to the training and evaluation of cutting-edge AI models. This role requires expertise in usability, design systems, and user research, providing essential feedback on design...Remote workWork from homeFlexible hours
Prolific
Tucson, AZ2 days ago - ...Prolific is seeking talented Product Designers and UX Specialists to join our Expert Network for training and evaluation of cutting-edge AI models. Candidates should have a relevant educational background and at least one year of experience in design-related fields. Responsibilities...Remote workWork from homeFlexible hours
Prolific
Arizona City, AZ3 days ago - Benture is seeking individuals to evaluate AI assistants live during MLB postseason games. Work remotely from the United States, on scheduled game days, asking questions and scoring AI responses for accuracy, usefulness, and clarity. Each task lasts about 2 to 2.5 hours...Remote job
- Synthires is seeking experienced Nurse Practitioners and advanced practice nursing professionals to join a remote project in the United States. You will evaluate AI-driven clinical content and provide expert feedback to improve accuracy and reliability of patient care...Remote job
$20 - $30 per hour
A remote-focused technology firm is looking for an individual proficient in evaluating large language model outputs. The role involves assessing AI systems, reviewing workflows, and providing actionable feedback to enhance product quality. Ideal candidates will have strong...Remote jobHourly pay- Cincinnatus LLC is seeking a Behavioral Health Expert for a remote, part-time role in the United States. You will review conversations on relationships, emotional wellbeing, and beliefs, assessing neutrality, safety and appropriateness while documenting rationale. You’...Remote jobPart time
- AI Trainer Jobs is seeking a Mental Health Clinical AI Evaluator to remotely review AI outputs affecting clinical safety and guideline adherence. You will assess differential reasoning, dosing logic, and flag safety issues, documenting corrected reasoning for model improvement...Remote jobHourly payPart time
$40 per hour
Prolific is recruiting Materials Science Specialists to join its Expert Network and contribute real-world expertise to train and evaluate cutting-edge AI models. If selected, you’ll be invited to join Prolific as a participant and paid to train and evaluate powerful AI...Remote jobHourly pay- A virtual AI evaluation firm is seeking individuals to review and evaluate AI-generated responses in therapeutic conversations. The ideal... ...This is a project-based role with immediate onboarding and a remote work environment, making it ideal for candidates looking for flexibility...Remote jobImmediate start
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluator Remote. Be the first to apply!
Related searches