Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Evaluator - Remote

YO AI Labs

New York, NY
  • Remote job

Job Description

Job Description

Role Title: AI Agent Evaluator

Role Type: Contractor
Location: United States

We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. You'll test two AI agents side by side, assess their performance, and provide feedback to improve their reliability and user experience.

Key Responsibilities
  • Test AI agents on everyday tasks, including shopping, travel, bookings, bill payments, email follow-ups, and forms.
  • Approve or stop agents at key steps.
  • Screen-record each testing session.
  • Score task completion, quality, control issues, user satisfaction, and time.
  • Provide clear written feedback on agent performance and usability.
Required Qualifications
  • Based in the United States with a bachelor's degree.
  • Regularly use AI tools in daily life.
  • 2–3 years of experience, preferably in engineering, business management, or office operations.
  • Strong attention to detail and ability to follow instructions.
  • Excellent written communication and documentation skills.
  • An active personal AI account (Free, Go, or Plus; not Pro) and a smartphone capable of screen recording.

Preferred: Relevant domain expertise. No coding or AI development experience is required.

Selection Process
  1. Submit an application and complete screening questions.
  2. Attend a 15–30-minute interview.
  3. Complete profile review and onboarding.
Project Details
  • Task duration: Approximately 1.5 hours per run, including evidence and scoring.
  • Kickoff: Monday, October 12.
  • Onboarding: Before the project begins, subject to scheduling.
Vacancy posted 6 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluator - Remote in New York, NY vacancy
  • AuraOne is seeking an Everyday AI Agent Tester specialist operations contractor for remote US-eligible work. You will review AI outputs against workflows, policy, and playbooks, grade tone and escalation, and document next steps for model training. Ideal candidates have... 
    Remote job
    For contractors

    Auraone

    New York, NY
    16 hours ago
  • $80 per hour

    A leading AI consultancy is searching for QA analysts to validate AI agents in a project focused on improving complex tasks. This opportunity is a flexible, remote, part-time role ideal for curious thinkers...  ...identify inconsistencies in AI evaluation. Competitive compensation... 
    Remote job
    Part time
    Flexible hours

    Mindrift

    Jackson, MS
    16 hours ago
  • $20 - $40 per hour

    Put AI agents through realistic multi-step jobs and judge whether they actually finish them. Your scenarios, ratings and notes show developers...  ...to follow detailed guidelines and work independently Contract Remote, hourly contract (independent contractor) Part-time, flexible:... 
    Remote job
    Hourly pay
    Weekly pay
    Contract work
    Part time
    For contractors
    Flexible hours

    Vettedlance

    New York, NY
    4 days ago
  • AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining. The role focuses on auditing... 
    Remote job

    Auraone

    New York, NY
    2 days ago
  • OpenTrain AI, Inc. is seeking individuals to remotely test two AI agents on everyday tasks using a smartphone. As a part-time contractor in the United States, you will guide agents, screen record sessions, and provide detailed feedback on completion, control, and user... 
    Remote job
    Part time
    For contractors

    OpenTrain AI, Inc.

    New York, NY
    2 days ago
  • $60 per hour

    A leading AI company is seeking QAs for autonomous AI agents to ensure quality assurance in complex task evaluations. This remote project-based role is ideal for individuals who excel in analytical thinking and have experience in evaluating system logic. The opportunity... 
    Remote job
    Flexible hours

    Mindrift

    Madison, WI
    16 hours ago
  • $20 - $40 per hour

     ...seeking detail-oriented contributors to craft realistic AI task scenarios and evaluate agent performance. You will describe steps, judge outcomes, and...  ...to help developers improve automation. The role is remote, part-time, hourly, 10-40 hours per week, paid weekly in... 
    Remote job
    Hourly pay
    Weekly pay
    Part time
    10 hours per week

    Vettedlance

    New York, NY
    4 days ago
  • $80 per hour

     ...technology company is seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong analytical thinking,...  ...expected behaviors for AI agents. This flexible remote opportunity allows you to work on your own schedule while... 
    Remote job
    Part time
    Flexible hours

    Mindrift

    Phoenix, AZ
    1 day ago
  •  ...AI Agent Evaluator is a remote evaluation track for reviewing ai agent evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain... 
    Remote job
    Hourly pay
    For contractors
    10 hours per week

    AuraOne Human Data

    Remote
    a month ago
  • $25 - $45 per hour

     ...as bills, returns, bookings, groceries and travel through two AI agents side by side, using their own accounts on their phone. They approve...  ..., step by step. Listed pay: $25 to $45 per hour. Where: Remote, US-based. Time: About 1.5 hours per run, on your own schedule.... 
    Remote work
    Hourly pay
    Contract work
    Work at office

    micro1

    Remote
    1 day ago
  • $14.5 per hour

    A technology company is seeking an AI Web Search Evaluator to enhance the quality of search engine results. This flexible, remote, part-time role focuses on analyzing search performance and providing feedback to improve algorithms. Ideal candidates will have strong analytical... 
    Remote work
    Hourly pay
    Part time
    Flexible hours

    Welo Data

    United States
    2 days ago
  • $14.5 per hour

     ...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology companies to provide datasets that are...  ...optimization or data analysis is beneficial. Seeks a flexible, remote job that can easily fit around other commitments. Pay... 
    Remote work
    Hourly pay
    Part time
    Immediate start
    Work from home
    10 hours per week
    Flexible hours

    Welo Data

    New York, NY
    2 days ago
  •  ...Alignerr Corp. is seeking Creative Writing Evaluators to shape AI training by judging AI-generated writing across fiction, poetry, essays, and...  ..., voice, coherence, and reader engagement. Work is fully remote on an hourly contract, with 10–40 hours per week available.... 
    Remote work
    Hourly pay
    Contract work
    10 hours per week

    Alignerr Corp.

    United States
    1 day ago
  • $20 per hour

     ...A tech company specializing in AI is hiring a Digital Web Designer. In this remote role, you will evaluate AI-generated designs and provide feedback to enhance the model’s understanding of aesthetics. An ideal candidate will have a strong background in UI/UX design and... 
    Remote work
    Flexible hours

    DataAnnotation

    Columbia, SC
    9 hours ago
  • $14.5 per hour

     ...AI Web Search Evaluator Welo Data works with technology companies to provide datasets that are high-quality, ethically sourced, relevant,...  ...suits you—choose your hours and achieve work-life balance. Remote Work: Enjoy the freedom of working from home anywhere in... 
    Remote work
    Bi-weekly pay
    Hourly pay
    Part time
    Immediate start
    Work from home
    Flexible hours

    Welo Data

    United States
    2 days ago
  • $25 - $45 per hour

    micro1 Inc. is seeking an AI Agent Evaluator contractor based in the United States. You will run real-life scenarios using two AI agents, screen record, and score results, providing detailed written feedback to help improve next‑gen AI agents. Onboarding is available today... 
    Hourly pay
    For contractors
    Immediate start

    micro1 Inc.

    New York, NY
    4 days ago
  • $25 - $45 per hour

    Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot Pay: $25-45/hour Time per task: ~1.5 hours per run (evidence and scoring included) Project kickoff: Monday, October 12 Onboarding: Today (Fri), Saturday, or Sunday, so you're... 
    For contractors
    Work at office
    Weekend work

    micro1 Inc.

    New York, NY
    4 days ago
  • Welo Data is seeking Mandarin-speaking experts for a 4-week remote language evaluation project. You will evaluate AI-generated Mandarin content, write and rewrite prompts, and refine responses to improve dataset quality. This is a freelance contract with flexible weekly... 
    Remote job
    Contract work
    Freelance
    Work from home
    Flexible hours

    Welo Global

    New York, NY
    16 hours ago
  • OpenTrain AI, Inc. seeks an experienced 211 Specialist to evaluate AI answers for government services. This part-time, remote role focuses on reviewing questions, writing rubrics, and giving precise feedback to improve accuracy and usefulness. You will assess AI responses... 
    Remote job
    Part time
    Flexible hours

    OpenTrain AI, Inc.

    New York, NY
    1 day ago
  • Prolific Academic Ltd. is seeking expert UX evaluators to assess AI-generated design work. You will judge usefulness, accessibility, and coherence...  ...and user journeys, with clear reasoning. The engagement is remote and paid, subject to a brief skills assessment and... 
    Remote job

    prolificacademicltd

    New York, NY
    2 days ago
  • Prolific Academic Ltd is seeking a remote, on-demand Physics expert to evaluate AI-generated graphical abstracts of scientific papers. You will review, compare, and judge the accuracy of the summaries, applying deep physics knowledge to ensure fidelity to the underlying... 
    Remote job
    Hourly pay
    Flexible hours

    prolificacademicltd

    New York, NY
    2 days ago
  • Prolific Academic Ltd is seeking individuals to help train and assess AI systems through paid, task-based work. Responsibilities include reviewing support conversations, drafting realistic customer tickets, and improving customer-facing guidance. A short skills assessment... 
    Remote job

    prolificacademicltd

    New York, NY
    2 days ago
  • prolificacademicltd is offering a remote, flexible-role focused on evaluating and improving AI-generated customer support content. You will assess model replies, create test tickets and dialogue flows, and review FAQs to ensure clarity and accuracy. The position requires... 
    Remote job
    Flexible hours

    prolificacademicltd

    New York, NY
    2 days ago
  • $40 per hour

     ...Ltd. is seeking a part-time Materials Science specialist to audit scientific papers and machine-generated graphical abstracts. This remote role in the US involves opt-in assignments based on client demand, with typical engagement at $40+ per hour and tasks usually under... 
    Remote job
    Hourly pay
    Part time

    prolificacademicltd

    New York, NY
    2 days ago
  • $80 per hour

    Develop and map analytic cases to defined taxonomies across multiple evaluation areas. Author complex test cases to evaluate model performance on source handling and analytic rigor. Assess and grade AI-generated responses based on reasoning, source use, and practical... 
    Remote job

    Crossing Hurdles

    New York, NY
    16 hours ago
  • AuraOne, Inc. is seeking a remote Deception Behavior Risk Evaluator to stress-test AI systems with adversarial prompts. You will design jailbreak-like prompts, document failures with reproduction steps, and map each incident to the applicable policy clause to enable rapid... 
    Remote job
    For contractors

    AuraOne Human Data

    New York, NY
    16 hours ago
  • A tech-focused company is seeking a qualified LLM - AI Quality Analyst Personalization, fluent in Italian, to evaluate and improve AI model responses. In this remote role, you will analyze AI outputs, design multi-turn prompts, and provide structured feedback to enhance... 
    Remote job
    Flexible hours

    Crossing Hurdles

    New York State
    4 days ago
  • Outlier is seeking detail-oriented English speakers to review AI-generated media from home on a flexible schedule. You will roleplay live conversations with AI voice models, score exchanges, and explain in writing what looked or sounded off. You will compare image and... 
    Remote job
    Flexible hours

    Outlier

    Boca Raton, FL
    16 hours ago
  • $50 per hour

    prolificacademicltd is seeking a product design and UX expert to help train and evaluate AI models through paid, task-based work. You’ll review generated...  .... A short skills assessment is part of the selection process. Remote, flexible work with payments per task, rates up to $50/hour... 
    Remote job
    Flexible hours

    prolificacademicltd

    New York, NY
    2 days ago
  • Turing is seeking detail-oriented AI Analysts based in the United States for a Google Wallet evaluation project. This role allows you to engage with advanced AI tools while contributing to the future of AI. You will evaluate model responses, review output quality, and provide... 
    Remote job
    Full time
    Contract work

    Turing

    Seattle, WA
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluator - Remote. Be the first to apply!