Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Agent Tester - Remote

YO AI Labs

New York, NY
  • Remote job

Job Description

Job Description

Job Title: AI Agent Tester – Real-World Tasks

Role Type: Contractor
Location: United States

Job Summary

We are seeking detail-oriented, U.S.-based AI Agent Testers to evaluate how AI assistants handle everyday tasks. You will test two AI agents using your own accounts and smartphone, assess their performance, and provide feedback to improve their real-world usability.

Key Responsibilities
  1. Test AI agents on tasks such as shopping, bookings, bill payments, returns, email follow-ups, travel, and online forms.
  2. Approve or stop agents at important decision points.
  3. Screen-record every task attempt.
  4. Score performance based on completion, quality, user control, satisfaction, and time.
  5. Provide clear written feedback on the overall user experience.
Required Qualifications
  1. Must be based in the United States.
  2. Bachelor's degree required.
  3. 2–3 years of professional experience, preferably in engineering, business management, or operations.
  4. Regular use of AI tools in daily life.
  5. Strong attention to detail, written communication, and documentation skills.
  6. An active personal AI assistant account (Plus tier), active Muse and Dots access, a smartphone with screen-recording capability, and a payment card for reimbursable test purchases.

No coding or AI technical background is required. Generalists are welcome, and domain experts are preferred.

Project Details
  • Kickoff: Project is already active.
  • Onboarding: Within 24–48 hours of selection.
  • Work Schedule: Flexible, entirely from your phone.
  • Task Duration: Approximately 1.5 hours per run, including evidence and scoring.
Selection Process
  1. Apply and complete the screening questions.
  2. Attend a 15–30-minute interview.
  3. Complete the profile review.
  4. Receive onboarding instructions from the Talent Operations team.
Why Join?
  • Work with AI tools and evaluate real-world performance.
  • Enjoy flexible, phone-based work.
  • Help shape and improve next-generation AI agents.
Vacancy posted 11 days ago
Similar jobs that could be interesting for youBased on the AI Agent Tester - Remote in New York, NY vacancy
  •  ...: Contractor Location: Global, Fully Remote Time Commitment: Approximately 15 hours...  ...Learning Researchers to contribute to an AI training project focused on model development...  ...experience with Python and coding agents. You will develop, review, and validate challenging... 
    Remote job
    For contractors
    Flexible hours

    YO AI Labs

    New York, NY
    5 days ago
  •  ...AI Agent Tester - Real-World Tasks is a remote review track for evaluating AI outputs across ai agent tester real world tasks specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document... 
    Remote job
    Hourly pay
    For contractors
    Work experience placement
    10 hours per week

    AuraOne Human Data

    Remote
    27 days ago
  • $25 - $45 per hour

     ...Testers put two AI agents through ordinary errands such as paying bills, shopping, travel booking and form filling, all from a personal phone....  ...documentation. Listed pay: $25 to $45 per hour. Where: Remote, US only. Time: About 1.5 hours per run. Contract work for... 
    Remote work
    Hourly pay
    Contract work
    Live in

    micro1

    Remote
    1 day ago
  •  ...Job Description Job Description Role Title: Everyday AI Agent Tester Role Type: Contractor Location: United States We are seeking detail-oriented, U.S.-based professionals to test AI agents performing everyday tasks. You'll evaluate two AI agents side by side... 
    Remote job
    For contractors
    Work at office

    YO AI Labs

    New York, NY
    11 days ago
  • $20 - $30 per hour

     ...Type: Contract, hourly Location: Remote Hours: 20–30 per week Pay: $20–30/hour, based on experience and language coverage...  ...Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open... 
    Remote work
    Hourly pay
    Full time
    Contract work
    Immediate start

    Bespoke Labs

    Syracuse, NY
    1 day ago
  •  ...AI Agent Evaluation Specialist This project aims to test and evaluate one AI agent in the development stage. By asking questions and...  ...content to the end users and make it safer. This role is fully remote, full-time with a duration of several months (depending on the... 
    Remote work
    Full time

    Tsmg

    United States
    1 day ago
  • $40 - $120 per hour

     ...AI Chatbot Tester $40-120/hr Remote Freelance GENERAL About the Role What if your curiosity and critical eye could help shape the future of artificial intelligence? We're looking for AI Chatbot Testers to put cutting-edge AI systems through their paces — having... 
    Remote work
    Hourly pay
    Contract work
    Freelance
    Worldwide
    Flexible hours

    Alignerr

    United States
    3 days ago
  •  ...next steps. Our partner is looking for a Technical Co-Founder, AI Agents based in United States. Join an early-stage AI venture as a...  ...foundations of a high-performing engineering team. This fully remote opportunity is ideal for an entrepreneurial engineer eager to build... 
    Remote work
    Full time

    Jobgether

    Remote
    6 days ago
  •  ...AI Chatbot Tester MCI is one of the fastest-growing tech-enabled business services companies in the USA, with a strong call center footprint...  .... Our contact centers are powered by both on-site and remote agents, leveraging advanced technologies to enhance customer... 
    Remote work
    Temporary work
    Work at office
    Local area

    AI Support Positions

    United States
    1 day ago
  • $62.4k - $83.2k

     ...customers across over 100 countries and regions. Job Title: AI Intern, Enterprise Agent Development Supervisor: Head of AI for Industry,...  ...: Piscataway, NJ (Onsite preferably, but open to US Remote for the right candidate) Compensation: Targeting $30... 
    Remote work
    Hourly pay
    Temporary work
    Internship
    Flexible hours

    GenScript

    Remote
    10 days ago
  • $185k - $260k

     ...AI Product Manager, Agents & Orchestration This is a fully remote opportunity and can be worked from any location in the United States. About Us 1mind is building the next generation of revenue teams. We are creating a new category of enterprise software: Autonomous... 
    Remote work
    Full time

    1mind

    Remote
    20 days ago
  •  ...Job Description Job Description Role: AI Agentic Tester Location: Denver, CO (Zip Code-80221)- REMOTE Skills: QA/Test Engineering experience Must have-...  ...skills needed : Functional Testing Validate AI agent workflows against business requirements. Test... 
    Remote work

    Talent Software Services, Inc.

    Jacksonville, FL
    26 days ago
  • $150k - $250k

     ...People make the decisions that matter. Agents do everything else. Agents write and review...  ...start in seconds Location Fully remote in the US. We aren't able to sponsor visas...  ...the application questions yourself, without AI. After you apply, we'll tell you how to use... 
    Remote work
    Full time
    Work at office
    Immediate start
    Home office
    Visa sponsorship

    Perimeter Compute

    Remote
    5 days ago
  •  ...Engineering Manager - AI Agents (Agent Platform, LLMs, Evals) Shape the future of trust in the age of AI At Oscilar, we're building...  ...frameworks), AWS Bedrock, Python + Java, MCP This is a remote senior engineering leadership role based in the US. You will report... 
    Remote work
    Full time

    Oscilar

    Remote
    4 days ago
  •  ...Senior Director, Data Science and AI Agents Location: San Mateo, CA · Nashville, TN · Sterling, VA Open to remote POSITION OVERVIEW The Senior Director of Data Science leads the technical vision, strategy, and execution for AI capabilities that power... 
    Remote work

    Asurion

    Nashville, TN
    18 days ago
  •  ...Everyone has felt this. No one should have to. At Assort Health, we're building one continuous conversation for every patient. An AI agent that knows who you are, remembers how you like to be reached, and carries that context from your first symptom to your full... 
    Remote work
    Work at office
    Flexible hours

    Assort Health

    United States
    5 days ago
  • Senior Engineer - AI Voice Agents Location: Chicago, IL (Partial Remote) | Job Type: C2C About the Role Join a dynamic technology team as a Senior Engineer specializing in AI Voice Agents. You will architect, develop, and deploy advanced voice-enabled AI solutions that... 
    Remote work
    Work from home

    Indotronix International Corporation

    Chicago, IL
    3 days ago
  •  ...Job Title: Member of Technical Staff Agent Infrastructure & Research Job Location: San...  ...Company is building infrastructure to make AI provably trustworthy for software development...  ...Every sixth week (reflection week) allows remote work. Dealbreakers ~0-to-1... 
    Remote work
    Full time
    Internship
    Work at office

    Staffingine LLC

    San Francisco, CA
    2 days ago
  • $200k - $275k

     ...Senior Software Engineer, AI Agents Compensation: $200,000 - $275,000 base + equity Location: Remote, USA Join an early-stage applied AI company building the infrastructure and agent layer required to automate complex enterprise knowledge work. The company... 
    Remote work
    Full time

    People In AI

    United States
    2 days ago
  • $170k - $225k

     ...Pros, excellent service for both sides, and an AI-run company that moves faster than its size should allow. About AI Agents at LawnStarter Our mission: power every AI...  ...Medical, dental, and vision Location: Fully remote (US). Evals, transcripts, and failure cases reward... 
    Remote work
    Full time
    Local area
    Flexible hours

    LawnStarter

    Remote
    6 days ago
  • $200k - $320k

     ...product-focused engineering team building AI-powered tools for data and marketing workflows...  ...different LLMs. Build production data agents that use company data to answer analytics...  .... Visa sponsorship is available. Location Fully remote in the United States.... 
    Remote work
    Full time
    Visa sponsorship

    True Talent Solutions Partners

    Remote
    23 hours ago
  • $158k - $198k

    Director, Conversational AI Agents Join our team at TELUS Digital , a leading digital product consultancy and the digital division of TELUS...  ...Work From Near (Hybrid) in a US office or Work From Anywhere (Remote) with travel to clients and TELUS Digital offices expected.... 
    Remote work
    Full time
    Work at office

    TELUS Digital

    Charlottesville, VA
    3 days ago
  • $160k - $180k

     ...About Credo AI The Credo AI Governance Platform operates as the trust layer for global...  ...powered by a knowledge graph that is updated by agents, enriched with global regulatory...  ...relevant education or training. Location & Remote Culture While this is a remote role and... 
    Remote work
    Full time
    Work at office

    Credo.AI

    Remote
    19 days ago
  • $126k - $170k

     ...Technical Product Manager, Identification & JS Agent based in United States. We are looking...  ..., experience, and expertise. Fully remote working environment within eligible...  ...\n How Jobgether works: We use an AI-powered matching process to ensure your application... 
    Remote work
    Full time

    Jobgether

    Remote
    7 days ago
  •  ...around the world, with over 20 nationalities from our offices and remotely, who truly enjoys their work. We are looking for the most...  ...backend engineers to build the core systems behind BJAK's AI Finance Agent. This role is for engineers who can build reliable APIs, workflows... 
    Remote work
    Full time

    Bjak

    Remote
    11 days ago
  •  ...Senior Director of Engineering, AI Agents Lehi, Utah, Open to Remote; Remote, US At Podium, we bring AI Employees to local businesses that turn every conversation into revenue. Trusted by 60,000+ businesses across Auto, Home Services, and Aesthetics, Podium captures... 
    Remote work
    Local area

    Podium

    Lehi, UT
    1 day ago
  • $87.7k - $114.2k

     ...partner is looking for a Software Engineer, Agents for Good based in the United States....  ...contribute to the development of innovative AI-powered products and intelligent agent frameworks...  ..., and other business factors. Fully remote and flexible working environment within... 
    Remote work
    Full time
    Flexible hours

    Jobgether

    Remote
    6 days ago
  • $126k - $264k

     ...workloads, embrace innovation, and unleash AI. Our industry-leading developer data...  ...generation, AI-powered applications. Atlas Agent Engine is an agent development platform built...  ...candidates in the United States based on our remote working model. Position Expectations... 
    Remote work
    Full time
    Local area
    Worldwide
    Flexible hours

    MongoDB

    Remote
    13 days ago
  • $93.82k - $131.5k

     ...inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer Tester - Healthcare Data & AI Validation - Remote to join our team in Philadelphia, Pennsylvania (US-PA), United States (US). Data Engineer Tester (2... 
    Remote work
    Full time
    Temporary work
    Work at office
    Flexible hours

    NTT DATA Services

    Remote
    5 days ago
  •  ...and communities. The Role This is not a traditional curriculum developer role - please do not apply if you have dabbled with AI agents. In this unique role, we are accepting applications for two different personas: The AI Solutions Engineer turned Mentor/Coach -... 
    Remote work
    Full time
    Work at office
    Work from home

    veeva

    Boston, MA
    9 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Tester - Remote. Be the first to apply!