Remote AI Agent Evaluator: Test & Score Real-Life Tasks
RemoteJobsOne
RemoteJobsOne in the United States is hiring an AI Agent Evaluator on a fully remote, contractor basis. You’ll test two AI agents, run real-life scenarios, and rate completion, quality, and user experience from a real user perspective.
Requirements include a US-based status with a Bachelor’s degree, 2–3 years in related fields, strong attention to detail, and daily use of AI tools. You’ll work from your phone on your schedule and help shape next-generation AI agents.
#J-18808-LjbffrVacancy posted 4 hours ago
Similar jobs that could be interesting for youBased on the Remote AI Agent Evaluator: Test & Score Real-Life Tasks in Houston, TX vacancy
- ...RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction...Remote workFor contractorsStart working today
RemoteJobsOne
Jacksonville, FL4 hours ago $25 - $45 per hour
...This is a fully remote position open to... ...hr Role Title: AI Agent Evaluator Role Type: Contractor... .../hour Time per task: 1.5 hours per run (evidence and scoring included)... ...starting to handle real-life errands: paying bills... ...Thats you. Youll test two AI agents side...Remote workFull timeFor contractorsWork at officeWeekend workRemoteJobsOne
Houston, TX4 days ago$25 - $45 per hour
...This is a fully remote position , open to... ...hr Role Title: AI Agent Evaluator Role Type: Contractor... .../hour Time per task: ~1.5 hours per run (evidence and scoring included)... ...starting to handle real-life errands: paying bills... ...’s you. You’ll test two AI agents side...Remote workFor contractorsWork at officeWeekend workRemoteJobsOne
Houston, TX4 hours ago- OpenTrain AI, Inc. is seeking a remote, part-time contractor to test two AI agents on a smartphone, completing realistic errands and recording outcomes. You will evaluate task completion, quality, control issues, user satisfaction, and time, then provide clear feedback....Remote jobPart timeFor contractorsFlexible hours
$25 - $45 per hour
...ChatGPT Muse AI agents Attention to... ...provide frontier evaluations and reinforcement... .../hour Time per task: ~1.5 hours per run (evidence and scoring included) Project... ...starting to handle real-life errands: paying bills... ...’s you. You’ll test two AI agents side...SuggestedFor contractorsWork at officeWeekend workAitrainer
Kentucky4 hours ago- ...micro1 is a data lab that helps AI labs and enterprises train foundational models and agents. We’re hiring an Everyday AI Agent Tester to run real-life scenarios, record outcomes, and provide actionable feedback from a user perspective. You’ll compare two AI agents,...
Aitrainer
Kentucky4 hours ago - AI agents are starting to handle real-life errands: paying bills, booking trips, doing the groceries. We want to know... ...of a real user. That’s you. You’ll test two AI agents side by side, using... ...steps. Screen record every attempt. Score each run on completion, quality, control...Work at office
- Mercor is seeking a Real Estate Appraisal — Visual Document Understanding contractor to support AI model evaluation in the real estate domain. This remote, 15-20 hours/week engagement emphasizes rigorous... .... You will review complex tasks, develop evaluation rubrics, and...Remote jobFor contractors
- ...Simulation-to-Real Task Evaluator is a remote evaluation track for reviewing simulation to real task evaluation... ...to retrain. Why this role matters AI data reviewers help turn simulation to... ...updates. Maintain reviewer-quality scores by calibrating against gold-standard...Remote jobHourly payFor contractors10 hours per week
- ...Gmail & Google Calendar AI Assistant Evaluator Role Type: Contractor, Remote Location: United... ...assistant quality through real-world task evaluation. In this... ...managing your digital life - email, calendars, bookings... ...model performance. # Test scenarios involving...Remote workFor contractors
micro1
Kentucky2 days ago - ...scientists and engineers to evaluate frontier AI models on real technical work, including... ...will design challenging tasks drawn from your practice,... ...them through frontier AI agents and assess results to professional... ...long-form work in a fully remote setting, with explicit...Remote job
$62.4k - $83.2k
.... Job Title: AI Intern, Enterprise Agent Development... ..., but open to US Remote for the right candidate... ...Design, develop, test, and continuously... ...capabilities such as task planning, tool... ...solutions in real-world enterprise... ...savings accounts, life and AD&D insurance...Remote workHourly payTemporary workInternshipFlexible hours- Mindrift connects specialists with project-based AI opportunities for testing, evaluating, and improving AI systems. You’ll design tasks within realistic simulated environments and... ...correct solutions. This freelance, remote role requires 5+ years in software development...Remote jobFreelance
$60 per hour
...the future of AI. What We Do The... ...by tapping into real‑world expertise... ...ambiguity, enjoy remote asynchronous... ...AI systems are tested and evaluated, we want to hear... ...autonomous AI agents in a project focused... ...complex task structures, policy... ...understanding of how scoring or evaluation...Remote workFreelanceFlexible hours- ...Obsidian is hiring expert Evaluators in Real estate, hospitality, and events to review AI-generated work for accuracy, rigor, and domain quality. This remote position requires deep expertise and involves grading outputs like documents and presentations. Applicants must...Remote workWork at office
Mercor
San Francisco, CA4 hours ago $60 per hour
...Engineers to join our Expert Network to train and evaluate AI models using real-world chemical knowledge. A quick 10- to 15-minute test will assess your skills; successful... ...join Prolific as participants and are paid for tasks. Researchers seeking your expertise may pay...Remote jobHourly payWork from homeFlexible hours$200k - $275k
...Labs provides AI-powered... ...involved in evaluating and integrating... ...the LLM and agent space — including... ...investigative tasks and augment analyst... ...impact real users (analysts... ...—onsite and remote—has full... ...operational reviews, testing strategies,... ...States. Life at TRM We...Remote workLocal areaWorldwideTRM Labs
United States3 days ago$80 per hour
...seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong... ...solving. The role includes reviewing tasks, identifying inconsistencies, and defining... ...for AI agents. This flexible remote opportunity allows you to work on your...Remote jobPart timeFlexible hours$209.5k
...Senior AI Agent Engineer Remote Up to $209,500/ year... ...speed with safety, evaluation, and maintainability... ...to deliver real productivity gains... ...Design, build, test, and deploy production... ...agents that support tasks such as SQL... ...term disability ~ Life insurance and AD&...Remote workTemporary workFlexible hoursHackajob
United States4 days ago$100 - $120 per unit
...technical talent with leading AI research labs.... ...Baseball Fan Live MLB Game AI Evaluator (US) Type:... ...Compensation: $100$120 per task Location: Remote Commitment:... ...and ask AI assistants real fan questions about live scores win probability player...Remote workFull timeContract workWork experience placementSummer workMercor
New York, NY8 days ago$161.3k - $274.2k
...the Agentic AI Assistant platform... ...and automate tasks. Powered by... ...' AI agents don't just generate... ..., and change real state in enterprise... ...: how do you score what an agent... ...of our agent evaluation platform: the... ...it isn't a testing role. It's... ...(flexible, remote, or required...Remote workContract workWork at officeFlexible hoursMoveworks
Mountain View, CA6 days ago$50 per hour
...Expert Network for training and evaluating AI models with real design expertise. After a quick test, you may join Prolific as a... ...participant and get paid for AI training tasks. Researchers pay up to $50 per... ...work, though many are shorter. Remote work and flexible hours are...Remote jobHourly payFlexible hours- Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Remote jobWork at office
$50 per hour
...Talent professionals to join our Expert Network. You will help train and evaluate AI models using real HR expertise and may be invited to join Prolific as a paid participant after a quick test. Tasks pay up to $50 per hour and require about an hour of uninterrupted work,...Remote jobHourly pay- Mercor is a remote contractor role focusing on evaluating AI-generated artifacts within real estate, hospitality, and events. You will assess quality against rubrics, spot factual, presentation, and aesthetic errors, and deliver structured feedback to help improve AI models...Remote jobFor contractorsWork at office
$9 - $30 per hour
...beyond words to perform each and every task with integrity and a passion to stay... .... Description About the Job The Testing Evaluator position is a remote, on demand, part-time position. The... ...will be evaluated according to given scoring criteria; therefore, paid training is...Remote workFull timePart timeFor contractorsCurrently hiringWork at officeFlexible hours$200k - $250k
...Labs provides AI-powered... ...involved in evaluating and integrating... ...the LLM and agent space - including... ...tasks and augment analyst... ...agents impact real users (analysts... ...team—onsite and remote—has full visibility... ...reviews, testing strategies,... ...process. Life at TRM We...Remote workLocal areaWorldwideTRM Labs
Kentucky14 hours ago$99.6k - $174k
...Stack Engineer, AI Platform &... ...Canada, Hybrid or Remote-Work Hours:... ...of AI agents. We also drive... ...resource into a real‑time medical... ...engineering / testing strategySecure... ...safety evaluations, iterative testing... ...traditional ML tasks like training... ...and personal life, including...Remote workFull timeWork at office2 days per weekWolters Kluwer
New York, NY3 days ago$225.2k - $394.1k
...automated a tedious task for his coworker,... ...ServiceNow is the AI control tower for business... ...frontier Ai agent currently in MVP state... ...of frontier AI and real-world workflow... ...products. You build, test, and iterate. 12+ years... ...(flexible, remote, or required in office...Remote workWork at officeImmediate startFlexible hoursShift workServiceNow
San Diego, CA5 days ago$60 per hour
...looking for Biology Experts and Life Science Professionals in Las Vegas, NV to evaluate AI-generated science. Successful candidates... ...up to $60 per hour for paid tasks. Responsibilities include... ...Participants must pass a brief assessment test before joining. #J-18808-Ljbffr...Remote jobHourly payWork from homeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote AI Agent Evaluator: Test & Score Real-Life Tasks. Be the first to apply!
