Remote AI Agent Tester - Real-World Task Evaluator
RemoteJobsOne
New York, NY
- Remote job
RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction. Onboarding starts today and you’ll provide clear feedback on the agent experience. The position requires a US-based candidate with a Bachelor’s degree and 2–3 years of relevant experience. #J-18808-Ljbffr RemoteJobsOne
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Remote AI Agent Tester - Real-World Task Evaluator in New York, NY vacancy
- ...Dover is seeking an AI Agent Evaluator contractor based in the United States to test two AI agents by performing real-life tasks and recording outcomes. You will assess usability, accuracy... ..., and rate performance. This role is remote and offers flexible scheduling. #J-1...Remote workFor contractorsFlexible hours
- OpenTrain AI, Inc. is seeking a remote, part-time contractor to test two AI agents on a smartphone, completing realistic errands and recording outcomes. You will evaluate task completion, quality, control issues, user satisfaction, and time, then provide clear feedback....Remote jobPart timeFor contractorsFlexible hours
- micro1 is a data lab that helps AI labs and enterprises train foundational models and agents. We’re hiring an Everyday AI Agent Tester to run real-life scenarios, record outcomes, and provide actionable feedback from a user perspective. You’ll compare two AI agents, document...Suggested
$25 - $45 per hour
...spam folder. This is a fully remote position open to candidates... ...Pay: $25$45/hr Role Title: AI Agent Evaluator Role Type: Contractor... ...Pay: $25-45/hour Time per task: 1.5 hours per run (evidence... ...agents are starting to handle real-life errands: paying bills booking...Remote workFull timeFor contractorsWork at officeWeekend workRemoteJobsOne
Houston, TX3 days ago- ...Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Required Skills:... ...role: AI agents are starting to handle real-life errands: paying bills, booking... ...felt. Opportunity Snapshot: Time per task: ~1.5 hours per run (evidence and scoring...Remote jobContract workFor contractorsWork at officeWeekend work
- ...scientists and engineers to evaluate frontier AI models on real technical work, including... ...will design challenging tasks drawn from your practice,... ...them through frontier AI agents and assess results to professional... ...long-form work in a fully remote setting, with explicit...Remote job
- CNTXT AI is seeking a remote contractor to evaluate AI-generated financial content and develop test cases that probe analytical reasoning. You will help improve... ...from balance sheets and reports; creating prompts and Excel-based evaluation tasks; and #J-18808-Ljbffr CNTXT AIRemote jobFor contractors
$60 per hour
...Chemical Engineers to join our Expert Network to train and evaluate AI models using real-world chemical knowledge. A quick 10- to 15-minute test will... ...candidates join Prolific as participants and are paid for tasks. Researchers seeking your expertise may pay up to $60 per...Remote jobHourly payWork from homeFlexible hours$80 per hour
...seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong... ...solving. The role includes reviewing tasks, identifying inconsistencies, and defining... ...for AI agents. This flexible remote opportunity allows you to work on your...Remote jobPart timeFlexible hours- Thespian Labs in Somerville, MA seeks an Agent Product Manager to lead our embodied AI products. You will define the roadmap for agent... ...agents that perceive, act, and speak in real environments. You will translate real-world needs into concrete product requirements, measure...
- Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Remote jobWork at office
- Archangel Health AI is seeking Clinical AI Evaluators to review AI-generated clinical outputs... ..., and refine responses to real-world medical queries. You will... .... This is a flexible remote role with no patient care... ...United States and earn per task, with no on-call shifts....Remote workFlexible hoursShift work
- Mercor is a remote contractor role focusing on evaluating AI-generated artifacts within real estate, hospitality, and events. You will assess quality against rubrics, spot factual, presentation, and aesthetic errors, and deliver structured feedback to help improve AI models...Remote jobFor contractorsWork at office
- ...micro1 is hiring Gmail & Google Calendar AI Assistant Evaluators to collaborate on a customer-driven project that enhances AI assistant quality through real-world task evaluation. In this contractor, remote role, you will use your everyday experience managing digital...Remote jobFor contractors
- ...Role Title: Gmail & Google Calendar AI Assistant Evaluator Role Type: Contractor, Remote Location: United States only micro1 is engaging... ...project enhancing AI assistant quality through real-world task evaluation. In this role, you'll use your everyday...Remote workFor contractors
- ...Recruiters / ESR Healthcare is seeking a meticulous Gmail & Google Calendar AI Assistant Evaluator for a remote contract role based in the United States. You will perform real-world digital tasks, give AI clear instructions, and review results to ensure accuracy under...Remote jobContract work
- ...Everyday AI Agent Tester is a remote review track for evaluating AI outputs across everyday ai agent tester specialist operations... ...rules that decide whether a task actually gets done. Responsibilities... ...tester specialist operations on real teams for Everyday AI Agent Tester...Remote jobHourly payFor contractorsWork experience placement10 hours per week
$225.2k - $394.1k
...that automated a tedious task for his coworker,... ...Today, ServiceNow is the AI control tower for business... ...frontier Ai agent currently in MVP state.... ...intersection of frontier AI and real-world workflow impact. You'll... ...Work personas (flexible, remote, or required in office)...Remote workWork at officeImmediate startFlexible hoursShift workServiceNow
San Diego, CA4 days ago- About Archangel Health AI Archangel Health AI is at the forefront of training, evaluating, and refining next-generation artificial... ...(HCPs) with AI development tasks to ensure medical AI systems are... ...reasoning, and refine AI responses to real-world medical queries. Clinical...Remote jobSecond jobShift workWeekend work
- ...finally made it possible for AI to impact clinical care... ...an entire ecosystem of real-world evidence to deliver... ...edge AI tools to scale tasks and augment insights.... ...statistical methods, while also evaluating study limitations.... ....Additionally, for remote roles open to individuals...Remote workFull time
Tempus
Redwood City, CA3 days ago - GenScript USA Inc./ProBio Inc. seeks an AI Intern for Enterprise Agent Development to design, develop, and... ...to CRM. The role emphasizes building task planning, tool calling, and workflow... ...business teams to deploy solutions in real-world enterprise scenarios, with Mandarin...Remote jobInternship
$50 per hour
...technical talent with leading AI research labs.... .../hour Location: Remote Commitment: 20–40... ...Role Responsibilities Evaluate the effectiveness of... ...personalized, real-world tasks. Use AI tools extensively... ...Codex , or other AI agents . Ability to...Remote workContract workSummer workMercor
San Francisco, CA15 days ago$70 - $110 per hour
...technical talent with leading AI research labs.... ...0/hour Location: Remote Duration: 4–6 weeks... ...Role Responsibilities Evaluate AI-generated... ...feedback. Leverage real-world expertise in directing... ...Compensation & Legal ~ Task-based payment upon first...Remote workHourly payContract workSummer workImmediate startMercor
New York, NY15 days ago- ...Simulation-to-Real Task Evaluator is a remote evaluation track for reviewing simulation to real task evaluation prompts and responses against AuraOne... ...team can use to retrain. Why this role matters AI data reviewers help turn simulation to real task evaluation...Remote jobHourly payFor contractors10 hours per week
$152k - $240k
...operations, gain real-time visibility,... .... Brex’s AI-native automation and world-class service eliminate... ...and accounting tasks for customers so... ...'re building AI agents to automate and... ...per year of fully remote work!... ...sources.Define evaluation frameworks, success...Remote workWork at officeWork from homeBrex
San Francisco, CA5 days ago- ...finally made it possible for AI to impact clinical care... ...an entire ecosystem of real-world evidence to deliver... ...edge AI tools to scale tasks and augment insights.... ...statistical methods, while also evaluating study limitations.... ....Additionally, for remote roles open to individuals...Remote workFull time
Tempus
Chicago, IL3 days ago $62.4k - $83.2k
...and regions. Job Title: AI Intern, Enterprise Agent Development Supervisor... ...preferably, but open to US Remote for the right candidate)... ...Build capabilities such as task planning, tool calling, and... ...deploy AI Agent solutions in real-world enterprise scenarios....Remote workHourly payTemporary workInternshipFlexible hours$30 - $90 per hour
...Help improve next-generation AI systems by applying... ...operational analysis. This remote contractor role focuses on creating real-world workflows and feedback that make AI agents more effective in professional... ...AI in everyday professional tasks. Use AI agents to research...Remote workHourly payFor contractorsSaidGig
United Statesa month ago- ...MERIT Beauty is seeking Turkish-speaking annotators for a contract role evaluating AI-generated content. You will review content for coherence, consistency, and alignment with real-world expectations in Turkish. The ideal candidates are native or fluent Turkish speakers...Remote workContract workTemporary workImmediate start
MERIT Beauty
United States5 days ago - ...stage startup building an AI coworker: an agent that lives inside a... ...it finishes a long task without falling over halfway... ...in the services world. They watched clients... ...so they know where the real demand is. The role starts... ...for both sides. It is remote-friendly anywhere in...Remote workFull timeContract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote AI Agent Tester - Real-World Task Evaluator. Be the first to apply!
