AI Agent Evaluator
$25 - $45 per hourmicro1 Inc.
Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot Pay: $25-45/hour Time per task: ~1.5 hours per run (evidence and scoring included) Project kickoff: Monday, October 12 Onboarding: Today (Fri), Saturday, or Sunday, so you're ready to start Monday October 10th You'll need: An active personal popular AI account (Free, Go, or Plus; not Pro), an active personal and a smartphone that can screen record Your tasks: Run everyday errands with two AI agents, approve key steps, screen record, and score the results About the role AI agents are starting to handle real-life errands: paying bills, booking trips, doing the groceries. We want to know how well they actually do it, from the point of view of a real user. That's you. You'll test two AI agents side by side, using your own accounts on your phone, and tell us what worked, what didn't, and how it felt. What you'll do Run real-life scenarios: bills, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms Approve or stop the agent at key steps Screen record every attempt Score each run on completion, quality, control issues, satisfaction, and time Share clear written feedback on the agent experience Who we're looking for US-based , with a Bachelor's degree Someone who already uses AI in daily life 2-3 years of experience, preferably in engineering, business management, or office/operations Detail-oriented, great at following instructions and documenting each step Clear written communication Generalists welcome, domain experts preferred No coding or AI background needed Why you'll like it Get paid to use the AI tools you already use Work entirely from your phone, on your schedule Have a direct say in how next-generation AI agents get built and improved How the selection process works Complete the screening questions Interview: 15-30 minutes max Profile review: if everything looks good, we present you to the client Onboarding: our Talent Ops team will contact you to get you started #J-18808-Ljbffr micro1 Inc.
Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluator in New York, NY vacancy
$25 - $45 per hour
micro1 Inc. is seeking an AI Agent Evaluator contractor based in the United States. You will run real-life scenarios using two AI agents, screen record, and score results, providing detailed written feedback to help improve next‑gen AI agents. Onboarding is available today...SuggestedHourly payFor contractorsImmediate start- HELP SHAPE THE FUTURE OF AI BY EVALUATING HOW WELL AI ASSISTANTS UNDERSTAND USERS, USE PERSONAL CONTEXT, AND DELIVER HELPFUL CONVERSATIONS. Location: Remote Type: Independent contractor / Part-time Project duration: up to 3 months ABOUT THE ROLE At Anyone AI, we create...SuggestedPart timeFor contractorsRemote work
$20 - $40 per hour
Put AI agents through realistic multi-step jobs and judge whether they actually finish them. Your scenarios, ratings and notes show developers where an agent clicks wrong, skips a step or reports success it did not earn. What you will do Write realistic task scenarios for...SuggestedRemote jobHourly payWeekly payContract workPart timeFor contractorsFlexible hours- AuraOne is seeking an Everyday AI Agent Tester specialist operations contractor for remote US-eligible work. You will review AI outputs against workflows, policy, and playbooks, grade tone and escalation, and document next steps for model training. Ideal candidates have...SuggestedRemote jobFor contractors
- OpenTrain AI, Inc. is seeking individuals to remotely test two AI agents on everyday tasks using a smartphone. As a part-time contractor in the United States, you will guide agents, screen record sessions, and provide detailed feedback on completion, control, and user...SuggestedRemote jobPart timeFor contractors
- AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining. The role focuses on auditing...Remote job
$20 - $40 per hour
Vettedlance is seeking detail-oriented contributors to craft realistic AI task scenarios and evaluate agent performance. You will describe steps, judge outcomes, and document failures in plain language to help developers improve automation. The role is remote, part-time...Remote jobHourly payWeekly payPart time10 hours per week- ...Role Title: AI Agent Evaluator Role Type: Contractor Location: United States We are seeking detail-oriented professionals to evaluate AI agents performing everyday tasks. Youll test two AI agents side by side assess their performance and provide feedback to...Full timeFor contractorsWork at officeRemote work
- CRM Agent Agent Evaluator is a remote evaluation track for reviewing crm agent agent evaluation prompts and responses against AuraOne's quality... ...the modeling team can use to retrain. Why this role matters AI data reviewers help turn crm agent agent evaluation outputs into...Hourly payFor contractorsRemote work10 hours per week
$14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology companies to provide datasets that are high-quality, ethically sourced, relevant, diverse, and scalable to supercharge their AI models. As a Welocalize brand, Welo Data...Hourly payPart timeImmediate startRemote workWork from home10 hours per weekFlexible hoursWelo Data
New York, NY3 days ago- ...Join a 16-week research study evaluating AI assistants from the perspective of everyday users. We’re looking for people who already use Gemini, Instinct, or Muse and can assess prompts and responses in English. A part-time commitment of 20 hours per week is available...Full timePart timeImmediate start
Braintrust
New York, NY2 days ago - Vettedlance is seeking a meticulous evaluator to rate and rewrite AI model responses in Modern Standard Arabic. You will assess accuracy, fluency and cultural fit, and compare pairs of replies to explain your choices. You will also craft original Arabic prompts and gold...Remote jobHourly payContract workPart timeFlexible hours
$85 per hour
OpenTrain AI, Inc. is seeking a Process Safety Engineer AI Training Evaluator to review AI-generated content on environmental health and safety, process safety, and energy compliance. This remote, flexible project pays up to $85 per hour and requires strong written communication...Remote jobHourly payFlexible hours- LILT is seeking detail-oriented human reviewers with a strong understanding of local cultural context to support a range of AI training and evaluation projects. You will work across diverse task types, including evaluating prompts and AI-generated outputs, reviewing and...For contractorsLocal areaRemote workFlexible hours
- Outlier, operated by Scale AI, is seeking detail-oriented English speakers to review AI-generated media from home. You will roleplay live conversations with an AI voice model and rate outputs, then explain issues in writing for continuous model improvement. You can set...Remote jobWeekly payFreelanceFlexible hours
- German Voice AI Evaluator (Healthcare Front Desk Conversations) What if your native German fluency could help teach AI how to handle a real phone call to a doctor's office? We're looking for native German speakers based in the New York area to hold natural spoken conversations...Contract workWork at officeRemote workFlexible hours
- ...well as review other experts' items for quality and difficulty. The role is remote and hourly, part-time with 10-40 hours per week. Strong English writing and LaTeX fluency are required to produce clear, publishable materials for AI evaluation. #J-18808-Ljbffr VettedlanceRemote jobHourly payPart time10 hours per week
- AuraOne, Inc. seeks a Finance Research Expert — AI Evaluation (Remote) to review AI outputs in finance and risk workflows, ensuring accuracy and policy adherence. You will flag rounding, classification, and treatment issues, documenting corrections with citations for model...Remote job
$25 - $35 per hour
Assess how AI-generated versions differ from the original images and document visual errors in clear, precise notes. AI image systems... ...or digital imaging, graphic design or photo editing, image evaluation, data annotation, image comparison, and visual quality assurance...Remote jobHourly payPart timeFor contractorsFlexible hours- LILT seeks detail-oriented human reviewers to support AI training and evaluation. You will work as an independent contractor across tasks like evaluating prompts, reviewing text, and analyzing multimedia content. Your work depends on applying local cultural context, brand...Remote jobFor contractorsLocal area
$48 - $52 per hour
...time, language-based role based in the Netherlands. You will write expert Dutch prompts and classify content to help improve safety for AI models. The position requires native NL Dutch and business English, a bachelor’s degree (or in progress), and keen attention to...Hourly payPart time- Mindrift is seeking US-based physicians (MD/DO) to evaluate clinical EHR vignettes for a medical AI evaluation program. Projects are part-time and non-permanent, with compensation paid per accepted task. Contributors will review vignette content across multiple medical...Permanent employmentPart time
- Create challenging prompts and evaluate large language model responses for accuracy, reasoning, and usefulness. This worldwide, part-time... ...workforce planning, and operational execution to assess how well AI handles management-related topics. Create complex prompts that...Contract workPart timeFor contractorsRemote workWorldwide
$26 per hour
Prolific is seeking fluent Danish speakers to act as evaluators for AI training. You will compare text and AI-generated speech, assess naturalness, and flag cultural mismatches. A typical task pays up to $26/hr and can be completed at home with one hour of uninterrupted...Remote job- Revelo is seeking an Entertainment & Pop Culture Evaluator to help refine AI models by reviewing prompts and evaluating responses related to movies, TV, music, gaming, and celebrity culture. You will compare multiple AI outputs, select the best, and write concise justifications...FreelanceFlexible hours
- ...film/theater and visual effects professionals to join a leading AI lab's cutting-edge research team. You will compare AI-generated... ...contribute at least 20 hours per week on weekdays. Collaboration with other experts ensures evaluation data accuracy. #J-18808-Ljbffr MarketframeWeekday work
- The Work You will evaluate how AI models respond to sensitive questions about explosives and energetic materials. Scenarios may involve mining, commercial blasting, pyrotechnics, propellants, bomb disposal, investigations, and safety. You will use your professional judgment...Hourly payFor contractorsRemote work
- YO AI Labs in the United States is seeking a native English-speaking contractor to contribute to language data and AI training projects. You will evaluate audio content for nativeness, fluency, and pronunciation, and provide structured written feedback to improve AI models...Remote jobFor contractors
- Mindrift (Toloka) is seeking US-licensed physicians to evaluate clinical EHR vignettes for a medical AI evaluation project. You will review written cases, score reasoning quality, and write concise rationales. This is a task-based, independent contractor role with fully...Remote jobFor contractorsFlexible hours
- Mercor is hiring for a part-time Danish-English evaluative role to help make advanced AI models safer. You will craft expert-level prompts in Danish and apply guidelines to classify prompts and conversations, documenting your judgments with clear reasoning. No prior AI...Part timeImmediate start
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluator. Be the first to apply!
Related searches