Everyday AI Agent Tester [Remote]
AuraOne Human Data
Remote
- Remote job
Everyday AI Agent Tester is a remote review track for evaluating AI outputs across everyday ai agent tester specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational risk; and document the right next step so the modeling team can train on it.
Why this role matters
Everyday AI Agent Tester specialist operations AI has to fit into an actual day at work. AuraOne uses experienced operators to grade outputs the way a senior peer would — checking workflow, policy, and the unwritten rules that decide whether a task actually gets done.
Responsibilities
- Review AI outputs against current everyday ai agent tester specialist operations workflows, playbooks, and firm policy for Everyday AI Agent Tester assignments.
- Grade tone, escalation logic, and stakeholder fit on a structured rubric.
- Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
- Capture the right next step so the modeling team can train on it.
- Adjudicate disputed treatments against published playbooks or firm guidance.
- Maintain reviewer-quality scores in inter-rater calibration cycles.
Qualifications
- Direct working experience in everyday ai agent tester specialist operations on real teams for Everyday AI Agent Tester work.
- Comfort applying multi-page rubrics consistently across long batches.
- Clear written reasoning that names the policy or workflow being applied.
- Strong attention to detail and the ability to flag when a prompt itself is the problem.
- Reliable async availability for at least 10 hours per week.
Example tasks
- Grade a model's response to a real everyday ai agent tester specialist operations ticket and rate workflow, tone, and escalation.
- Flag a missed escalation with the right severity tag and corrected next step.
- Adjudicate a disputed playbook call between two reviewers using firm guidance.
- Audit a 25-row batch for rubric consistency and report drift to the program lead.
Nice to have
- Prior experience training, calibrating, or QA-ing operations teams.
- Familiarity with AI-assisted workflow tooling and its failure modes.
- Bilingual experience for cross-region operations.
Skills
- Operational review
- Policy adherence
- Workflow judgment
- Stakeholder communication
- Everyday AI Agent Tester specialist operations
- AI model evaluation
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Everyday AI Agent Tester [Remote] in Remote vacancy
$25 - $45 per hour
...spam folder. This is a fully remote position open to candidates based in United States. Pay: $25$45/hr Role Title: Everyday AI Agent Tester Role Type: Contractor Location: U.S based Opportunity Snapshot Pay: $25-45/hour Time per task: 1.5 hours...SuggestedFull timeFor contractorsWork at officeRemote workWeekend workRemoteJobsOne
Jacksonville, FL3 days ago$85 per hour
...Category: Enterprise Workflow & Tool Use · Pay: $85 / hr · Location: Remote — US-eligible · Contractor Everyday AI Agent Tester is a remote review track for evaluating AI outputs across everyday ai agent tester specialist operations workflows. Reviewers grade workflow...SuggestedHourly payContract workFor contractorsWork experience placementRemote work10 hours per weekAuraOne
San Jose, CA2 days ago- RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction...SuggestedRemote jobFor contractorsStart working today
- ...Dover is seeking an AI Agent Evaluator contractor based in the United States to test two AI agents by performing real-life tasks and recording outcomes. You will assess usability, accuracy, and user experience, providing detailed feedback from your own accounts on your...SuggestedFor contractorsRemote workFlexible hours
$40 - $120 per hour
...AI Chatbot Tester $40-120/hr Remote Freelance GENERAL About the Role What if your curiosity and critical eye could help shape the... ...chatbots in conversations spanning a wide range of topics — from everyday questions to complex or creative scenarios Evaluate...SuggestedHourly payContract workFreelanceRemote workWorldwideFlexible hoursAlignerr
United States1 day ago- ...curiosity and sharp instincts could help shape the AI tools millions of people rely on every day? We're looking for AI Chatbot Testers to engage directly with cutting-edge AI... ...across a wide range of topics — from everyday questions to creative and complex scenarios...Hourly payContract workFreelanceRemote workFlexible hours
Alignerr Corp.
New York, NY1 day ago - ...Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Required Skills: ChatGPT Muse AI agents... ...personal and a smartphone that can screen record Your tasks: Run everyday errands with two AI agents, approve key steps, screen record, and...Remote jobContract workFor contractorsWork at officeWeekend work
$30 - $90 per hour
...Help improve next-generation AI systems by applying practical expertise in workplace... ...world workflows and feedback that make AI agents more effective in professional settings.... ...clear, repeatable guidance for using AI in everyday professional tasks. Use AI agents to...Hourly payFor contractorsRemote workSaidGig
United Statesa month ago- ...We're hiring for our partner company, an early-stage startup building an AI coworker: an agent that lives inside a team's everyday tools and actually gets work done. They need a senior Founding Harness Engineer to build the layer that makes it behave. How it pulls in context...Full timeContract workRemote work
- You'll join the team designing Okta for AI Agents - one of Okta's newest and most critical strategic initiatives, providing identity management... ...+ Wellness Financial Benefits Pay + Incentives Time Off Everyday Living Resources Experience designing for a product with a...Work experience placementWork from homeShift work
- ...Description Driver Skills Test Examiner (Driver License Agent I) - Grade 10 Current part time vacancies exist within the MVA Oakland Branch (Garrett County) only. **Previous Applicants Need Not Reapply** This is an interview and hire position...Full timePart timeWork experience placementH1bImmediate startVisa sponsorshipFree visaAfternoon shift
Maryland Department of Transportation
Oakland, MD5 days ago $200k - $275k
...Build a Safer World. TRM Labs provides AI-powered intelligence solutions that help... ...integrating cutting-edge tools in the LLM and agent space - including open-source stacks,... ...outcomes endtoend, and invest in getting better everyday. Inspiring Colleague: We add clarity...Local areaRemote workWorldwideTRM
United States2 days ago- ...AI Agent Builder Right hand to the CEO | ClearDesk + incubated companies | Full-time ClearDesk is a California-based global staffing company supporting US small and mid-sized businesses in home care, real estate, bookkeeping, and home services. The founders are also...Full timeWork at officeRemote work
ClearDesk
San Diego, CA17 hours ago $216k - $270k
About Scale AIScale AI is the data foundation for AI, helping organizations build and deploy reliable production AI applications. We... ...manufacturing, media and telecommunications to build production AI agents that automate complex workflows, help humans, reason over...Full timeScale AI
San Francisco, CA4 days ago$90 - $140 per hour
...AI Agent Power User Role Type: Contractor Location: Remote US Canada UK Ireland Australia New Zealand We are seeking experienced AI Agent Power Users to contribute to an enterprise AI training project focused on evaluating and improving AI assistants in real...Full timeContract workFor contractorsRemote workYO AI Labs
Austin, TX9 days ago$147.6k - $274k
...with the people we love. That’s what makes us Roche.Advances in AI, data, and computational sciences are transforming drug discovery... ...Learning Scientist / Senior Machine Learning Scientist building agents for applied small-molecule drug design. You will develop autonomous...Full timeLocal areaWorldwideRelocation packageGenentech
New York, NY5 days ago- ...AI Agent Evaluation Specialist This project aims to test and evaluate one AI agent in the development stage. By asking questions and requests, based on various categories (the training is provided), you will try to force the AI agent to say something harmful, offensive...Full timeRemote work
TSMG
United States12 hours ago - ...AI Chatbot Tester MCI is one of the fastest-growing tech-enabled business services companies in the USA, with a strong call center footprint... .... Our contact centers are powered by both on-site and remote agents, leveraging advanced technologies to enhance customer...Temporary workWork at officeLocal areaRemote work
AI Support Positions
United States3 days ago - ...IQ Media Agent bietet eine zu 100% remote arbeitende Nebenbeschäftigung mit flexibler Zeiteinteilung. Der monatliche Zusatzverdienst kann... ...von Einsatz und Erfolg. Du arbeitest eigenverantwortlich mit AI-Tools, baust dein eigenes Online-Business auf und profitierst von...Remote work
IQ Mediengruppe LLC
Sheridan, WY1 hour ago - ...overseeing Piper's intelligence and behavior across chat, voice, video, and email channels. You will define success metrics, partner with AI/engineering teams, and drive end-to-end improvements in Piper’s conversation quality for a fast-moving B2B SaaS product. The role...Remote job
$200k - $275k
...Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted... ...on this mission. If you are too, let's talk. About Okta for AI Agents Okta secures access for 20,000 organizations and billions of...Local areaRemote workWorldwideFlexible hoursShift workOkta, Inc.
San Francisco, CA12 hours ago$32.5 - $36 per hour
...Utilize resources such as government websites, professional tools, and team expertise to deliver accurate tax solutions in personalized everyday language. \n \n \n Deliver a Best in Class Customer Experience \n \n Create an exceptional customer experience through...Local areaIntuit
Fontana, CA6 days ago- ...that everyone else has simply learned to live with. We value candidates who have the instinct to spot a "normalized" problem and the AI-native curiosity to create a solution using the latest tools. Our culture is built on iteration, leveraging AI to ship faster today to...Internship
$225.2k - $394.1k
...the role We are hiring a Distinguished Product Manager to lead the product vision and execution for a transformational frontier Ai agent currently in MVP state. This role brings together cutting-edge autonomous agent capabilities with enterprise governance...Full timeWork at officeRemote workFlexible hoursShift workServiceNow
San Diego, CA7 days ago- ...The Company Our client is building long-horizon AI agents and an agent-driven distribution platform, starting with relationship-native... ...-stack engineer to turn research and agent capabilities into everyday experiences people can use and trust. You will move from an...Remote work
Pivot Search
United, PA3 days ago $62.4k - $83.2k
...approximately 6,165 employees and has served more than 200,000 customers across over 100 countries and regions. Job Title: AI Intern, Enterprise Agent Development Supervisor: Head of AI for Industry, Digital & Intelligent Technology Center Location: Piscataway...Hourly payTemporary workInternshipRemote workFlexible hours- ...Staff Security Engineer – AI / Agent Security Remote (US) | Mid-$200Ks + Early-Stage Equity We’re partnering with a fast-growing, early-stage cybersecurity company building the security layer for enterprise AI agents, MCP servers, skills, and plugins. We’re...Remote work
Colossus Technologies Group
United States3 days ago $150k - $235k
A fast-growing AI startup is seeking a Technical Product Manager to lead their AI agent builder platform. The ideal candidate will have over 5 years of experience in B2B SaaS product management and a strong technical background. This role involves collaborating with engineering...Remote workThe Resume Database
San Francisco, CA13 hours ago- Vercel in San Francisco, CA is seeking a Product Manager to own the compute platform—Fluid compute, Vercel Functions, Sandbox, and more—and set the direction for how developers configure, scale, and run code on Vercel. You will lead a cross-functional effort shaping...Work at officeRemote work
$20 - $30 per hour
...Immediate ABOUT THE ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open... ...processes, build failures, and containers. • Workflow — Everyday Git, diffs, pull requests, and issue tracking. • Debugging...Hourly payFull timeContract workImmediate startRemote workBespoke Labs
Phoenix, AZ3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Everyday AI Agent Tester [Remote]. Be the first to apply!
