Remote AI Agent Evaluation Scenario Writer (QA Testing)
Mindrift
New York, NY
- Remote job
Mindrift connects specialists with project-based AI opportunities for testing, evaluating, and improving AI systems. You’ll design tasks within realistic simulated environments and write tests that validate multiple correct solutions. This freelance, remote role requires 5+ years in software development and a core stack of Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis, plus QA/testing experience. English proficiency (B2+) is required. #J-18808-Ljbffr Mindrift
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Remote AI Agent Evaluation Scenario Writer (QA Testing) in New York, NY vacancy
$55 per hour
...experts with cutting-edge AI projects. We are seeking an Evaluation Scenario Writer - QA for a project focused... ...created for LLM agents. This is a flexible, project... ...validation, automated test thinking, and collaboration... ...part in a flexible, remote, freelance project that...Remote workPart timeFreelanceInternshipFlexible hours$80 per hour
...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is... ...masters Iterate based on feedback from expert QA reviewers who score your work on 7 quality criteria...SuggestedPermanent employmentTemporary work- ...technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will have a background in...Remote jobPart time
$60 per hour
...the future of AI. What We Do The... ...ambiguity, enjoy remote asynchronous... ...AI systems are tested and evaluated, we want to hear... ...We are seeking QA experts for autonomous AI agents in a project focused... ...tasks and scenarios for logic, completeness... ...with QA, writers, or developers...Remote workFreelanceFlexible hours$135.2k
...benchmark artificial intelligence agents. Creating robust evaluation harnesses ensures that AI models are tested against realistic software engineering scenarios. This work directly feeds into... .... How It Works During this remote session, you will review a series...Remote workHourly payContract work$55 per hour
A leading AI innovation firm in Dallas is seeking QAs for autonomous AI agents to ensure the quality of complex systems and scenarios. This flexible, remote project is ideal for those with excellent analytical... ...should be adept at evaluating scenarios and documenting findings...Remote jobFlexible hours$60 per hour
...A leading AI firm in Austin is looking for QA experts to validate and improve AI systems. This remote, freelance role requires strong analytical... ...Candidates will review AI evaluation tasks, identify inconsistencies... ...expected behaviors for agents. Ideal applicants have...Remote workFreelanceMind Rift
Austin, TX5 days ago- ...RemoteJobsOne is seeking an Everyday AI Agent Tester for a fully remote contractor role based in the United States. You will run real-life scenarios with two AI agents, screen record each attempt, and score results on completion, quality, control issues, and user satisfaction...Remote workFor contractorsStart working today
RemoteJobsOne
Jacksonville, FL4 hours ago $60 per hour
...Missouri is seeking contributors for a part-time QA project focused on autonomous AI agents. This flexible remote opportunity requires strong analytical and... ...structured data formats. Candidates will review evaluation tasks, identify inconsistencies, and help define...Remote jobPart timeFlexible hours- OpenTrain AI, Inc. is seeking a remote, part-time contractor to test two AI agents on a smartphone, completing realistic errands and recording outcomes. You will evaluate task completion, quality, control issues, user satisfaction, and time, then provide clear feedback...Remote jobPart timeFor contractorsFlexible hours
$180k - $200k
...Arbiter is the AI-powered care orchestration... ...As a QA Engineer reporting... ...dropped calls, wrong agent responses, failed handoffs... ...workflow changes: test scripts, edge cases... ...Exposure to LLM evaluation, prompt testing, or... ...This role will be remote or onsite in our New...Remote workFull timeWork experience placementWork at office$25 - $45 per hour
...spam folder. This is a fully remote position open to candidates... ...: $25$45/hr Role Title: AI Agent Evaluator Role Type: Contractor... ...real user. Thats you. Youll test two AI agents side by side... ...youll do # Run real-life scenarios: bills returns email follow-...Remote workFull timeFor contractorsWork at officeWeekend workRemoteJobsOne
Houston, TX4 days ago- ...RemoteJobsOne in the United States is hiring an AI Agent Evaluator on a fully remote, contractor basis. You’ll test two AI agents, run real-life scenarios, and rate completion, quality, and user experience from a real user perspective. Requirements include a US-based...Remote workFor contractors
RemoteJobsOne
Houston, TX4 hours ago $25 - $45 per hour
...This is a fully remote position , open to candidates based in... ...$25–$45/hr Role Title: AI Agent Evaluator Role Type: Contractor... ...user. That’s you. You’ll test two AI agents side by side... ...you’ll do Run real-life scenarios: bills, returns, email follow...Remote workFor contractorsWork at officeWeekend workRemoteJobsOne
Houston, TX4 hours ago$80 per hour
...-time opportunity focused on quality assurance for autonomous AI agents. You will analyze complex systems, review tasks for logic, and... ...analytical and detail-oriented skills, with experience in policy evaluation or logic puzzles preferred. Compensation can reach up to $80/...Remote jobPart timeFlexible hours- ...currently seeks I Agent Engineer to join our team in remotely. What You'... ...deploy enterprise-grade AI agents that support... ...components. Test and validate agent... ...of real-world scenarios, including edge cases... ..., including how to evaluate and iterate on model...Remote workHourly payTemporary workFlexible hours
The Nippon Telegraph and Telephone Corporation (NTT)
United States4 days ago $80 per hour
A technology company is seeking a part-time QA contributor for an AI project to validate autonomous agents. Candidates must have strong analytical thinking, attention... ...expected behaviors for AI agents. This flexible remote opportunity allows you to work on your own...Remote jobPart timeFlexible hours- ...Frontier AI Lab is seeking a Generalist Writer for a remote, contractor position. The role focuses on reviewing AI-generated and human-written content for... ...completeness, with rigorous attention to guidelines and evaluation criteria. You will deliver precise, actionable...Remote workFor contractorsImmediate start
Remotebridge
Kentucky4 hours ago $30 - $60 per hour
...QA Engineer – AI Training & Evaluation Please double check you have the right level of... ...Contractor Location: Remote Compensation: $30–$60/... ...evaluation workflows by testing software, reviewing AI-generated... ...review test cases, test scenarios, evaluation criteria, bug...Remote workFor contractorsWork from homeCodefeast
United, PA10 days ago- Equiliem is seeking an AI Evaluation/Test Engineer to test AI-powered agent experiences across customer and internal workflows... ...on prompt and agent evaluation, QA testing, observability, and defect... ...and user experience in AI agents. Remote work is offered. #J-18808-Ljbffr...Remote job
- YO AI Labs is seeking an Adobe Marketing Technology Expert to support an AI training and evaluation project focused on enterprise marketing operations. The role involves testing workflows with Adobe Workfront, AEM, CJA,... ...Experience Cloud from a remote setting. The ideal...Remote job
- ...dropped calls, incorrect agent responses, failed... ...Perform pre-launch QA of scripts, edge... ...Build and maintain test cases, scoring... ...experience includes voice AI, conversational AI,... ...includes LLM evaluation, prompt testing, conversation... ...programs. Remote work or onsite work...Remote workFull timeWork at office
- ...Help evaluate natural, voice-driven interactions with an advanced AI assistant by completing everyday tasks in Brazilian Portuguese and documenting how the assistant responds. Role Overview In this remote, per-task role, you will use a specialized application on your...Remote workImmediate start
SaidGig
United States10 days ago - aFit Staffing, Inc. is aWoman-owned staffing firm based in Indianapolis, IN, seeking a User Testing Scenario Writer for a remote role with travel to Sacramento, CA. You will craft detailed test scenarios from user stories and process docs, ensuring coverage of workflows...Remote job
$190.9k - $334.1k
...ServiceNow is the AI control tower... ...Engineer, Voice AI Agent Engineering... ...through automated evaluation that the result... ...evaluation and test non-deterministic... ...calibrated to human QA review, CI gates... ...call datasets, scenario suites, model-as... ...(flexible, remote, or required in...Remote workFull timeWork at officeLocal areaImmediate startFlexible hours- ...first truly converged AI workspace unifying... ...AI daily, and we evaluate AI fluency as part... ...Engineer - Multi-Agent Frameworks to join... ...frameworks for testing AI agents in challenging... ...and complex scenarios, focusing on individual... ...Agent Systems.#LI-REMOTE#LI-AK2#LI-CC1Equal...Remote work
$40 - $50 per hour
...Overview Help improve the accuracy, reliability, and overall performance of next-generation AI systems by evaluating written content and AI-generated responses. This remote contract role calls for precise writing judgment, analytical thinking, and consistent application...Remote workHourly payContract workImmediate startSaidGig
Remotemore than 2 months ago- ...first truly converged AI workspace unifying... ...AI daily, and we evaluate AI fluency as part... ...Engineer – Multi-Agent Frameworks to join... ...frameworks for testing AI agents in challenging... ...and complex scenarios, focusing on individual... ...Systems. #LI-REMOTE #LI-AK2 #LI-...Remote workFull timeVisa sponsorshipWork visa
ClickUp
Remotea month ago $150k - $250k
Senior Applied AI Engineer, Agent Quality & Evaluations Building AI dev tools and evaluation... ...). Create, test, version, and document prompts... ...representative datasets, behavioral scenarios, scoring rubrics,... ...learning and development. Remote-first company with globally...Remote workFull timeTemporary workFlexible hours$90 - $140 per hour
...Role Overview Apply journalistic and editorial judgment to evaluate AI generated writing and help improve how next generation AI systems learn, reason, and perform. This remote contract role focuses on assessing written responses for accuracy, clarity, usefulness, tone...Remote workHourly payContract workSaidGig
United States27 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Remote AI Agent Evaluation Scenario Writer (QA Testing). Be the first to apply!
Related searches