AI Voice Evaluation Specialist [Remote]
$20 - $25 per hourInnodata Inc.
- Remote job
Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.
Scope of the Role:
We are looking for detail-oriented Voice Specialists to evaluate the performance of AI models through real-time, voice-based conversations. In this role, you will interact with two different AI models using the same assigned scenario, carefully compare their responses, and provide structured evaluations based on defined quality criteria.
The goal of the evaluation is to create a fair, consistent comparison between models and identify which model delivers the stronger overall conversational experience.
What You’ll Own:
- Review an assigned scenario and roleplay a natural conversation with two different AI models .
- Conduct comparable conversations with Model A and Model B, keeping the scenario, conversational approach, and overall interaction as consistent as possible.
- Maintain approximately the same number of conversational turns with each model to support a fair comparison.
- Record your voice during each interaction and pay close attention to the quality and behavior of the model's responses.
- Evaluate each model across five defined evaluation dimensions .
- Identify and categorize relevant error clusters that may occur in the audio or conversation.
- Compare the performance of Model A and Model B based on the evaluation criteria and your observations.
- Select the model that demonstrated the stronger overall performance according to the defined evaluation framework.
- Write a detailed rationale explaining your final preference, referencing specific examples and observations from both conversations.
- Apply evaluation guidelines consistently across different scenarios and model interactions
You’ll Thrive in This Role If You Have
- Bachelors Degree
- Strong attention to detail and ability to notice subtle differences in conversational quality.
- Excellent listening and comprehension skills.
- Strong written communication skills, particularly the ability to explain observations clearly and objectively.
- Ability to follow detailed evaluation guidelines consistently.
- Comfort speaking naturally and roleplaying different conversational scenarios.
- Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
- Strong critical-thinking and analytical skills.
- Reliability and consistency when completing structured evaluation tasks.
- Familiarity with AI assistants, voice-based AI, or conversational systems
The expected hourly salary range for this position is $20-$25 p/hour, based on experience, skills, and qualifications.
Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at
If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at View email address on job-boards.greenhouse.io and consider reporting it to the FTC at ReportFraud.ftc.gov .
$45 - $70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... .... Position: Behavioral Health Expert, AI Safety and Model Evaluation Type: Contract Compensation: $45–$70/hour Location...SuggestedContract workSummer workRemote workWeekday work$207k - $275k
CoreWeave is The Essential Cloud for AI. Built for pioneers by... ...market-maker. As a Solution Specialist for AI Runtime Services, you... ...are the field's authoritative voice back to engineering, translating... ...multi-stakeholder technical evaluations without losing momentum.Why...SuggestedPermanent employmentFull timeTemporary workCasual workWork at officeFlexible hours$176.6k - $239k
...Market (GTM) at AWS using generative AI (GenAI)?AWS Worldwide Specialists Org (WWSO) is responsible for... ...models) on AWS, model performance evaluations, develop demos and proof-of-concepts... ...delivery, you will serve as a critical voice of the customer, collaborating directly...SuggestedLocal areaWorldwideFlexible hours- ...About the Opportunity Our client, a leading AI research organization, is looking for experienced architects and architectural visualization specialists to support AI training and evaluation. You’ll develop plan-to-render tasks, assess floor plans, and contribute...SuggestedAfternoon shift
$100k - $125k
...Job Title: Senior IT Specialist Location: New York, NY (in-office) Salary: $100K – $125... ...and outside IT providers • Manage the evaluation, selection, and onboarding process for new... ..., accounting systems, and legal AI tools • A bachelor’s degree in a computer...SuggestedWork at office- ...About the role Calibration Technology Evaluation Specialist is a remote review track for evaluating AI outputs across calibration technology evaluation specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational...Hourly payFor contractorsWork experience placementRemote work10 hours per week
$110k - $138k
...organizations design, build and scale intelligent AI, digital, cybersecurity, cloud and... ...will you be doing? The Computer Vision Specialist is a member of WWT’s Global Solutions &... ...synthetic data, and on how models will be evaluated, optimized for edge deployment, monitored...Full timeWork experience placementRemote workFlexible hoursShift work$162.7k - $220.2k
...meaningful business changes with Agentic AI? The Agentic Development & Developing Agents... ...as a Sr. Technical Business Development Specialist, Agentic AI for Application... ...business teams across AWS and bring the voice of the customer into our product development...Immediate startWorldwideFlexible hours$142.8k - $193.2k
...moderate travel.About the teamThis position is part of the AWS Specialist and Partner Organization (ASP). Specialists own the end-to-end... ...is fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do.Mentorship & Career GrowthWe...Flexible hours$147.9k - $200.1k
...multinational enterprises. You have domain expertise in Generative AI and key ML-powered use cases, and the technical depth to... ...is fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do.Mentorship & Career GrowthWe’...WorldwideFlexible hours- ...Senior WW Partner GTM Specialist For Agentic Ai The AWS Data and AI Strategic Partner GTM team supports the world's most innovative partners... ...fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do. Mentorship & Career...Worldwide
$8 - $65 per hour
...in French eager to shape the future of AI? Large-scale language models are evolving... ...generation of AI. We're looking for healthcare specialists who live and breathe clinical workflows,... ...to our prompt engineering and evaluation metrics. A master's degree in nursing, public...Hourly payContract workFor contractorsFreelanceRemote work$8 - $65 per hour
...Role overview Help improve AI capabilities for video production by evaluating model responses to practical creative and technical questions. The work draws on production expertise across planning, capture, and post-production, with a focus on checking accuracy and creative...Hourly payContract workFor contractorsFreelanceRemote work$8 - $65 per hour
...Video Production Specialist - Freelance AI Trainer Project United States of America Are you a video production expert eager to shape the future... ..., and suggest improvements to our prompt engineering and evaluation metrics. A bachelor’s or master’s degree in film, media production...Hourly payContract workFor contractorsFreelanceImmediate startRemote work$8 - $65 per hour
...Korean-Japanese Bilingual Specialist - Freelance AI Trainer Project World Wide - Remote Are you a bilingual professional fluent in Korean and... ...to training data. You’ll work with cutting‑edge AI tools, evaluate translations and conversations between Korean and Japanese...Hourly payContract workFor contractorsFreelanceRemote workWorldwide$85k - $95k
Developer and Emerging Technologies Specialist The University of Virginia Library seeks a Developer... ...expertise in artificial intelligence (AI), large language models, and emerging... ...with a commitment to exploring and evaluating new tools and methods. This role collaborates...Local areaRemote workVisa sponsorship2 days per week$70 - $110 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Project Management Specialists Type: Contract Compensation: $... ...+ hours/week Role Responsibilities Evaluate AI-generated project plans , status reports...Hourly payContract workSummer workImmediate startRemote work$45k - $65k
Social Media Marketing Specialist Branch Metrics, Inc. New York, New York... ...strategy across LinkedIn and evaluate opportunities on additional... ...that reflects Branch's brand voice. Create and publish a mix of... ...about emerging social platforms, AI tools, and content trends. Strong...Work at office$8 - $65 per hour
Video Editing Specialist - Freelance AI Trainer Project United States of America Are you a video editing expert eager to shape the future of AI... ..., and suggest improvements to our prompt engineering and evaluation metrics. A bachelor's or master's degree in Film, Media Production...Hourly payContract workFor contractorsFreelanceImmediate startRemote work$8 - $65 per hour
Audio Editing Specialist - Freelance AI Trainer Project North America - Remote Are you an audio editing expert eager to shape the future of AI... ...traces, and suggest improvements to our prompt engineering and evaluation metrics. A bachelor's or master's degree in Audio...Hourly payContract workFor contractorsFreelanceImmediate startRemote work$169k - $228.6k
...about pioneering the future of generative AI and eager to help the world's largest... ...technology stacks • Define and implement evaluation frameworks for AI systems, including model... ...join a focused, high-calibre group of specialist Solutions Architects supporting the world...Flexible hours$11 - $30.65 per hour
Spanish Audio Specialist - Freelance AI Trainer Project World Wide - Remote A project dedicated to assessing and benchmarking advanced agentic... ...models against leading systems. The program's mission is to evaluate and optimize model performance for real-world customer...Hourly payContract workFor contractorsFreelanceRemote work- ...Every day, our teams harness cutting-edge AI and breakthrough technologies to... ...seeking a future team member for the role of Specialist Full Stack Engineer to join our Finance Hub... ...product teams to design & develop solutions Evaluate and experiment with emerging AI...Full timeWorldwideFlexible hours
- ...Job Description If you understand AI well enough to explain it in a customer meeting, and you get just as excited about solving a business problem as a technical one, these roles were written for you. Customers come to SAP with specific problems, a timeline, and a board...Permanent employmentFull timeFlexible hours
$182.8k - $247.3k
...strategy for technology that reimagines how AI agents consume governed context at... ...across a segment or territory understand, evaluate, and adopt AWS Context. You will guide customers... .... About the team: The customer-facing specialist Solutions Architecture team is the...Local areaWorldwideFlexible hoursDay shift- ..., including Y Combinator and Box Group, and support all leading AI labs. About the Role AfterQuery is seeking Law Enforcement Supervisor... ...AI models. In this remote, flexible role, you will design and evaluate supervisory scenarios that test the AI\'s ability to interpret...Remote workFlexible hoursShift work
- ...structure create an open environment where all feel comfortable to voice ideas, and we love the smart, creative minds of people who take... ...in what they do. What You’ll Do Meet with clients and evaluate jewelry items for purchase in person Build and cultivate a...Work experience placementWork at officeRelocation
$131k - $218.3k
...experience leveraging contemporary development tools, including AI-assisted coding workflows, to improve productivity and software... ...through technical leadership, coaching, and knowledge sharing. Evaluate and adopt emerging development tools and practices that improve...$162.7k - $220.2k
...you passionate about the intersection of AI/ML and partner ecosystems? Do you thrive... ...looking for a Senior-level WW Partner GTM Specialist to drive our worldwide AI GSI go-to-market... ..., fresh perspectives, and passionate voices our teams bring to everything we do.Mentorship...Local areaWorldwideFlexible hours$110.3k - $183.8k
...agile ceremonies, technical planning, and design reviews Leverage AI tools in your day-to-day development workflow Priority will be... ...performance, experience and skills, equity, regular job market evaluations, and geographical markets. The pay range shown below is aligned...Work at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Voice Evaluation Specialist [Remote]. Be the first to apply!
- ai data scientist New York, NY
- ai scientist New York, NY
- remote data architect New York, NY
- remote data manager New York, NY
- remote property manager New York, NY
- office manager remote New York, NY
- it manager remote New York, NY
- claims specialist remote New York, NY
- project manager remote europe New York, NY
- remote video game New York, NY






