Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Voice Evaluation Specialist [Remote]

$20 - $25 per hour

Innodata Inc.

New York, NY
  • Remote job

Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.

Scope of the Role: 

 

We are looking for detail-oriented Voice Specialists to evaluate the performance of AI models through real-time, voice-based conversations. In this role, you will interact with two different AI models using the same assigned scenario, carefully compare their responses, and provide structured evaluations based on defined quality criteria.

The goal of the evaluation is to create a fair, consistent comparison between models and identify which model delivers the stronger overall conversational experience.

What You’ll Own:

  • Review an assigned scenario and roleplay a natural conversation with two different AI models .
  • Conduct comparable conversations with Model A and Model B, keeping the scenario, conversational approach, and overall interaction as consistent as possible.
  • Maintain approximately the same number of conversational turns with each model to support a fair comparison.
  • Record your voice during each interaction and pay close attention to the quality and behavior of the model's responses.
  • Evaluate each model across five defined evaluation dimensions .
  • Identify and categorize relevant error clusters that may occur in the audio or conversation.
  • Compare the performance of Model A and Model B based on the evaluation criteria and your observations.
  • Select the model that demonstrated the stronger overall performance according to the defined evaluation framework.
  • Write a detailed rationale explaining your final preference, referencing specific examples and observations from both conversations.
  • Apply evaluation guidelines consistently across different scenarios and model interactions

You’ll Thrive in This Role If You Have

  • Bachelors Degree 
  • Strong attention to detail and ability to notice subtle differences in conversational quality.
  • Excellent listening and comprehension skills.
  • Strong written communication skills, particularly the ability to explain observations clearly and objectively.
  • Ability to follow detailed evaluation guidelines consistently.
  • Comfort speaking naturally and roleplaying different conversational scenarios.
  • Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
  • Strong critical-thinking and analytical skills.
  • Reliability and consistency when completing structured evaluation tasks.
  • Familiarity with AI assistants, voice-based AI, or conversational systems

The expected hourly salary range for this position is $20-$25 p/hour, based on experience, skills, and qualifications.

 

Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at  

If you believe you’ve been targeted by a recruitment scam, please report it to Innodata at  View email address on job-boards.greenhouse.io and consider reporting it to the FTC at  ReportFraud.ftc.gov .

Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the AI Voice Evaluation Specialist [Remote] in New York, NY vacancy
  • $45 - $70 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  .... Position: Behavioral Health Expert, AI Safety and Model Evaluation Type: Contract Compensation: $45–$70/hour Location... 
    Suggested
    Contract work
    Summer work
    Remote work
    Weekday work

    Mercor

    New York, NY
    18 days ago
  • $207k - $275k

    CoreWeave is The Essential Cloud for AI. Built for pioneers by...  ...market-maker. As a Solution Specialist for AI Runtime Services, you...  ...are the field's authoritative voice back to engineering, translating...  ...multi-stakeholder technical evaluations without losing momentum.Why... 
    Suggested
    Permanent employment
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours

    CoreWeave

    New York, NY
    2 days ago
  • $176.6k - $239k

     ...Market (GTM) at AWS using generative AI (GenAI)?AWS Worldwide Specialists Org (WWSO) is responsible for...  ...models) on AWS, model performance evaluations, develop demos and proof-of-concepts...  ...delivery, you will serve as a critical voice of the customer, collaborating directly... 
    Suggested
    Local area
    Worldwide
    Flexible hours

    AmazonWebServices

    New York, NY
    3 days ago
  •  ...About the Opportunity Our client, a leading AI research organization, is looking for experienced architects and architectural visualization specialists to support AI training and evaluation. You’ll develop plan-to-render tasks, assess floor plans, and contribute... 
    Suggested
    Afternoon shift

    Braintrust

    New York, NY
    4 hours ago
  • $100k - $125k

     ...Job Title: Senior IT Specialist Location: New York, NY (in-office) Salary: $100K – $125...  ...and outside IT providers • Manage the evaluation, selection, and onboarding process for new...  ..., accounting systems, and legal AI tools • A bachelor’s degree in a computer... 
    Suggested
    Work at office

    Addition Management

    New York, NY
    3 days ago
  •  ...About the role Calibration Technology Evaluation Specialist is a remote review track for evaluating AI outputs across calibration technology evaluation specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational... 
    Hourly pay
    For contractors
    Work experience placement
    Remote work
    10 hours per week

    Auraone

    New York, NY
    3 days ago
  • $110k - $138k

     ...organizations design, build and scale intelligent AI, digital, cybersecurity, cloud and...  ...will you be doing? The Computer Vision Specialist is a member of WWT’s Global Solutions &...  ...synthetic data, and on how models will be evaluated, optimized for edge deployment, monitored... 
    Full time
    Work experience placement
    Remote work
    Flexible hours
    Shift work

    World Wide Technology

    New York, NY
    3 days ago
  • $162.7k - $220.2k

     ...meaningful business changes with Agentic AI? The Agentic Development & Developing Agents...  ...as a Sr. Technical Business Development Specialist, Agentic AI for Application...  ...business teams across AWS and bring the voice of the customer into our product development... 
    Immediate start
    Worldwide
    Flexible hours

    AmazonWebServices

    New York, NY
    2 days ago
  • $142.8k - $193.2k

     ...moderate travel.About the teamThis position is part of the AWS Specialist and Partner Organization (ASP). Specialists own the end-to-end...  ...is fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do.Mentorship & Career GrowthWe... 
    Flexible hours

    AmazonWebServices

    New York, NY
    2 days ago
  • $147.9k - $200.1k

     ...multinational enterprises. You have domain expertise in Generative AI and key ML-powered use cases, and the technical depth to...  ...is fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do.Mentorship & Career GrowthWe’... 
    Worldwide
    Flexible hours

    AmazonWebServices

    New York, NY
    4 days ago
  •  ...Senior WW Partner GTM Specialist For Agentic Ai The AWS Data and AI Strategic Partner GTM team supports the world's most innovative partners...  ...fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do. Mentorship & Career... 
    Worldwide

    A W S

    New York, NY
    2 days ago
  • $8 - $65 per hour

     ...in French eager to shape the future of AI? Large-scale language models are evolving...  ...generation of AI. We're looking for healthcare specialists who live and breathe clinical workflows,...  ...to our prompt engineering and evaluation metrics. A master's degree in nursing, public... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    Meridial

    New York, NY
    4 days ago
  • $8 - $65 per hour

     ...Role overview Help improve AI capabilities for video production by evaluating model responses to practical creative and technical questions. The work draws on production expertise across planning, capture, and post-production, with a focus on checking accuracy and creative... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    Agency.com

    New York, NY
    1 day ago
  • $8 - $65 per hour

     ...Video Production Specialist - Freelance AI Trainer Project United States of America Are you a video production expert eager to shape the future...  ..., and suggest improvements to our prompt engineering and evaluation metrics. A bachelor’s or master’s degree in film, media production... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Immediate start
    Remote work

    Meridial

    New York, NY
    2 days ago
  • $8 - $65 per hour

     ...Korean-Japanese Bilingual Specialist - Freelance AI Trainer Project World Wide - Remote Are you a bilingual professional fluent in Korean and...  ...to training data. You’ll work with cutting‑edge AI tools, evaluate translations and conversations between Korean and Japanese... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work
    Worldwide

    Meridial

    New York, NY
    1 day ago
  • $85k - $95k

    Developer and Emerging Technologies Specialist The University of Virginia Library seeks a Developer...  ...expertise in artificial intelligence (AI), large language models, and emerging...  ...with a commitment to exploring and evaluating new tools and methods. This role collaborates... 
    Local area
    Remote work
    Visa sponsorship
    2 days per week

    Virginia Department of Human Resource Management

    New York, NY
    3 days ago
  • $70 - $110 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Dorsey . Position: Project Management Specialists Type: Contract Compensation: $...  ...+ hours/week Role Responsibilities Evaluate AI-generated project plans , status reports... 
    Hourly pay
    Contract work
    Summer work
    Immediate start
    Remote work

    Mercor

    New York, NY
    5 days ago
  • $45k - $65k

    Social Media Marketing Specialist Branch Metrics, Inc. New York, New York...  ...strategy across LinkedIn and evaluate opportunities on additional...  ...that reflects Branch's brand voice. Create and publish a mix of...  ...about emerging social platforms, AI tools, and content trends. Strong... 
    Work at office

    Branch Metrics, Inc.

    New York, NY
    5 days ago
  • $8 - $65 per hour

    Video Editing Specialist - Freelance AI Trainer Project United States of America Are you a video editing expert eager to shape the future of AI...  ..., and suggest improvements to our prompt engineering and evaluation metrics. A bachelor's or master's degree in Film, Media Production... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Immediate start
    Remote work

    Meridial

    New York, NY
    3 days ago
  • $8 - $65 per hour

    Audio Editing Specialist - Freelance AI Trainer Project North America - Remote Are you an audio editing expert eager to shape the future of AI...  ...traces, and suggest improvements to our prompt engineering and evaluation metrics. A bachelor's or master's degree in Audio... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Immediate start
    Remote work

    Meridial

    New York, NY
    3 days ago
  • $169k - $228.6k

     ...about pioneering the future of generative AI and eager to help the world's largest...  ...technology stacks • Define and implement evaluation frameworks for AI systems, including model...  ...join a focused, high-calibre group of specialist Solutions Architects supporting the world... 
    Flexible hours

    AmazonWebServices

    New York, NY
    1 day ago
  • $11 - $30.65 per hour

    Spanish Audio Specialist - Freelance AI Trainer Project World Wide - Remote A project dedicated to assessing and benchmarking advanced agentic...  ...models against leading systems. The program's mission is to evaluate and optimize model performance for real-world customer... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    Meridial

    New York, NY
    3 days ago
  •  ...Every day, our teams harness cutting-edge AI and breakthrough technologies to...  ...seeking a future team member for the role of Specialist Full Stack Engineer to join our Finance Hub...  ...product teams to design & develop solutions Evaluate and experiment with emerging AI... 
    Full time
    Worldwide
    Flexible hours

    The Bank of New York Mellon

    New York, NY
    4 days ago
  •  ...Job Description If you understand AI well enough to explain it in a customer meeting, and you get just as excited about solving a business problem as a technical one, these roles were written for you. Customers come to SAP with specific problems, a timeline, and a board... 
    Permanent employment
    Full time
    Flexible hours

    SAP

    New York, NY
    3 days ago
  • $182.8k - $247.3k

     ...strategy for technology that reimagines how AI agents consume governed context at...  ...across a segment or territory understand, evaluate, and adopt AWS Context. You will guide customers...  .... About the team: The customer-facing specialist Solutions Architecture team is the... 
    Local area
    Worldwide
    Flexible hours
    Day shift

    A W S

    New York, NY
    3 days ago
  •  ..., including Y Combinator and Box Group, and support all leading AI labs. About the Role AfterQuery is seeking Law Enforcement Supervisor...  ...AI models. In this remote, flexible role, you will design and evaluate supervisory scenarios that test the AI\'s ability to interpret... 
    Remote work
    Flexible hours
    Shift work

    AfterQuery Experts

    New York, NY
    2 days ago
  •  ...structure create an open environment where all feel comfortable to voice ideas, and we love the smart, creative minds of people who take...  ...in what they do. What You’ll Do Meet with clients and evaluate jewelry items for purchase in person Build and cultivate a... 
    Work experience placement
    Work at office
    Relocation

    CIRCA

    New York, NY
    10 days ago
  • $131k - $218.3k

     ...experience leveraging contemporary development tools, including AI-assisted coding workflows, to improve productivity and software...  ...through technical leadership, coaching, and knowledge sharing. Evaluate and adopt emerging development tools and practices that improve... 

    McKesson

    New York, NY
    3 days ago
  • $162.7k - $220.2k

     ...you passionate about the intersection of AI/ML and partner ecosystems? Do you thrive...  ...looking for a Senior-level WW Partner GTM Specialist to drive our worldwide AI GSI go-to-market...  ..., fresh perspectives, and passionate voices our teams bring to everything we do.Mentorship... 
    Local area
    Worldwide
    Flexible hours

    AmazonWebServices

    New York, NY
    5 days ago
  • $110.3k - $183.8k

     ...agile ceremonies, technical planning, and design reviews Leverage AI tools in your day-to-day development workflow Priority will be...  ...performance, experience and skills, equity, regular job market evaluations, and geographical markets. The pay range shown below is aligned... 
    Work at office

    McKesson

    New York, NY
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Voice Evaluation Specialist [Remote]. Be the first to apply!