Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Health AI Conversation Evaluator - AI Trainer

$20 - $160 per hour

Mercor

Job Description

Job Description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote

Role Responsibilities

  • Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
  • Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
  • Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
  • Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
  • Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications

Must-Have

  • Strong written English skills.
  • Ability to explain judgments in two to four specific sentences.
  • Careful reading to notice omissions in responses.
  • Comfort with applying a detailed rubric consistently.
  • Reliable availability for time-boxed batches.

Preferred

  • Experience in annotation, human-feedback, evaluation, or content review.
  • Background in explaining technical concepts to non-experts.
  • Experience with rubric-based grading or quality assurance.

Important

  • Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

#hiringmercor
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Health AI Conversation Evaluator - AI Trainer in New York, NY vacancy
  • $80 - $120 per hour

     ...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Larry Summers , and Jack Dorsey . Position: Public health communications Evaluator Type: Contract Compensation: $80–$120/hour... 
    Suggested
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    New York, NY
    12 days ago
  • $137.3k - $166k

     ...that when people achieve their health goals, they can achieve their...  ...by more fully unlocking Conversational Analytics. You thrive in a startup...  ...documentation, context, and evaluations to strengthen our LLMs’...  ...materials Develop and maintain AI evaluations for... 
    Suggested
    Full time
    Work experience placement
    Local area

    Ro

    New York, NY
    4 days ago
  • $100 per hour

     ...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our...  ...Summers , and Jack Dorsey . Position: Clinical Mental Health Expert — AI Conversation Evaluation Type: Contract Compensation: $100/hour... 
    Suggested
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    20 days ago
  • $117.6k - $161.7k

     ...solutions enabling the efficient conversion of structured and...  ...or data requires an in-depth evaluation of variable factors.OverviewHumana...  ...our mission of improving the health and well-being of our consumers...  ...Genie)Leverage Databricks Genie (AI/BI capabilities) to enable... 
    Suggested
    Full time
    Temporary work
    Apprenticeship
    Work at office
    Remote work
    Work from home
    Home office

    Humana

    New York, NY
    2 days ago
  • $50 - $190 per hour

     ...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco,...  ...: 20+ hours/week Role Responsibilities Evaluate AI systems on complex personal workflows, including personal health , travel , and activity planning .... 
    Suggested
    Hourly pay
    Contract work
    For contractors
    Summer work
    Remote work
    Trial period

    Mercor

    New York, NY
    12 days ago
  • $8 - $65 per hour

     ...Language Specialist - Freelance AI Trainer Project United States of...  .... On a typical day, you will converse with the model on language scenarios...  ...our prompt engineering and evaluation metrics. A master’s or PhD in...  ...‑sponsored benefits such as health insurance and PTO do not... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Immediate start
    Remote work

    Meridial

    New York, NY
    19 hours ago
  • $198.72k - $260.82k

     ...Engineer to join our Data team.Oscar is the first health insurance company built around a full stack...  ...across web and app, our member facing health AI agent (Oswell), tools used by Oscar teammates during member conversations, and intentional outbound engagement.The work... 
    Full time
    Work at office
    Flexible hours

    Oscar Health Insurance

    New York, NY
    6 hours ago
  • $8 - $65 per hour

     ...to shape the future of AI? Large-scale language models...  ..., diagnostics, public health, health systems...  ...typical day, you will converse with the model in both...  ...prompt engineering and evaluation metrics. A master's degree...  ...Fluent in French) - AI Trainer Employment type: Contract... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    Meridial

    New York, NY
    1 day ago
  • YO AI Labs is seeking experienced physicians to support an advanced healthcare AI project. This part-time contractor role involves applying clinical expertise to evaluate AI-generated medical content, analyze cases, and improve AI systems used in healthcare decision-making... 
    Remote job
    Part time
    For contractors

    YO AI Labs

    New York, NY
    4 days ago
  • $80 - $120 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Jack Dorsey . Position: Data analysis / quantitative readouts Evaluator Type: Contract Compensation: $80–$120/hour... 
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    New York, NY
    1 day ago
  • $80 - $120 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...and Jack Dorsey . Position: Biology / environmental science Evaluator Type: Contract Compensation: $80–$120/hour Location... 
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    New York, NY
    1 day ago
  • $40 - $125 per hour

     ...DataAnnotation is looking for an experienced Computational Chemist - AI Trainer to design chemistry problems that today's leading AI models...  ...ask for any money from you. PayPal will handle any currency conversions from USD. Only applicants in Australia, Canada, Ireland, New... 
    Hourly pay
    For contractors

    DataAnnotation

    New York, NY
    2 days ago
  • $8 - $65 per hour

     ...to shape the future of AI? Large-scale language models...  ...typical day, you will converse with the model in...  ...prompt engineering and evaluation metrics. A Master's degree...  ...benefits such as health insurance and PTO do not...  ...) Language Expert - AI Trainer Employment type: Contract... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    Agency.com

    New York, NY
    19 hours ago
  • $155k - $410k

     ...advanced data science methods to complex health payer challenges, using statistical...  ...client needs- Leading executive-level client conversations across multiple healthcare data efforts-...  ...closely with team members. We evaluate these factors thoughtfully to establish... 
    Full time
    H1b

    PwC

    New York, NY
    4 days ago
  • $8 - $65 per hour

     ...Production Specialist - Freelance AI Trainer Project United States of...  .... On a typical day, you will converse with the model on real‑world...  ...to our prompt engineering and evaluation metrics. A bachelor’s or master...  ...‑sponsored benefits such as health insurance and PTO do not... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Immediate start
    Remote work

    Meridial

    New York, NY
    4 days ago
  • $8 - $65 per hour

     ...Japanese Bilingual Specialist - Freelance AI Trainer Project World Wide - Remote Are you...  ...work with cutting‑edge AI tools, evaluate translations and conversations between Korean and Japanese, and...  ...company‑sponsored benefits such as health insurance and PTO do not apply.... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work
    Worldwide

    Meridial

    New York, NY
    3 days ago
  •  ...Description PAX Health is a leading mental and behavioral health organization dedicated to providing comprehensive and compassionate...  ...objective, non-treatment psychological and neuropsychological evaluations for medico-legal cases, including workers’ compensation,... 

    PAX

    New York, NY
    14 hours ago
  • $61 - $71 per hour

     ...Senior AI Agent Developer – Conversational AI, Python & GCP (Hybrid) Job Title: Senior AI Agent Developer – Conversational AI, Python & GCP Location...  ..., and intent classification . Experience building evaluation frameworks to measure conversational AI accuracy,... 
    Long term contract
    Full time
    Contract work
    Remote work
    Monday to Friday
    3 days per week

    Seneca Resources

    New York, NY
    4 days ago
  • $50 - $60 per hour

     ...legal professional to help train AI models. You will measure the...  ...of these AI chatbots, evaluate their logic, and solve problems...  ...immigration law, real estate law, health law, entertainment law, human...  ...will handle any currency conversions from USD. Only applicants in... 
    Hourly pay
    Contract work
    For contractors

    DataAnnotation

    New York, NY
    3 days ago
  •  ...creative and technical talent with leading AI research labs. Headquartered in San...  ...time hours Role Responsibilities Evaluate AI answers to neurology questions for...  ...your resume. Complete a 25-minute conversational interview on your background, experience... 
    Contract work
    Part time
    Summer work
    Immediate start
    Remote work
    Flexible hours

    Mercor

    New York, NY
    27 days ago
  • $165k - $175k

     ...insurance directories to search engines, AI platforms, and more. Healthcare still lacks...  ...investigations into metric changes and evaluate plausible drivers. Partner with Product,...  ...Zocdoc is the country’s leading digital health marketplace that helps patients easily find... 
    Remote work

    Zocdoc

    New York, NY
    3 days ago
  • $77k - $90k

     ...and customer focus to the business.   Day Shift - 7.5 Hours (United States of America) The Data Scientist for the Richmond Health Network is the data custodian responsible for cleaning and analyzing data from various sources for the development, implementation... 
    Work experience placement
    Day shift

    Richmond University Medical Center

    New York, NY
    14 hours ago
  • $150k - $180k

     ...Data Scientist Boston or NYC Layer Health was founded in 2023 by leading machine learning...  ...Medical School. We are building an AI layer that can accurately and scalably synthesize...  ...for data manipulation, treatment, and evaluation, with the ability to wrangle large,... 
    Work at office
    2 days per week
    3 days per week

    Layer Health

    New York, NY
    19 hours ago
  •  ...Data Scientist to strengthen our organization-wide monitoring, evaluation, and learning (MEL) systems during a period of significant growth...  ..., and add to the global evidence-base on toxic pollution and health. We are looking for someone who combines rigorous analytical... 
    Full time
    Work at office
    Remote work
    Visa sponsorship
    Work visa
    Weekend work
    Afternoon shift
    Early shift
    Weekday work

    Pure Earth

    New York, NY
    3 days ago
  • $125.4k - $182.6k

     ...This team is responsible for delivering AI solutions to our business partners that will...  ...with Prompt Engineering, Chunking, LLM evaluation techniques ~ Ability to multi-task, learn...  ..., personal accident and supplemental health insurance, reinsurance, and life insurance... 
    Full time
    Temporary work
    Local area

    Chubb

    New York, NY
    2 days ago
  • $175.5k - $180k

    Tech & AI Senior Data Scientist Job ID: 111258 You will work at the intersection of healthcare and AI, building and deploying advanced...  ...concept; they are production systems used by some of the largest health systems in the country. That combination of technical rigor,... 
    Hourly pay
    Apprenticeship
    Work at office
    Easy work

    McKinsey & Company

    New York, NY
    14 hours ago
  • $72.39k - $109.48k

     ...Media, Digital business transformation with Publicis. Sapient, and Health & Wellbeing with Publicis Health. Since 2014 with the...  ...personalization through creative, media and production through our AI platform and persistent, durable identity-linked activation... 
    Temporary work
    Freelance
    Flexible hours
    Shift work

    Publicis Groupe Holdings B.V

    New York, NY
    4 days ago
  • $225k - $250k

     ...and benefits packages. About Adonis Adonis is the leading AI orchestration platform for healthcare revenue cycle management (RCM...  ...mission to restore sanity to RCM and equip provider groups and health systems with AI-driven revenue cycle operations at scale. Compensation... 
    Work at office

    Adonis Technology, Inc

    New York, NY
    14 hours ago
  • $163.94k - $215.18k

     ...a Senior Data Scientist to join our Data Team.Oscar is the first health insurance company built around a full stack technology platform and...  ...paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial intelligence... 
    Full time
    Work experience placement
    Work at office
    Flexible hours

    Oscar Health Insurance

    New York, NY
    2 days ago
  • $163.94k - $215.18k

     ...Senior Data Scientist, Lucie to join our Data Team.Oscar is the first health insurance company built around a full stack technology platform...  ...paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial... 
    Full time
    Temporary work
    Work experience placement
    Work at office
    Flexible hours

    Oscar Health Insurance

    New York, NY
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Health AI Conversation Evaluator - AI Trainer. Be the first to apply!