Health AI Conversation Evaluator - AI Trainer
$20 - $160 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote
Role Responsibilities
- Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
- Strong written English skills.
- Ability to explain judgments in two to four specific sentences.
- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.
Preferred
- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.
Important
- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercor$80 - $120 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Larry Summers , and Jack Dorsey . Position: Public health communications Evaluator Type: Contract Compensation: $80–$120/hour...SuggestedContract workSummer workWork at officeRemote work$137.3k - $166k
...that when people achieve their health goals, they can achieve their... ...by more fully unlocking Conversational Analytics. You thrive in a startup... ...documentation, context, and evaluations to strengthen our LLMs’... ...materials Develop and maintain AI evaluations for...SuggestedFull timeWork experience placementLocal area$100 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our... ...Summers , and Jack Dorsey . Position: Clinical Mental Health Expert — AI Conversation Evaluation Type: Contract Compensation: $100/hour...SuggestedContract workSummer workRemote work$117.6k - $161.7k
...solutions enabling the efficient conversion of structured and... ...or data requires an in-depth evaluation of variable factors.OverviewHumana... ...our mission of improving the health and well-being of our consumers... ...Genie)Leverage Databricks Genie (AI/BI capabilities) to enable...SuggestedFull timeTemporary workApprenticeshipWork at officeRemote workWork from homeHome office$50 - $190 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco,... ...: 20+ hours/week Role Responsibilities Evaluate AI systems on complex personal workflows, including personal health , travel , and activity planning ....SuggestedHourly payContract workFor contractorsSummer workRemote workTrial period$8 - $65 per hour
...Language Specialist - Freelance AI Trainer Project United States of... .... On a typical day, you will converse with the model on language scenarios... ...our prompt engineering and evaluation metrics. A master’s or PhD in... ...‑sponsored benefits such as health insurance and PTO do not...Hourly payContract workFor contractorsFreelanceImmediate startRemote work$198.72k - $260.82k
...Engineer to join our Data team.Oscar is the first health insurance company built around a full stack... ...across web and app, our member facing health AI agent (Oswell), tools used by Oscar teammates during member conversations, and intentional outbound engagement.The work...Full timeWork at officeFlexible hours$8 - $65 per hour
...to shape the future of AI? Large-scale language models... ..., diagnostics, public health, health systems... ...typical day, you will converse with the model in both... ...prompt engineering and evaluation metrics. A master's degree... ...Fluent in French) - AI Trainer Employment type: Contract...Hourly payContract workFor contractorsFreelanceRemote work- YO AI Labs is seeking experienced physicians to support an advanced healthcare AI project. This part-time contractor role involves applying clinical expertise to evaluate AI-generated medical content, analyze cases, and improve AI systems used in healthcare decision-making...Remote jobPart timeFor contractors
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Jack Dorsey . Position: Data analysis / quantitative readouts Evaluator Type: Contract Compensation: $80–$120/hour...Contract workSummer workWork at officeRemote work$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...and Jack Dorsey . Position: Biology / environmental science Evaluator Type: Contract Compensation: $80–$120/hour Location...Contract workSummer workWork at officeRemote work$40 - $125 per hour
...DataAnnotation is looking for an experienced Computational Chemist - AI Trainer to design chemistry problems that today's leading AI models... ...ask for any money from you. PayPal will handle any currency conversions from USD. Only applicants in Australia, Canada, Ireland, New...Hourly payFor contractors$8 - $65 per hour
...to shape the future of AI? Large-scale language models... ...typical day, you will converse with the model in... ...prompt engineering and evaluation metrics. A Master's degree... ...benefits such as health insurance and PTO do not... ...) Language Expert - AI Trainer Employment type: Contract...Hourly payContract workFor contractorsFreelanceRemote work$155k - $410k
...advanced data science methods to complex health payer challenges, using statistical... ...client needs- Leading executive-level client conversations across multiple healthcare data efforts-... ...closely with team members. We evaluate these factors thoughtfully to establish...Full timeH1b$8 - $65 per hour
...Production Specialist - Freelance AI Trainer Project United States of... .... On a typical day, you will converse with the model on real‑world... ...to our prompt engineering and evaluation metrics. A bachelor’s or master... ...‑sponsored benefits such as health insurance and PTO do not...Hourly payContract workFor contractorsFreelanceImmediate startRemote work$8 - $65 per hour
...Japanese Bilingual Specialist - Freelance AI Trainer Project World Wide - Remote Are you... ...work with cutting‑edge AI tools, evaluate translations and conversations between Korean and Japanese, and... ...company‑sponsored benefits such as health insurance and PTO do not apply....Hourly payContract workFor contractorsFreelanceRemote workWorldwide- ...Description PAX Health is a leading mental and behavioral health organization dedicated to providing comprehensive and compassionate... ...objective, non-treatment psychological and neuropsychological evaluations for medico-legal cases, including workers’ compensation,...
$61 - $71 per hour
...Senior AI Agent Developer – Conversational AI, Python & GCP (Hybrid) Job Title: Senior AI Agent Developer – Conversational AI, Python & GCP Location... ..., and intent classification . Experience building evaluation frameworks to measure conversational AI accuracy,...Long term contractFull timeContract workRemote workMonday to Friday3 days per week$50 - $60 per hour
...legal professional to help train AI models. You will measure the... ...of these AI chatbots, evaluate their logic, and solve problems... ...immigration law, real estate law, health law, entertainment law, human... ...will handle any currency conversions from USD. Only applicants in...Hourly payContract workFor contractors- ...creative and technical talent with leading AI research labs. Headquartered in San... ...time hours Role Responsibilities Evaluate AI answers to neurology questions for... ...your resume. Complete a 25-minute conversational interview on your background, experience...Contract workPart timeSummer workImmediate startRemote workFlexible hours
$165k - $175k
...insurance directories to search engines, AI platforms, and more. Healthcare still lacks... ...investigations into metric changes and evaluate plausible drivers. Partner with Product,... ...Zocdoc is the country’s leading digital health marketplace that helps patients easily find...Remote work$77k - $90k
...and customer focus to the business. Day Shift - 7.5 Hours (United States of America) The Data Scientist for the Richmond Health Network is the data custodian responsible for cleaning and analyzing data from various sources for the development, implementation...Work experience placementDay shift$150k - $180k
...Data Scientist Boston or NYC Layer Health was founded in 2023 by leading machine learning... ...Medical School. We are building an AI layer that can accurately and scalably synthesize... ...for data manipulation, treatment, and evaluation, with the ability to wrangle large,...Work at office2 days per week3 days per week- ...Data Scientist to strengthen our organization-wide monitoring, evaluation, and learning (MEL) systems during a period of significant growth... ..., and add to the global evidence-base on toxic pollution and health. We are looking for someone who combines rigorous analytical...Full timeWork at officeRemote workVisa sponsorshipWork visaWeekend workAfternoon shiftEarly shiftWeekday work
$125.4k - $182.6k
...This team is responsible for delivering AI solutions to our business partners that will... ...with Prompt Engineering, Chunking, LLM evaluation techniques ~ Ability to multi-task, learn... ..., personal accident and supplemental health insurance, reinsurance, and life insurance...Full timeTemporary workLocal area$175.5k - $180k
Tech & AI Senior Data Scientist Job ID: 111258 You will work at the intersection of healthcare and AI, building and deploying advanced... ...concept; they are production systems used by some of the largest health systems in the country. That combination of technical rigor,...Hourly payApprenticeshipWork at officeEasy work$72.39k - $109.48k
...Media, Digital business transformation with Publicis. Sapient, and Health & Wellbeing with Publicis Health. Since 2014 with the... ...personalization through creative, media and production through our AI platform and persistent, durable identity-linked activation...Temporary workFreelanceFlexible hoursShift work$225k - $250k
...and benefits packages. About Adonis Adonis is the leading AI orchestration platform for healthcare revenue cycle management (RCM... ...mission to restore sanity to RCM and equip provider groups and health systems with AI-driven revenue cycle operations at scale. Compensation...Work at office$163.94k - $215.18k
...a Senior Data Scientist to join our Data Team.Oscar is the first health insurance company built around a full stack technology platform and... ...paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial intelligence...Full timeWork experience placementWork at officeFlexible hours$163.94k - $215.18k
...Senior Data Scientist, Lucie to join our Data Team.Oscar is the first health insurance company built around a full stack technology platform... ...paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial...Full timeTemporary workWork experience placementWork at officeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Health AI Conversation Evaluator - AI Trainer. Be the first to apply!
- education evaluator New York, NY
- work from home web search evaluator New York, NY
- clinical evaluator New York, NY
- evaluator New York, NY
- work from home social media evaluator New York, NY
- ai evaluator New York, NY
- transcript evaluator New York, NY
- program evaluator New York, NY
- quality evaluator New York, NY
- social media evaluator New York, NY





