Health AI Conversation Evaluator - AI Trainer
$20 - $160 per hourFull-time
Mercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote
Role Responsibilities
- Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
- Strong written English skills.
- Ability to explain judgments in two to four specific sentences.
- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.
Preferred
- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.
Important
- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on jobswipe.net
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercorVacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Health AI Conversation Evaluator - AI Trainer in New York, NY vacancy
$137.3k - $166k
...that when people achieve their health goals, they can achieve their... ...by more fully unlocking Conversational Analytics. You thrive in a startup... ...documentation, context, and evaluations to strengthen our LLMs’... ...materials Develop and maintain AI evaluations for conversational...SuggestedFull timeWork experience placementLocal areaRo
New York, NY20 days ago$100 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our... ...Summers , and Jack Dorsey . Position: Clinical Mental Health Expert — AI Conversation Evaluation Type: Contract Compensation: $100/hour...SuggestedContract workSummer workRemote work$144.2k - $288.4k
...We're building a world of health around every individual - shaping a more connected, convenient... ..., machine learning, and generative AI solutions that improve IT operations and... ...analytics, interactive reporting, and conversational "chat with your data" experiences that...SuggestedHourly payFull timeTemporary workWork experience placementLocal areaCVS Health
New York, NY18 hours ago- ...apply your expertise in the growing world of AI? We're looking for professionals... ...in: ✔️ Medical Coding & Billing ✔️ Health Information Management (HIM) ✔️ Utilization... ...What you'll be doing: • Evaluate and improve AI-generated healthcare content...SuggestedImmediate startRemote workFlexible hours
Green Key Resources
New York, NY16 hours ago $100 per hour
...A healthcare AI company seeks a Remote Registered Nurse – AI Trainer to provide clinical nursing expertise for training AI systems. Responsibilities include evaluating clinical data, translating nursing workflows into structured data, and guiding AI interactions. Candidates...SuggestedHourly payContract workRemote work10 hours per weekCrossing Hurdles
New York, NY4 days ago$175k - $250k
Applied AI EngineerAbout MillenniumMillennium is a global, diversified... ...systems are core to the health and growth of the business, including... ...solution. You own everything from that conversation through architecture, deployment, evaluation, and iteration, then turn what...Work at officeFlexible hoursMillennium Management
New York, NY18 hours ago$40 per hour
...Senior AI Engineer Doctronic | New York City ONSITE (Hybrid) | Reports to CEO... ...extracting structured data from patient conversations (HIPAA-compliant) QHIN integrations... ...interfaces, multilingual medical NLP, predictive health modeling, and infrastructure for the next...Doctronic
New York, NY2 days ago$145k - $165k
...than a thousand multinational employers, health plans, and health systems trust Carrot to... ...Interviews - 45 minutes Executive Leadership Conversation - 30 minutes Hiring Manager Wrap-Up - 15... ...skills and experience. A Note on AI At Carrot, we believe AI can help us work...Temporary workWorldwideFlexible hoursShift work- ...dependencies, and validation steps. Use AI to improve productivity in data analysis... ...Python . Experience supporting data imports, conversions, migrations, or similar data movement... ...serve. What we can do for you: Along with health and financial benefits, athenistas enjoy...Full timeTemporary workWork at officeRemote work
$65 - $75 per unit
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...chemical safety . Label them as benign, dual-use, or adversarial. Evaluate AI model responses against a defined policy standard. Judge...Contract workSummer workRemote workFlexible hoursMercor
New York, NY7 days ago$163.94k - $215.18k
...'re Oscar. We're hiring a Senior Data Scientist, AI Enablement to join our Data Team.Oscar is the first health insurance company built around a full stack technology... ..., reliability, performance, and cost.Establish evaluations and monitoring for analytical accuracy, generated...Full timeWork at officeFlexible hoursOscar Health Insurance
New York, NY18 hours ago$131.2k - $172.2k
...hiring a Data Scientist II to join our Data team.Oscar is the first health insurance company built around a full stack technology platform... ...paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial intelligence...Full timeWork experience placementWork at officeFlexible hoursOscar Health Insurance
New York, NY4 days ago$119.2k - $156.45k
...hiring a Data Scientist I to join our Data team.Oscar is the first health insurance company built around a full stack technology platform... ...and paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial...Full timeWork at officeFlexible hoursOscar Health Insurance
New York, NY18 hours ago$95k - $125k
...scientists who have helped to shape the course of medical history and enrich the lives of countless people. An integral part of NYU Langone Health, the Grossman School of Medicine at its core is committed to improving the human condition through medical education, scientific...Full timeInterim roleNYU Langone Medical Center
New York, NY2 days ago- Company Overview:Blink Health is the fastest growing healthcare technology company that builds... ...and build measurement frameworks that evaluate customer engagement, adoption, retention, and business impact.Design and build AI-powered analytical tools and agentic workflows...
Blink Health
New York, NY2 days ago $163.94k - $215.18k
...a Senior Data Scientist to join our Data Team.Oscar is the first health insurance company built around a full stack technology platform and... ...paid wellness time and reimbursements.Artificial Intelligence (AI): Our AI Guidelines outline the acceptable use of artificial intelligence...Full timeWork experience placementWork at officeFlexible hoursOscar Health Insurance
New York, NY4 days ago- ...Senior AI Agent Developer - Contract - HYBIRD NYC *** In Person Interview Required Position Overview We are seeking a seasoned... ..., development, testing, and deployment of enterprise-grade conversational AI agents and intelligent workflow automations. In this role...Contract workLocal area
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... .... Position: User/customer research and feedback synthesis Evaluator Type: Contract Compensation: $80–$120/hour Location...Contract workSummer workWork at officeRemote work$190k - $270k
...Job Description Job Description Why Charlie Health? Millions of people across the country are navigating mental health conditions... ...studies]), & align on consistent metric definitions (e.g. CAC, conversion, retention) Develop spend optimization & channel allocation frameworks...Full timeTemporary workWork at officeLocal areaCharlie Health Engineering, Product & Design
New York, NYa month ago- To enhance conversational experiences, the full-time Conversational AI Developer will manage chatbot implementations, drive architectural migrations, and design robust API integrations while working remotely. Key Responsibilities Drive the full lifecycle development of...Full timeRemote work
- ...providing the tools and resources they need to achieve financial health and work-life balance. DCWP also conducts research and... ...investigators and attorneys to identify the data necessary to evaluate compliance with NYC worker and consumer protection laws; Communicating...Full timeFlexible hoursNight shift
CONSUMER AND WORKER PROTECTION
New York, NY4 days ago $78k - $80k
...Data Science and Research Unit Division Family and Child Health Bureau Office of School Health Location 30-30 47th Avenue... ...the delivery of services to school-aged children necessary to evaluate student health and educational outcomes. About the Program...Contract workLive inWork at officeRemote workFlexible hoursSupportFinity
New York, NY4 days ago- ...Brigit is a holistic financial health company helping everyday Americans build a brighter financial future. Role overview As a Data Scientist on our team, you’ll be responsible for building, improving, and maintaining the key ML models that enable our services....Local areaFlexible hoursShift work
Apply
New York, NY2 days ago - ...providing the tools and resources they need to achieve financial health and work-life balance. DCWP also conducts research and... ...empowerment objectives. Apply research and analytic best practices to evaluate program success and division performance. Utilize internal...Full timeRemote workFlexible hours2 days per week
New York City Department of Consumer and Worker Protection
New York, NY4 days ago $175k - $250k
...to adapt, evolve and build lasting impact over time. Discover how transformative growth accelerates impact.Meet the TeamCore to the health and growth of our business, Millennium’s Information Technology organization supports a flexible, scalable technology environment...For contractorsFlexible hoursMillennium Management
New York, NY3 days ago$190k - $220k
...the small business economy. We handle the hard stuff — payroll, health insurance, 401(k)s, and HR — so owners can focus on their craft and... ..., and location. Learn more about ourTotal Rewards philosophy.AI is a fundamental part of how work gets done at Gusto. We expect all...Full timeWork at officeLocal area2 days per week3 days per weekGusto
New York, NY3 days ago- Company DescriptionWHO WE AREEquinox is a category of one, setting the global standard for luxury health, wellness, and hospitality. Our spaces are intentionally elevated, our standards uncompromising, and our expectations high. We attract people who take pride in excellence...Work at office
Equinox Holdings
New York, NY1 day ago $90k - $135.29k
...than one-off analysesSupport execution of research and clinical AI projects across the Department of SurgeryTrack project milestones... ...stakeholdersPrior experience working with healthcare data, including electronic health record (EHR) systems, clinical registries, or medical imaging...TraineeshipLocal areaImmediate startFlexible hoursMount Sinai Health System
New York, NY3 days ago$8 - $65 per hour
...eager to shape the future of AI? Large-scale language... ...with cutting-edge AI tools, evaluate translations and conversations between Korean and Japanese... ...benefits such as health insurance and PTO do not apply... ...Bilingual Specialist – AI Trainer Employment type: Contract...Hourly payContract workFor contractorsFreelanceRemote work- ...About Us Health is the most important thing in life, and the American healthcare system is completely broken - poor outcomes, high cost,... ...country’s largest dietitian-led metabolic health clinic. We’re an AI-native digital health system matching patients with 10,000+ Registered...Work at office2 days per week3 days per week
Nourish Inc
New York, NY1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Health AI Conversation Evaluator - AI Trainer. Be the first to apply!
Related searches
- ai evaluator New York, NY
- education evaluator New York, NY
- work from home web search evaluator New York, NY
- evaluator New York, NY
- program evaluator New York, NY
- clinical evaluator New York, NY
- social media evaluator New York, NY
- quality evaluator New York, NY
- ai trainer New York, NY
- life health New York, NY