Clinical Expert - AI System Evaluator

Job Description

Job Description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: General Clinician (MD/DO) - HLS Expert Pool
Type: Contract
Compensation: $150/hour
Location: Remote

Role Responsibilities

• Develop grading criteria by breaking clinical questions into discrete, checkable criteria for consistent model response grading.
• Evaluate clinical dialogues for accuracy, safety, completeness, and appropriate escalation advice.
• Annotate clinical reasoning by recording case work-throughs, including differentials considered and rejected.
• Review outputs to flag hallucinated findings, dangerous omissions, and clinically unsafe advice.
• Author guidelines to define edge cases and standards of care, ensuring annotation consistency.
• Write difficult-case questions to probe current model reasoning limits.

Qualifications

Must-Have

• MD or DO with a completed residency in any specialty
• Active, unrestricted medical license in your country of practice
• 3+ years post-residency clinical experience
• Comfort writing structured clinical rationale
• Written and spoken English fluency
• Minimum 20 hours per week

Preferred

• Board certification in your specialty
• U.S. licensure and familiarity with U.S. standards of care
• Background in primary care, internal medicine, emergency medicine, or hospitalist roles
• Experience in clinical annotation, AI evaluation, or medical education
• Grading-criteria design or clinical guideline development experience
• Published research or sustained technical writing (please link a sample)

Application Process (Takes 20–30 mins to complete)

• Upload resume
• AI interview based on your resume
• Submit form

Resources & Support

• For details about the interview process and platform information, please check:
• For any help or support, reach out to: View email address on ziprecruiter.com

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.