Clinical Expert AI System Evaluator
$150 per hourFull-time
Mercor
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include Benchmark General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey .
Position: General Clinician (MD/DO) - HLS Expert Pool
Type: Contract
Compensation: $150/hour
Location: Remote
Role Responsibilities
- Develop grading criteria by breaking clinical questions into discrete checkable criteria for consistent model response grading.
- Evaluate clinical dialogues for accuracy safety completeness and appropriate escalation advice.
- Annotate clinical reasoning by recording case work-throughs including differentials considered and rejected.
- Review outputs to flag hallucinated findings dangerous omissions and clinically unsafe advice.
- Author guidelines to define edge cases and standards of care ensuring annotation consistency.
- Write difficult-case questions to probe current model reasoning limits.
Qualifications
Must-Have
- MD or DO with a completed residency in any specialty
- Active unrestricted medical license in your country of practice
- 3 years post-residency clinical experience
- Comfort writing structured clinical rationale
- Written and spoken English fluency
- Minimum 20 hours per week
Preferred
- Board certification in your specialty
- U.S. licensure and familiarity with U.S. standards of care
- Background in primary care internal medicine emergency medicine or hospitalist roles
- Experience in clinical annotation AI evaluation or medical education
- Grading-criteria design or clinical guideline development experience
- Published research or sustained technical writing (please link a sample)
Application Process (Takes 2030 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information please check:
- For any help or support reach out to:
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the Clinical Expert AI System Evaluator in San Francisco, CA vacancy
$1,350 per unit
...Bring your expertise to the problems today’s most advanced AI agents still struggle to solve. We’re looking for experienced systems engineering or hardware engineering professionals to develop challenging tasks for an AI benchmark. You’ll draw on problems from your own...SuggestedBraintrust
San Francisco, CA3 days ago$110 per hour
...Senior Clinical Medicine Domain Expert Join a leading AI lab's cutting-edge GenAI team to be at the core of the AI revolution, where your expertise fuels... ...depth: Design challenging clinical tasks and evaluation sets, and help build medicine-specific skills and tools...SuggestedFull timeContract workPart timeFreelanceLive inRelocationRelocation package$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Summers , and Jack Dorsey . Position: Healthcare / clinical Evaluator Type: Contract Compensation: $80–$120/hour Location...SuggestedContract workSummer workWork at officeRemote work$187.5k - $205k
Job TitleStructural Heart Clinical Adoption Expert - CV Ultrasound (West Zone)Job DescriptionStructural Heart Clinical Adoption Expert - CV Ultrasound... ...onsite customer training, case study support days, remote system training, software changes, upgrades, new technologies, as...SuggestedLocal areaImmediate startRemote workWork visaRelocation packagePhilips
San Francisco, CA3 days ago$100 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...: Software & Firmware Engineering Experts Type: Contract Compensation:... ...tasks involving real physical systems. Review and evaluate embedded firmware , FPGA/RTL designs...SuggestedContract workSummer workRemote workMercor
San Francisco, CA2 days ago- ...UCSF Health is seeking a Pediatric Clinical Staff Pharmacist II to deliver pharmaceutical care across UCSF Health System inpatient settings. You will review patient medication histories, ensure safe drug use, and collaborate with physicians, nurses, and other pharmacists...
UCSF Health, AG
San Francisco, CA2 days ago $70 - $110 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ..., and Jack Dorsey . Position: Clinical Medicine Domain Expert Type: Contract Compensation:... .../week Role Responsibilities Evaluate the quality of clinical knowledge work...Hourly payWeekly payContract workFor contractorsSummer work$45 - $50 per hour
...advanced) Written reporting (advanced) LLM evaluation (intermediate) Contract Type W2... ...role involves meticulously reviewing AI-generated data narratives to ensure accuracy... ...errors or misinterpretations that automated systems might miss. Your feedback will be crucial...Hourly payContract workPart timeRemote workAkraya Inc.
San Francisco, CA21 hours ago$90 per hour
...What You'll Do Review de-identified clinical cases and pharmaceutical product images. Develop scoring rubrics based on standard clinical practices. Provide written expert evaluations to serve as reference benchmarks. Assess the accuracy of various medical data interpretations...Contract workFor contractorsRemote workTerac
San Francisco, CA17 days ago$60 - $70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...$60–$70/hour Location: Remote Role Responsibilities Evaluate AI-generated responses for safety, factual accuracy, policy compliance...Contract workSummer workRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Government / public administration Evaluator Type: Contract Compensation: $... .... Collaborate with subject matter experts to ensure consistency and quality. Work...Contract workSummer workWork at officeRemote work$90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...genuinely correct replications from those that merely look correct. Evaluate responsive behavior and semantic quality, ensuring proper use of...Contract workSummer workLocal areaRemote work$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Software Engineer Expert Type: Contract Compensation: $... ...Location: Remote Role Responsibilities Evaluate AI-generated artifacts for usability in...Contract workSummer workWork at officeRemote work$20 - $22 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Jack Dorsey . Position: AI Safety Experts — English & Odia Type: Contract... ...failures, classify vulnerabilities, and flag systemic risks. Apply structure using taxonomies...Contract workSummer workRemote workMercor
San Francisco, CA16 days ago$20 - $22 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Jack Dorsey . Position: AI Safety Experts — English & Punjabi Type: Contract... ...classifying vulnerabilities, and flagging systemic risks. Apply structure by following taxonomies...Contract workSummer workRemote workMercor
San Francisco, CA13 days ago$80 - $150 per hour
...technical talent with leading AI research labs.... ...Adolescent Mental Health Clinical Advisor (AI Safety Benchmark... ...clinician-informed evaluation rubrics for AI chatbot... ...conversations and provide expert judgment to calibrate automated evaluation systems. Advise on best...Contract workTraineeshipSummer workRemote work$90 - $150 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...research teams to enhance infrastructure for machine learning systems . Qualifications Must-Have ~4+ years professional experience...Contract workSummer workRemote workMercor
San Francisco, CA19 days ago$93 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...resolution of claim denials. Evaluate AI-generated appeal letters, denial... ..., and denial category to identify systemic root causes. Develop and manage clinical and technical appeal strategies...Contract workSummer workRemote work$120 per hour
Prolific, located in San Francisco, is looking for Licensed Pharmacists to train and evaluate AI models. You will guide these systems by reviewing AI-generated pharmaceutical scenarios and providing your expertise. With competitive pay rates of up to $120 USD per hour...Remote jobHourly payWork from homeFlexible hours$180 per hour
...technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Clinical Law Professor / Clinic Director... ...Role Responsibilities Evaluate AI-generated legal responses... ...structured feedback to improve AI systems . Apply experience supervising...Contract workSummer workRemote work$170 - $190 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Physician (MD) - Outpatient Clinical Documentation & AI Evaluation Type: Contract Compensation... ...teams can act on to improve AI systems . Contribute expert input on annotation guidelines,...Contract workWork experience placementSummer workRemote work$70 - $110 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...+ hours/week Role Responsibilities Evaluate AI-generated operational plans , staff... ...performance improvement decisions. Provide expert feedback. Work independently and asynchronously...Hourly payContract workSummer workImmediate startRemote work$90 per hour
...technical talent with leading AI research labs. Headquartered in... ...with researchers and fellow experts to continually improve the benchmark... ...-engineering, security, or AI-evaluation roles. ~ Ability to identify... ...modes in LLMs or ML systems . ~ Proficiency in Python...Contract workSummer workRemote workMercor
San Francisco, CA2 days ago- ...re building a research cohort of senior domain experts across seven specialist tracks — software/systems engineering, formal methods and computational... ...hardware design, and creative technology — to help evaluate how frontier AI models reason about hard, real-world expert-...Temporary work
AfterQuery
San Francisco, CA25 days ago $90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...thresholds, and create prompts for model evaluation. Evaluate model designs using an agentic... ...Have Deep applied expertise in control systems, analog or RF circuits, power electronics...Contract workSummer workRemote workMercor
San Francisco, CA2 days ago$150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Investment Banking Expert Type: Contract Compensation: $... ...~ Prior experience with AI training , evaluation, or human-data projects. Application...Contract workSummer workRemote workMercor
San Francisco, CA2 days ago- ...transformation company, is seeking an Ads Quality Rater to review internet advertisements in English and Italian. You’ll help train AI by rating ads based on keywords, working remotely from Germany as an independent contractor. Set your schedule with a minimum of 5...For contractorsImmediate startRemote workWork from homeFlexible hours
Remote
San Francisco, CA21 hours ago $105 per hour
...technical talent with leading AI research labs. Headquartered in... ...workflows for medical and clinical services across multiple payer... ...or payer-specific criteria. Evaluate AI-generated prior authorisation... ...with authorisation management systems and EHR platforms ( Epic ,...Contract workSummer workWork at officeRemote work$45 - $70 per hour
...Social Science Help a leading AI lab ensure its models... ...is seeking behavioral health experts to help evaluate and improve how its AI models... ...moralizing, or overstepping into clinical or directive advice, and... ..., relationships, belief systems and worldviews, and to explain...Contract workPart timeWeekday workMercor
San Francisco, CAa month ago$40 - $90 per hour
...We are looking for PhD-qualified biology experts to contribute their scientific expertise to a short-term annotation project supporting a leading AI lab. Expert annotation helps AI researchers develop and evaluate how models handle specialized scientific information....Hourly payTemporary workFor contractors10 hours per weekBraintrust
San Francisco, CA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Clinical Expert AI System Evaluator. Be the first to apply!
Related searches
- subject matter expert San Francisco, CA
- fulfillment expert San Francisco, CA
- technology expert San Francisco, CA
- guest service support expert San Francisco, CA
- ai evaluator San Francisco, CA
- education evaluator San Francisco, CA
- evaluator San Francisco, CA
- work from home web search evaluator San Francisco, CA
- social media evaluator San Francisco, CA
- quality evaluator San Francisco, CA