Practicing Clinician for AI Model Evaluation

Role Overview

Apply your clinical judgment to improve how frontier AI models reason about real-world medicine. You will work closely with an AI research and program management team, setting rigorous standards for clinical knowledge tasks, model outputs, and evaluation benchmarks. This role is designed for a practicing clinician with deep specialty expertise who can distinguish sound clinical reasoning from plausible but unsafe or off-guideline answers.

Key Responsibilities

• Review clinical knowledge tasks and model outputs for missing behaviors, weak reasoning, unsafe recommendations, guideline misalignment, and responses that would not withstand clinical scrutiny.
• Create detailed instruction specifications, gold-standard clinical solutions, and new tasks that reflect real clinical practice.
• Design challenging clinical benchmarks and evaluation sets to measure model improvement.
• Partner with researchers to develop medicine-specific capabilities and tools.
• Calibrate standards with researchers and adjacent-domain specialists, translating nuanced clinical judgment into clear, teachable evaluation criteria.

Qualifications

• MD or DO from an accredited medical school and completed residency in a recognized specialty.
• At least 4 years of post-residency clinical practice. Residency and fellowship training do not count toward this requirement.
• Active, unrestricted license to practice medicine in at least one U.S. state and board certification in your specialty.
• Deep expertise in a defined clinical specialty, such as internal medicine, oncology, radiology, emergency medicine, surgery, psychiatry, or another medical subspecialty.
• Senior-level clinical progression, such as Attending Physician, Medical Director, Division Chief, Associate or full Professor, or Chief Medical Officer, with meaningful ownership of clinical decisions.
• Hands-on professional use of large language models and the ability to assess rigorous reasoning versus convincing but incorrect answers.
• Excellent written communication skills and the ability to provide precise, structured feedback.
• Experience in utilization management, clinical informatics, or medical affairs is a plus.

Work Terms

• Full-time W-2 employment, embedded with a client research team and working within the client''s tools using client-issued accounts and equipment.
• Hybrid role based in the Bay Area, California. You must live in the Bay Area and be available to work on-site multiple days per week when required.
• This is not a remote role. Candidates outside the Bay Area must relocate at their own expense before starting. Relocation assistance is not available.
• Reliable availability for 40 hours per week is required.

Compensation

• Hourly pay of $70 to $110.

Application Process

Candidates may discover this opportunity through a job platform. Employment, onboarding, payroll, benefits, and compliance are managed by the employer of record, while the selected candidate works directly with the client team.

Equal Opportunity and Accommodations

Qualified applicants are considered without discrimination based on legally protected characteristics. Reasonable accommodations are available for qualified individuals with disabilities and disabled veterans throughout the application process.