Clinical AI Safety Evaluator

Overview Join a select team of healthcare experts focused on safeguarding and refining AI systems. This role centers on evaluating clinical precision, safety standards, and overall response quality in generative healthcare AI applications. Key responsibilities involve performing structured evaluations using established clinical frameworks and conducting adversarial stress-testing (Red-Teaming) to identify potential vulnerabilities and safety concerns before public release. We are looking for experts operating at the intersection of clinical practice, AI expertise, and health informatics safety. Candidates should be practicing clinicians passionate about health technology, possessing the analytical rigor needed to push model limits and protect patient safety. This is a vendor role with Highspring for a client's team, a Fortune 100 company. This position is remote until further notice. Responsibilities Structured Grading Reviewing prompt-response pairs and scoring them across dimensions like clinical accuracy, safety, and various health risk domains. Dynamic Red-Teaming Sprints Participating in "sweeps" where they dynamically converse with the model to probe specific health risks (e.g., attempting to get the model to prescribe medication or diagnose a complex condition). Providing Clinical Rationale Writing clear, concise, and scientifically backed justifications for why a model response was graded as unsafe or inaccurate. Minimum Qualifications Clinical Credentials MD, MBBS, or local equivalent from an accredited institution. Candidates must be Board-Certified in one of the following priority specialties: Psychiatry / Clinical Psychology (Mental health, self-harm risks) Primary Care / Family Medicine / Internal Medicine (General health queries) Active, unrestricted clinical licensure in good standing in their practicing geography (US). Direct, current practice in patient care. Understanding how patients, carers and families ask questions in the real world is critical for realistic testing. AI Literacy Active experience using generative AI platforms (e.g., Gemini, ChatGPT, Claude) for clinical workflows, research, or everyday queries. Red Teaming Clinician Rater Understanding of the inherent limitations of LLMs, including hallucinations, sycophancy (the model agreeing with the user even when wrong), bias, and the risks of generating unsafe clinical advice. Ability to write clear, structured, and complex prompts to interact with models dynamically Evaluations & Red-Teaming Ability to adopt different user personas (e.g., a worried parent, a patient seeking to bypass safety rules, an anxious patient misinterpreting symptoms) to stress-test the model. Experience working with complex, multi-dimensional rubrics (such as Likert scales). Comfort with aligning their grading to strict "Anchor Examples" to minimize subjective bias. Preferred Qualifications An additional advanced degree in a relevant field (e.g., Master of Public Health (MPH), MS/PhD in Medical Informatics, Computer Science, or Behavioral Science). Fluency in multiple languages and deep understanding of cultural health beliefs and regional clinical guidelines (vital for global scaling). Previous experience designing, implementing, or participating in AI safety evaluations, Trust & Safety workflows, or RLHF (Reinforcement Learning from Human Feedback) for medical models. Debarment and Exclusion. Contractor represents and warrants that it has not been and will not use in any capacity any individual or entity to perform any Services if such individual or entity has been: (i) debarred by the Food and Drug Administration("FDA") pursuant to 21 U.S.C. § 335a; (ii) excluded from participation in any state or federal healthcare program, as defined in 42 U.S.C.§1320a-7b(f); (iii) debarred from participation in federal procurement and/or non-procurement programs; (iv) on FDA's list of investigators with disqualification proceedings, or otherwise deemed ineligible to receive investigational products; (v) charged, or convicted of a criminal offense, or named or found liable in an action under any applicable law pertaining to the regulation of a health related product or the provision of health related services; (vi) the subject of a final or pending adverse action, as defined in 42 U.S.C. § 1320a-7a(e) and 42 U.S.C. § 1320a-7a(g); (vii) an employer, employee, owner, shareholder, member, subsidiary, affiliate, or partner of a person or entity that is subject to any of the foregoing clauses (i)-(vi), inclusive; (viii) contracted with any employee, contractor, agent, vendor or vendor's affiliate knowing that the contracting party is subject to any of the foregoing clauses (i)-(vi), inclusive; or (ix) otherwise subject to any restrictions or sanctions by the FDA or any other state or federal governmental or regulatory authority or professional body with respect to the performance of scientific or clinical investigations or related to the regulation of a health related product or the provision of health related services. If during the SOW Term, Contractor or any Personnel: (1) becomes subject to any of the foregoing clauses (i)-(ix), inclusive, or (2) receives notice or threat of an action with respect to any of the foregoing clauses (i)-(ix), inclusive, Contractor will notify Our Client immediately and will immediately remove such person(s) from participation in performing any Services Schedule Requirements Monday - Friday, flexible hours but requires daily check in Part time, 5 hours per week Equal Opportunity Notice Highspring LLC (d/b/a Vaco by Highspring) and its parents, affiliates, and subsidiaries("we," "our," or "Vaco by Highspring") are committed to the full inclusion of all qualified individuals and does not discriminate against any employee or applicant for employment because of race (including but not limited to traits historically associated with race such as hair texture and hair style), color, sex (includes pregnancy or related conditions), religion or creed, national origin, citizenship, age, disability, status as a veteran, union membership, ethnicity, gender, gender identity, gender expression, sexual orientation, marital status, political affiliation, or any other protected characteristics as required by applicable law. The company is also committed to ensuring that persons who need them are provided with reasonable accommodations; if an accommodation is needed to participate in the job application or interview process, please contact View email address on click.appcast.io . Vaco by Highspring also wants all applicants to know their rights that workplace discrimination is illegal. Representation Notice By submitting to this position, you agree that you will be giving Vaco by Highspring the exclusive right to present your as a candidate for the foregoing employment opportunity. Additionally, you agree to be included in our talent pool for future hiring for similarly qualified positions. You further agree that you have represented information about yourself accurately and have not affirmatively misrepresented your qualifications. Lastly, you agree to maintain as confidential, to the fullest extent permitted by law, any information you learn from Vaco by Highspring about the position and you will limit disclosure of information about the position only to the extent necessary to perform any obligations in furtherance of your application. In exchange, Vaco by Highspring agrees to exercise reasonable efforts to represent you through all solicitation, job screening and resume dispersal. Pay Transparency Notice Determining compensation for this role (and others) at Vaco by Highspring depends upon a wide array of factors including but not limited to:

• the individual's skill sets, experience and training;
• licensure and certification requirements;
• office location and other geographic considerations; and
• other business and organizational needs.

With that said, as required by local law, Vaco by Highspring believes that the following salary range referenced above reasonably estimates the base compensation for an individual hired into this position in geographies that require salary range disclosure. The individual may also be eligible for discretionary bonuses and/or participation in medical, dental, and vision benefits as well as the company's retirement plan (or similar retirement benefits). #J-18808-Ljbffr Vaco by Highspring