AI Evaluation Engineer

Our client, a global pharmaceutical, biotechnology, and medical technology company, is actively looking for an experienced AI Evaluation Engineer to join their team! This role is 100% remote, however they are only considering candidates currently on the East Coast. ***This is a 6 month initial contract with the opportunity to extend based on business needs*** This Engineer will build and run the evaluations for a GenAI regulatory science co-pilot used by regulatory submission teams. The system works like a virtual Health Authority reviewer. It extracts and assesses regulatory risks, maps submission content to relevant guidance and precedent, checks that key messages are supported by the underlying clinical data, and anticipates likely Health Authority questions. Because its outputs directly inform regulatory submission strategy, they must be accurate, grounded in cited sources, and traceable. You will work with regulatory affairs subject-matter experts (SMEs) and the product engineering team. Responsibilities Build gold sets with SMEs: annotate regulatory and clinical documents with the expected outputs, write annotation guidelines, run adjudication, and measure inter-rater agreement. Evaluate extraction: score risk extraction for precision, recall, and completeness, and compare successive engine versions against a baseline. Evaluate retrieval and ranking: measure retrieval recall, ranking quality, and question coverage when the system retrieves regulatory guidance, precedent, and prior Health Authority interactions. Evaluate grounding and traceability: build code-based checks that every finding traces back to a valid source and citation, and detect fabricated guidance, fabricated precedent, and unsupported regulatory claims. Evaluate reasoning with LLM judges: design and validate LLM-as-judge rubrics for regulatory relevance, evidence sufficiency, risk assessment, and explanation quality, calibrated against SME judgment. Evaluate each stage of the multi-stage pipeline (evidence structuring, question coverage, retrieval, ranking, mapping, grounding), plus an end-to-end review with SMEs. Turn failure analysis into clear readouts for the product team, and package the evaluations as regression suites that run in CI/CD. Required Skills & Experience 5+ years of hands-on work in AI/ML evaluation, NLP, or applied data science, including recent delivery on LLM, RAG, or agent systems. Experience building evaluation frameworks and gold datasets for information extraction and retrieval/ranking systems (please be ready to share examples). Strong grounding in evaluation metrics and statistics: precision/recall/F1, View email address on click.appcast.io, MRR, nDCG, inter-rater reliability (e.g., Cohen’s kappa or Krippendorff’s alpha), and confidence intervals for small evaluation sets. Experience designing and validating LLM-as-judge systems against human expert ratings. Experience running annotation projects with domain experts, including guidelines, adjudication, and quality control. Strong Python, with experience using evaluation harnesses (e.g., Ragas, DeepEval, promptfoo, Inspect, or in-house equivalents), LLM APIs, and Git/CI/CD. Preferred Experience Regulatory affairs, regulatory intelligence, or submissions experience: FDA/EMA guidance, CTD structure (e.g., Modules 2.5 and 2.7), Health Authority questions, and labeling. Experience evaluating AI in regulated (GxP) environments. Familiarity with clinical trial outputs (CSRs, TLFs, statistical outputs) and drug development. Publications or open-source work in AI evaluation, information retrieval, or biomedical NLP.