Associate Director - RWD Data Engineering
$136.5k - $200.2kEli Lilly & Co
At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.
Job Description
Purpose:
The Associate Director, RWD Data Engineering is a hands-on technical role that builds and delivers the engineering vision for Lilly’s real-world data (RWD) infrastructure set by the Senior Director – RWD Engineering & Architecture. This individual designs, codes, tests, and deploys the scalable, cloud-native pipelines, data products, and analytic applications that allow HEOR, SDIA, Statisticians, Medical, and Clinical teams to generate evidence faster, more reproducibly, and at greater scientific depth than is possible through traditional vendor engagements.
This role requires deep, practical knowledge of secondary healthcare data, with a particular focus on administrative claims and electronic health records (EHR), and of the Databricks Lakehouse platform used to engineer it. The Associate Director turns raw RWD into validated, analysis-ready data products; automates data ingestion, quality checks, and product creation; and builds AI-enabled tools and analytic applications that put those data products directly in the hands of scientists. This individual applies AI coding assistants and frontier AI models as an everyday part of software development, and is accountable for the correctness, performance, and reproducibility of the code they deliver. The ideal candidate is an expert builder who pairs strong engineering fundamentals with rigor in data quality and a passion for leveraging data to drive business outcomes.
This position reports to the Senior Director – Real World Data Engineering & Architecture within HEOR Central and works in close partnership with CIA, HEOR, SDIA, Statisticians, Medical, and Clinical teams.
Responsibilities:
- undefined
- Design, develop, and deploy cloud-native data products and high-throughput data pipelines on Databricks that transform raw real-world data into scalable, reliable, analysis-ready assets supporting analytics, reporting, and evidence generation.
- Leverage AI coding assistants and agentic development tools (e.g., Claude Code, GitHub Copilot, Cursor, Databricks Assistant) to write, refactor, test, and document Python, SQL, and R code, applying rigorous human review, testing, and validation to every AI-generated contribution.
- Deliver assigned RWD data products through the full development lifecycle, from technical requirements and prototyping through production deployment and optimization, ensuring scalability, reliability, performance, and reproducibility within the established Databricks Lakehouse environment.
- Build, optimize, and maintain ETL/ELT ingestion and transformation pipelines for large-scale secondary RWD, with a particular focus on administrative claims and complex EHR data, handling data volumes ranging from tens of millions to billions of records.
- Build and maintain data assets within the established lakehouse architecture (e.g., medallion bronze/silver/gold patterns) using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows to produce versioned, partitioned, and audit-ready data assets.
- Write and maintain reusable, version-controlled transformation logic incorporating healthcare coding and terminology standards (e.g., ICD-10/ICD-9, NDC, RxNorm, SNOMED, CPT/HCPCS, LOINC) to produce domain-level datasets such as demographics, diagnoses, treatments, procedures, encounters, and labs.
- Optimize SQL and distributed processing workloads (e.g., Databricks SQL and Spark jobs) for performance and compute cost across very large datasets, reading query plans and applying partitioning, liquid clustering/Z-ordering, predicate pushdown, caching, and other optimization strategies appropriate to analytical workloads.
- Translate analytic and research requirements into reproducible data extraction and transformation logic, supporting cohort construction, temporal logic, and consistent reuse of RWD across teams.
- Design, implement, and run data quality and validation frameworks for every data product, including automated checks for completeness, conformance, plausibility, and referential integrity; reconciliation of record counts and distributions against source; regression testing between data refreshes; and monitoring and alerting that surface issues before they reach analytical teams.
- Apply the team’s version control and CI/CD standards in daily work, including Git branching, pull requests, peer code review, automated unit and data tests, and deployment via Databricks Asset Bundles, ensuring all code is version-controlled, tested, and deployable in a reproducible manner.
- Build and deploy interactive analytic applications (e.g., Dash, R Shiny, Streamlit, Databricks Apps) that allow HEOR scientists, statisticians, and medical teams to explore data products, review cohort counts and data quality results, and self-serve common analyses.
- Build AI-enabled solutions that operate directly on governed data products using frontier AI models (e.g., Anthropic Claude, OpenAI GPT, Google Gemini) and Databricks Model Serving, such as natural-language query of data products, AI agents that execute data workflows, and retrieval over data dictionaries, code lists, and documentation; evaluate these solutions for accuracy and ensure they meet Lilly’s privacy, security, and responsible-AI requirements.
- Collaborate closely with data scientists, statisticians, and other technical partners to understand technical requirements, and contribute to documentation of RWD engineering standards, transformation templates, code list repositories, and pipeline performance guidelines.
- Serve as the technical point of contact for assigned data sources and data products, advising collaborators on appropriate use, structural limitations, join strategies, and performance considerations; contribute to source-specific training materials for HEOR scientists, SDIA, and statisticians.
- Contribute to an inclusive culture where producing and maintaining high-quality data is a core discipline.
Technical Skills:
Core Engineering Skills
· Expert SQL optimization on Databricks SQL and Spark SQL (with Redshift, Snowflake, or BigQuery a plus): complex joins, window functions, query plan analysis, query tuning, and compute cost management on billion-row tables.
· Python fluency: pandas, PySpark, Polars for large-scale data manipulation; workflow orchestration with Apache Airflow, Prefect, or Dagster for production pipeline scheduling and monitoring — including Databricks Workflows for orchestrating multi-task jobs within the Lakehouse.
· Distributed computing: Apache Spark (PySpark), Dask, or Ray — ability to write, tune, and debug distributed jobs processing multi-terabyte datasets across partitioned cloud storage, including Databricks clusters with auto-scaling and spot instance optimization.
· Databricks expertise: hands-on, production development on Databricks (AWS, Azure, or GCP), including notebooks, Workflows, Delta Live Tables, Unity Catalog, SQL warehouses, and cluster configuration; experience with AWS (S3, Glue) or Azure (ADLS, ADF) object storage and services.
· Delta Lake / Apache Iceberg: time travel, schema evolution, upsert/merge operations, partition optimization — building versioned, ACID-compliant data assets at scale; Delta Lake experience ideally hands-on within the Databricks Lakehouse platform using Delta Live Tables (DLT) for declarative pipeline authoring.
· ETL/ELT tooling: AWS Glue, dbt, dbt-databricks, or equivalent for building tested, documented transformation pipelines with automated data quality checks; familiarity with Databricks Asset Bundles (DABs) for packaging and deploying notebooks, jobs, and DLT pipelines as code.
· Version control and DevOps: daily use of Git and GitHub/Azure DevOps (branching strategies, pull requests, code review, merge conflict resolution), CI/CD pipelines (GitHub Actions, Azure DevOps), and Docker, maintaining all code as version-controlled, testable, and deployable artifacts; experience deploying to Databricks via Asset Bundles, CLI, or REST API.
· Data modeling: building star/snowflake schemas, OMOP-compliant structures, and flat domain long files optimized for analytical workloads in a research context, materialized as managed Delta tables within Databricks Unity Catalog with appropriate access controls and lineage tracking.
Analytic Application Development
- Building and deploying interactive analytic applications with Dash (Plotly), R Shiny, Streamlit, or Databricks Apps, connected to Databricks SQL warehouses and Unity Catalog with appropriate authentication and access controls.
- Front-end fundamentals sufficient to deliver usable, performant applications for scientific users, including caching, pagination, and efficient query patterns against large tables.
- R programming: advanced R development for data engineering and analytics, including building and deploying production R Shiny dashboards; writing and maintaining internal R packages that standardize how teams connect to, query, and analyze data products (e.g., shared connection functions to Databricks, cohort-building utilities, and reusable analytic functions); and applying good package development practices such as roxygen2 documentation, testthat unit tests, version control, and renv for reproducible environments.
Data Quality & Validation
- Designing automated data quality frameworks using tools such as Delta Live Tables expectations, Great Expectations, dbt tests, or OHDSI Data Quality Dashboard (DQD), with results tracked across data refreshes.
- Validating claims and EHR data products against source, vendor documentation, and published benchmarks, including record reconciliation, distribution and trend checks, and investigation and resolution of data anomalies.
Healthcare RWD — Claims and EHR
- Administrative claims: deep, hands-on working knowledge of medical and pharmacy claims from sources such as Optum Clinformatics, IQVIA PharMetrics, and Komodo, including enrollment/eligibility tables, revenue codes, place-of-service codes, inpatient vs. outpatient claim splitting, drug identification at NDC and GPI level, and known structural quirks of each source.
- Complex EHR data: HL7 FHIR resources, Epic/Cerner/Truveta data models, clinical note schemas, problem list hierarchies, medication order and administration tables, lab result normalization, and vital sign time series — including semi-structured and nested JSON/XML from EHR exports and FHIR APIs.
- Healthcare terminology and ontology mapping: ICD-10-CM/PCS, ICD-9-CM, NDC, RxNorm, SNOMED CT, CPT-4, HCPCS, LOINC, ATC — building, versioning, and governing code list repositories joined to raw tables to produce concept-labeled analysis-ready datasets.
- OMOP CDM: transforming source claims and EHR data to OMOP v5.x including vocabulary loading (Athena), ETL specification documentation, and Achilles/DQD data quality checks.
- Phenotyping and cohort construction: translating clinical study protocols into reproducible extraction logic — index date derivation, washout periods, time-varying covariates, censoring — suitable for pharmacoepidemiology and HEOR studies.
- Regulatory and privacy frameworks: HIPAA-compliant data handling, de-identification standards (Safe Harbor, Expert Determination), DUA compliance, and audit trail requirements for FDA-grade RWE submissions.
AI Fluency
- AI-assisted software development: expert, daily use of AI coding assistants and agentic coding tools (e.g., Claude Code, GitHub Copilot, Cursor, Databricks Assistant) to write, refactor, debug, test, and document production code, with strong judgment on reviewing and validating AI-generated code.
- Building with frontier models: hands-on experience integrating frontier AI model APIs (e.g., Anthropic, OpenAI, Google, Databricks Foundation Model APIs) into solutions that operate on data products, including prompt engineering, tool use/function calling, structured outputs, retrieval-augmented generation (RAG), and agentic workflows.
- Evaluating AI solutions: designing test sets and evaluation methods to measure the accuracy and reliability of AI outputs (e.g., natural-language-to-SQL correctness, extraction accuracy), and implementing guardrails appropriate for patient-level data.
- Implementing AI-powered data profiling and anomaly detection to automate quality checks across large ingestion runs and surface issues before they reach analytical teams.
- Familiarity with MLOps tooling (MLflow, Databricks Model Serving, Vector Search) for versioning, deploying, and monitoring AI models integrated with data pipelines; experience applying NLP or LLMs to extract structured data from unstructured clinical notes a plus.
Minimum Qualification Requirements:
- Bachelor’s in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, Bioinformatics.
- Minimum 5 years' experience of hands-on data engineering experience with a demonstrated focus on healthcare or life sciences RWD, including direct work with large administrative claims and/or EHR datasets.
- Minimum of 1 year experience of hands-on, production development on the Databricks platform using Python (PySpark), SQL, and Delta Lake, applied directly to real-world healthcare data at scale.
- Minimum of 1 year experience designing and implementing data quality and validation checks for large clinical datasets.
- Minimum of 1 year of experience using AI coding assistants to develop production software, and experience building at least one solution using frontier AI model APIs.
Other Information/Additional Preferences:
- Experience with OMOP CDM transformation, vocabulary management (Athena), and federated analytics networks (OHDSI, PCORnet, Sentinel).
- Master’s degree in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, Bioinformatics.
- Strong problem-solving skills, attention to detail, and ability to work effectively in matrixed, cross-functional teams with both technical and scientific partners.
- Experience with distributed computing frameworks (Spark, Dask) for large-scale RWD processing.
- Experience building and deploying analytic applications using Dash, R Shiny, Streamlit, or similar frameworks.
- Strong understanding of healthcare coding standards (ICD-10, NDC, RxNorm, SNOMED CT, CPT, LOINC), claims and EHR data structures, and major RWD vendors and platforms (e.g., Merative MarketScan, Optum, IQVIA, Komodo, Truveta, HealthVerity).
- Daily proficiency with Git-based version control, including branching, pull requests, and code review.
- Advanced SQL optimization skills on Databricks SQL / Spark SQL, including query plan analysis, query tuning, and compute cost management.
- Demonstrated ability to build production-grade ingestion pipelines for multi-terabyte secondary healthcare datasets, including administrative claims and complex EHR data.
- Experience with data pipeline observability and quality tooling (e.g., Great Expectations, Monte Carlo, Lakehouse Monitoring).
- Familiarity with FDA RWE Framework guidance, EMA RWD guidance, and audit-readiness requirements for observational studies.
- Knowledge of Agile/Scrum methodologies and project management tools such as JIRA
- Databricks certification (e.g., Data Engineer Associate or Professional, Generative AI Engineer Associate).
- Experience with Model Context Protocol (MCP) or similar frameworks for connecting AI models to data and tools.
- A portfolio of code (e.g., GitHub repositories) or deployed analytic applications demonstrating hands-on engineering work.
- Recruitment open across sectors: pharma/biotech data teams, RWD vendors (Truveta, Komodo, Optum, IQVIA, HealthVerity), cloud providers (AWS, Google, Microsoft), genomics platforms (Illumina, DNAnexus, Broad Institute), or academic health systems with large-scale RWD programs.
NOTE: This job description is intended to provide a general overview of the job requirements at the time it was prepared. The job requirements of any position may change over time and may include additional responsibilities not specifically described in the job description. For GMP purposes, the job description should be updated for significant changes. As always, you should consult with your supervisor regarding your actual job responsibilities and any related duties that may be required for the position.
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form () for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.
Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia (AMECA), Black Employees at Lilly (View email address on waitwhatdoyoudo.com), Chinese Culture Network (CCN), EnAble, Evolve, Lilly Indian Network (LIN), Organization of Latinx at Lilly (OLA), Pride (LGBTQ+ Allies), Veterans Leadership Network (VLN) and Women’s Initiative for Leading at Lilly (WILL).
Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is
$136,500 - $200,200Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
#WeAreLilly
$136.5k - $200.2k
...bring your best to work that truly matters for patients, we invite you to join us. Job Description Purpose: The Associate Director, RWD Data Engineering is a hands-on technical role that builds and delivers the engineering vision for Lilly’s real-world data (RWD) infrastructure...SuggestedFull timeFlexible hours$130.9k - $268.7k
...for this role ends on 12/7/2026. Work you’ll do US Cyber Data Protection team is responsible for securing and protecting... ...our brand. We are seeking an experienced, strategic Cyber Engineering Associate Director to join the US Cyber Data Protection team in the Deloitte...Suggested$129.3k - $177.8k
...Become a part of our caring community The Associate Director, Data Science uses mathematics, statistics, modeling, business analysis, and emerging technologies to transform high volumes of complex structured and unstructured data into advanced analytic solutions. The...SuggestedFull timeTemporary workApprenticeshipWork at officeRemote workWork from homeHome officeFlexible hours$170k
...A Day in the Life: The Director, Data Engineer owns the strategy, reliability, and evolution of the enterprise business intelligence and analytics platform ecosystem. This role ensures BI platforms are scalable, secure, cost-efficient, and operationally excellent-enabling...SuggestedWorldwide- We Are The beginning of a new Data & AI decade that will reshape work and society has begun. Accenture is stepping boldly into this... ..., and Power Ecosystem automation.The WorkWe are seeking a Data Engineering, Management & Governance Manager to own the technical integrity...SuggestedLive inWork at officeLocal area
- We Are The beginning of a new Data & AI decade that will reshape work and society has... ...You are A strong Data Architect / Data Engineering Lead who can provide technical leadership... ...(minimum 12 years) work experience. (If Associate’s Degree, must have minimum 6 years work...Temporary workWork experience placementLive inWork at officeLocal area
$102.3k - $209.5k
...operation of mission-critical electrical systems supporting OCI data centers. This role owns electrical maintenance execution,... ...maintenance completion, and corrective action closure. Partner with engineering, commissioning, and deployment teams on upgrades, lifecycle...Temporary workFor contractorsFlexible hours- ...Mid-Level Data Engineer We are seeking a Mid-Level Data Engineer to support a growing data and AI team focused on modernizing how enterprise data is discovered, integrated, governed, and made available for analytics and machine learning. This role will focus primarily...
- ...Data Engineer Our business technology team is enhancing its data engineering capabilities with focus in financial services data including accounting, finance, and broader Enterprise Data initiatives. We are seeking a skilled Data Engineer, along with at least three...
$120.47k
...Employment Information: 1. Location of Employment (City, State): 450 Elanco Circle, Indianapolis, IN 46221 2. H-1B Job Title: Data Engineer 3. Occupation Code: 15-2051.00 (Data Scientists) 3. Period of Employment (mm/dd/year to mm/dd/year): 07/30/2026 to 07/29/2029 4...H1bWork at office- ...startups on high-impact projects across Data, AI, Cloud, and Software Engineering. Our expertise in AI solution... ...leads, and research informatics directors to translate operational friction... ...Core/Advanced, or AWS Data Engineer Associate/Professional. What This Role...Full timeContract workFor contractors3 days per week
$150k - $190k
...Advanced Solutions Group of SHI that is building new digital experiences for our internal users, customers and partners. The Senior Data Engineer will be responsible for the analysis, design, and development of solutions focused on data engineering and ETL workflows. As a...Full timeWork experience placementRemote workWorldwideFlexible hoursShift work$99k - $149k
...enthusiasm for innovation and creating the best experience for job seekers. (*Comscore, Total Visits, March 2026) Day to Day The Data Engineer's role at Indeed is to integrate data from a variety of internal and external sources into common data mesh domain models, in...Work experience placementLocal areaRemote work- ...Databricks Data Engineer Anywhere Type: Permanent Category: Development Industry: Energy Workplace Type: Remote Standard... ...routing number) until you have worked with someone clearly associated with Eliassen Group. If you have any indication of...Hourly payPermanent employmentFull timeLocal areaRemote work
- ...Data Engineer The Data Engineer plays a critical role in the successful execution of the client’s data strategy. This role will be focused on data acquisition, curation, and transformation of data into a useable asset. They will work cross-functionally to develop and...
- ...Job Description Position Overview: The Senior Data Engineer is critical in shaping the strategic data framework of the organization. This role demands a deep understanding of data systems and analytics, tasked with leading the development and integration of extensive...
$98k - $198k
...healthcare intelligence. Our company purpose is to empower easy, data-driven decision-making on important healthcare questions.... ...Position Summary: We are seeking a seasoned Senior Data Engineer proficient in Spark and SQL to spearhead the design and enhancement...Full timeTemporary workFor contractorsWork experience placementRemote workFlexible hours$125.24k - $187.86k
...U.S. Visa sponsorship/sponsorship assumption and other immigration support are not available for this position. As a Senior Data Engineer on the Data Platform team, you will build the data foundation that Lyric's products run on. Data Platform team owns the canonical...Permanent employmentFull timeRemote workVisa sponsorship$171k - $299.2k
...invite you to join us. Overview The Director/Senior Director RWD Scientist will be an integral leader... ...knowledge and expertise in real-world data, analytics, and outcomes research, this... ...methods to solve HEOR issues associated with registration, pricing, reimbursement...Full timeRemote workFlexible hours- ...growth. We treat our people like family, we are mission-focused, and we give back! Join us today. We are seeking a skilled Data Analytics Engineer to join our team to support the U.S. Army Financial Management Command (USAFMCOM) in Indianapolis, IN. You will be...Work experience placement
$91.1k - $179.5k
...Recruiting for this role ends on 10/30/2026. Work you'll do The Data Engineer Consultant will support data architecture and engineering... ...not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Deloitte...Local area$52 - $67 per hour
...building, optimizing, and managing large-scale data pipelines and solutions that power... ...information systems, Computer Science, Engineering or equivalent years of experience in data... ...such as: -Azure Data Engineer Associate -Azure Solutions Architect Expert Experience...Contract workTemporary work$86.7k - $170.9k
Position Summary As a Data Engineer II on the AI & Data team, you will be responsible for: Design, build, and maintain data pipelines... ...not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At...Local area$150k - $180k
...The Data Analytics Engineer is responsible for designing, developing, and supporting enterprise reporting and analytics solutions that enable data-driven decision-making across the organization. This role combines technical expertise in data integration, data modeling,...Contract workWork at office- ...Job Description Job Description Job Title: Senior Databricks Data Engineer Work Location: St. Louis, MO, Richardson, TX 75082 or Indianapolis, IN 46241 Contract duration: 06 Months Detailed Job Description: Bachelor s degree or foreign equivalent required...Contract work
$159k - $233.2k
...purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. NLP & Market Research Data Engineering Lead Customer Insights and Analytics (CIA) At Lilly, we unite caring with discovery to make life better for people...Full timeContract workFlexible hours- ...include: Leverage Elanco's Enterprise Data and PaaS/SaaS Platforms to deliver... ...in the design and execution of systems engineering solutions, driving standards across the... ...Cloud platforms (Microsoft Azure, GCP) and associated automation/integration services (Logic...Work at office
$145k - $225k
...2027-10-21Job Summary:BDO is seeking a Director to join BDO's Cyber Trust & Transformation... ...and hands-on engagement leader across data privacy, data governance, AI governance,... ...members ranging from Senior Managers through Associates:Supervises workload and staffing across...Work at office- Position Summary Deloitte Global is the engine of the Deloitte network. Our professionals reach across disciplines and borders... ...programs and services that unite our organization. The Director, Data Management Capabilities & AI Readiness is responsible for defining...Visa sponsorship
$147k - $190k
...being redefined by AI. About the Role You own LILTs data transformation layer: the dbt layer and warehouse behind every... ...you make it a role. You lead a new Data sub-team in Platform Engineering reporting to the head of Platform as a hands-on player-manager...Full timeLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Associate Director - RWD Data Engineering. Be the first to apply!
- associate director clinical operations Indianapolis, IN
- remote associate product manager Indianapolis, IN
- associate director contracts Indianapolis, IN
- associate scientific director Indianapolis, IN
- associate director clinical research Indianapolis, IN
- associate product manager web Indianapolis, IN
- associate clinical manager Indianapolis, IN
- associate director Indianapolis, IN
- associate director clinical data management Indianapolis, IN
- associate manager Indianapolis, IN





