Principal Data Engineer

Job Description

a pharmaceutical company based in San Diego is seeking a Principal level Data Engineer to join their team in support of their Research Data Solutions. The Principal Data Engineer is a senior technical leader responsible for designing, building, and optimizing scalable data solutions that support research and scientific data initiatives. This role requires deep expertise in modern data engineering practices, with a strong emphasis on Databricks, Spark, Delta Lake, lakehouse architecture, data quality, version control, performance optimization, and engineering best practices.The Principal Data Engineer partners primarily with business analysts, platform engineers, and other data engineers to translate research data needs into reliable, governed, and reusable data assets. This individual serves as a hands-on technical expert, architectural advisor, and mentor, helping establish standards and patterns for research-focused data engineering across the organization.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

• 7+ years of experience as Data Engineer, with proven knowledge or experience in the Biopharma/Life Science industry.

• Expert‑level experience architecting and building scalable data solutions using Databricks, including Spark, Delta Lake, Unity Catalog, and lakehouse best practices.

• Advanced proficiency in Python and SQL, with deep experience in Spark‑based data processing, performance tuning, and production analytics workloads.

• Strong experience designing and operating production ETL/ELT data pipelines, including orchestration, incremental processing, monitoring, and reliability.

• Hands‑on experience working in cloud data environments (Azure and/or AWS) supporting enterprise‑scale data platforms.

• Proven technical leadership experience, including leading architecture decisions, mentoring engineers, and partnering with analytics, platform, and research teams. - Hands‑on experience with LIMS, ELN, or scientific/discovery data sources.

• Strong understanding of data governance, security, privacy, and compliance practices in regulated environments.

• Experience working with Snowflake or other cloud data warehouses alongside Databricks.