Azure Databricks Engineer

Job Description

Job Description

We are seeking an experienced Azure Databricks Data Engineer to support the modernization and enhancement of enterprise healthcare data platforms. This role will focus on designing, building, and maintaining scalable data pipelines within Azure Databricks while supporting the migration of legacy SQL-based workloads to a modern cloud architecture. Approximately 70% of the role will involve new development and pipeline engineering, while 30% will focus on production support, troubleshooting, and remediation of existing data pipelines.

The ideal candidate will have strong expertise in Azure Databricks, PySpark, Delta Live Tables (DLT), Medallion Architecture, and Infrastructure as Code / Pipeline as Code concepts leveraging YAML and configuration-driven development. Experience supporting Medicare, payer, or claims-based healthcare data environments is highly preferred. The role aligns closely with a configuration-driven, "Pipeline as Code" architecture built around YAML configurations, reusable frameworks, and automated CI/CD deployments.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

equirements

5+ years of Data Engineering experience building large-scale data platforms

Strong hands-on experience with Azure Databricks, Delta Lake, and ADLS

Advanced PySpark development experience including DataFrames, transformations, joins, aggregations, and performance tuning

Experience developing and maintaining Delta Live Tables (DLT) pipelines

Strong SQL experience and background supporting legacy data warehouse environments

Experience migrating SQL-based pipelines into modern Python/PySpark-based architectures

Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies

Proficiency working with YAML, JSON, and configuration-driven development patterns

Experience automating CI/CD deployments through DevOps pipelines

Strong understanding of Medallion Architecture (Bronze, Silver, Gold layers) and associated data processing patterns

Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers

Experience troubleshooting, monitoring, and supporting production data pipelines

Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake technologies is preferred

Healthcare industry experience preferred

Experience working with Medicare, payer, member enrollment, or claims data strongly preferred

Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Key Responsibilities

Design, develop, and deploy new Azure Databricks data pipelines using PySpark and Delta Live Tables

Translate and modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions

Build configuration-driven data pipelines utilizing YAML, JSON, and reusable framework patterns

Develop and maintain automated CI/CD deployment processes for data engineering workflows

Implement and support Medallion Architecture data pipelines across Bronze, Silver, and Gold layers

Partner with upstream and downstream teams to ensure data quality, reliability, and platform scalability

Troubleshoot and resolve production pipeline failures within both Databricks and legacy SQL environments

Enhance monitoring, observability, and operational support processes for healthcare data pipelines

Support Medicare and payer-focused data initiatives, including member, claims, and operational datasets

Follow established architecture patterns and engineering standards to ensure consistency across the platform