AWS Healthcare Data Engineer

Benefits:

401(k) matching

Dental insurance

Health insurance

AWS Healthcare Data Engineer 100% Remote

Skills:- We are looking for a Senior Data Engineer with strong Databricks and streaming experience to build the core ingestion layer of a modern healthcare data platform. You will port a high-volume event pipeline into Databricks, integrate with Kafka, and implement complex patient matching and event handling logic.

KEY RESPONSIBILITIES • Port the core ingestion pipeline into a Databricks-native architecture conforming to standard ingestion layer patterns • Implement per-job success/failure tracking and metrics capture in alignment with platform engineering standards • Integrate with bulk patient matching libraries to accurately process patient update signals and lifecycle events • Build event handlers for patient merges, practice merges, and facility-level changes • Develop Databricks-to-Kafka (D2K) jobs for ingestion model outputs and downstream event streams • Ensure the solution is low-maintenance, well-documented, and observable in production

REQUIRED SKILLS & EXPERIENCE • Databricks — Delta Lake, Jobs, cluster management, Notebooks • Apache Kafka — producer/consumer patterns, event-driven architecture • PySpark / Spark Structured Streaming • Python — advanced data engineering • High-volume stateful event stream processing • Experience porting or refactoring large-scale data pipelines

NICE TO HAVE • Delta Live Tables (DLT) • Databricks Asset Bundles / CI/CD for Databricks • Confluent Kafka or AWS MSK • dbt on Databricks • Healthcare data or patient identity matching experience

This is a remote position.