Senior Data Engineer

Role Description

The Senior Data Engineer will be a cornerstone of Careforth’s data platform team, architecting and operating the cloud-native, lakehouse-based infrastructure that powers our full suite of AI and analytics products — from clinical and caregiver risk intelligence to LLM-enabled chatbots, real-time signal capture, and payer-facing outcome reporting.

The ideal candidate brings deep, hands-on expertise with Databricks, Apache Spark, and AWS Cloud services, and can serve as both a technical leader and mentor. You will own end-to-end pipeline design from raw source ingestion through curated analytics-ready datasets, ensuring data systems are reliable, scalable, and HIPAA-compliant.

• Architect and build robust, scalable, and secure data pipelines leveraging Databricks (Delta Lake, Spark, Unity Catalog, Workflows) and AWS (S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis), supporting both batch and real-time processing modes.
• Design and evolve the enterprise lakehouse architecture, including data modeling, Delta table optimization, data versioning, and a semantic data layer and metrics store serving BI tools and downstream ML feature engineering.
• Build and maintain ETL/ELT orchestration (Airflow, dbt) and stream processing pipelines (Kafka/Kinesis) ingesting claims, EHR/EMR, pharmacy, IoT, survey, CRM, and digital engagement event data.
• Partner with Data Scientists to design and maintain feature store architecture, real-time member context assembly, and temporal feature computation services supporting risk and predictive models.
• Develop data infrastructure for LLM and NLP use cases: vector databases, knowledge base ingestion, transcript storage and indexing, secure audio storage, and LLM response logging and audit trails.
• Build APIs and integrations for downstream consumers: care team dashboards, lead scoring, pathway recommendation, queue prioritization, CRM connectors (Salesforce, HubSpot), and payer-facing data exchange.
• Implement HIPAA-compliant data handling across all pipelines including PHI detection, redaction, audit trails, row-level security, SSO, and secrets management (AWS Secrets Manager, Vault).
• Build reporting infrastructure including a report template engine, multi-format export (PDF, Excel, CSV), scheduled report generation, and delivery APIs for internal and payer-facing reporting.
• Conduct pull request reviews, enforce engineering best practices, mentor junior engineers, and research and apply emerging data engineering tools and patterns.
• Participate in design discussions with technical leads across product lines; perform other duties and special projects as assigned.

Qualifications

• Bachelor’s or Master’s Degree in Computer Science, Data Engineering, or a related field.
• 7–10 years of professional data engineering experience, including 3+ years with modern cloud-based data lake or lakehouse architectures.
• Deep expertise in Apache Spark (PySpark, Scala, or Java) for large-scale distributed data processing and strong hands-on Databricks experience across Delta Lake, Unity Catalog, and Workflows.
• Hands-on experience with AWS data services: S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis, DynamoDB, EMR, Bedrock, Transcribe, and Comprehend Medical.
• Experience in healthcare or regulated data environments; familiarity with HIPAA technical safeguards required.
• Demonstrated experience building data infrastructure in support of ML feature stores and model serving pipelines.

Requirements

• Strong Python (preferred); Java or Scala also valued. Proficient SQL across relational, NoSQL, and lakehouse environments.
• ETL/ELT orchestration with Airflow, dbt, and AWS Step Functions; stream processing with Kafka/Kinesis.
• Experience with vector databases, embedding stores, document ingestion pipelines, and RAG pipeline data layers for LLM use cases.
• Container orchestration with Docker and Kubernetes/EKS; CI/CD for data pipelines; familiarity with data quality and data contract frameworks.
• BI tool integration, ad-hoc query execution engine design, query result caching, and multi-format report delivery pipelines.

Soft Skills

• Strong communicator able to translate data architecture decisions to clinical, operational, and executive stakeholders.
• Collaborative by nature; comfortable driving alignment across data science, product, engineering, and clinical teams.
• Detail-oriented and committed to data quality, HIPAA compliance, and operational excellence; self-directed with a continuous learning mindset.

Benefits

At Careforth your well-being matters. With flexible schedules, a remote-first culture, and a nationally recognized wellness program, our benefits are designed to help you thrive, both professionally and personally. Discover how we invest in you:

The pay range for this position is $112000 - $170000. The actual wage offered may be lower or higher depending on budget and candidate experience, knowledge, skills, qualifications, and geographic location.