Lead Data Engineer #3624301

Job Description

Job Description

Be Part Of A High-Performing Team

Join a mission-driven data and machine learning organization building modern technology that improves how individuals and families navigate aging and long-term care. This team combines the agility and innovation of a growing digital business with the resources and industry knowledge of an established financial services organization.

The environment brings together data engineers, analysts, data scientists, ML/AI engineers, product teams, and platform specialists. The Lead Data Engineer will help establish the reliable, governed, and scalable data foundation required to support analytics, machine learning, operational reporting, and customer-facing products.

What's In Store For You

This is a full-time, remote opportunity for candidates residing in approved Eastern Time states or Washington, DC.

The role offers the opportunity to lead technical delivery across a modern Databricks Lakehouse platform while remaining closely involved in architecture, pipeline development, modeling, governance, quality, and optimization.

Employees have access to comprehensive healthcare coverage, multiple retirement savings options, an employer-funded retirement account feature, generous paid time off, paid holidays, volunteer time, paid family leave, disability and life insurance, tuition reimbursement, student loan repayment support, professional training and certification assistance, wellness programs, caregiver resources, and mental health support.

How You Will Make An Impact

• Lead the design, development, and operation of scalable ETL and ELT pipelines using Python, SQL, Spark, and Databricks.
• Create reusable ingestion frameworks for batch and streaming data sources.
• Own production reliability by ensuring pipelines meet service levels, quality expectations, and operational standards.
• Design robust Lakehouse data models across raw, curated, semantic, and domain-oriented layers using Delta Lake.
• Build dimensional models, star schemas, analytics-ready datasets, and reusable data products for analytics and machine learning.
• Establish engineering standards for schema design, metadata, documentation, lineage, testing, and maintainability.
• Implement automated validation, anomaly detection, freshness monitoring, completeness checks, alerting, and observability.
• Partner with analysts to define business KPIs and deliver trusted curated datasets.
• Collaborate with data scientists and ML engineers to build reusable feature pipelines and ML-ready data assets.
• Optimize Spark jobs, SQL queries, cluster configurations, orchestration, compute usage, and storage patterns.
• Apply strong practices for Unity Catalog, RBAC, data privacy, PII handling, governance, and regulatory documentation.
• Reduce technical debt, simplify complex pipelines, mentor engineers, and help raise the organization’s engineering standards.

Do you have the expertise to lead modern Lakehouse data engineering?

• 7+ years of experience in data engineering or a closely related technical discipline.
• Advanced hands-on experience with Python, SQL, Spark, and distributed data processing.
• Strong production experience with Databricks, Delta Lake, and Lakehouse architecture.
• Demonstrated ownership of scalable ETL/ELT pipelines from design through production support.
• Deep understanding of dimensional modeling, star schemas, semantic layers, and domain-oriented datasets.
• Experience implementing automated testing, validation, monitoring, alerting, observability, and SLA controls.
• Experience supporting analytics, machine learning, data science, and product teams.
• Knowledge of Unity Catalog, RBAC, PII protection, data privacy, metadata, and lineage.
• Ability to improve performance and cost through Spark, SQL, cluster, compute, and storage optimization.
• Proven ability to establish technical standards, mentor engineers, communicate architectural decisions, and lead through influence.
• Must reside in an approved Eastern Time state or Washington, DC.
• Must not require current or future employment visa sponsorship.