Dir Data Engineering (Databrick/ADO/Heathcare) POST NUMBER: 484414 [Remote]

Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual’s skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs. With that said, as required by local law in geographies that require salary range disclosure, Vaco/Highspring notes the salary range for the role is noted in this job posting. The individual may also be eligible for discretionary bonuses, and can participate in medical, dental, and vision benefits as well as the company’s 401(k) retirement plan. Additional disclaimer: Unless otherwise noted in the job description, the position Vaco/Highspring is filing for is occupied. Please note, however, that Vaco/Highspring is regularly asked to provide talent to other organizations. By submitting to this position, you are agreeing to be included in our talent pool for future hiring for similarly qualified positions. Submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. Further assessment of candidates beyond this initial phase within Vaco/Highspring will be otherwise assessed by recruiters and hiring managers. Vaco/Highspring does not have knowledge of the tools used by its clients in making final hiring decisions and cannot opine on their use of AI products.

Director, Data Engineering & Platform

Reports to: Vice President, Enterprise Data Platforms
Direct Reports: Approximately 10 Data Engineers
Peer Leaders: Director, Data Products; Director, Data Architecture
Core Platform: Databricks
Delivery Environment: Azure DevOps (ADO), Agile/Scrum

Role Summary

The Company is seeking a Director of Data Engineering & Platform to serve as the engineering and platform leader for the enterprise data organization. Reporting to the Vice President of Enterprise Data Platforms, this leader will own the teams, engineering practices, platform operations, and delivery processes responsible for turning the Company’s enterprise data strategy and architecture into scalable, production-ready solutions.

The Director will lead a team of approximately 10 data engineers operating across multiple Databricks environments and will be accountable for the reliability, scalability, performance, and disciplined delivery of the Company’s enterprise data platform.

This is a highly technical leadership role requiring deep Databricks experience combined with strong people leadership and delivery management capabilities. The Director will remain close enough to the technology to guide engineering decisions, challenge designs, establish standards, and troubleshoot complex platform issues while developing and leading a high-performing engineering organization.

The role will operate as a key member of the VP’s data leadership team alongside the Director of Data Products and Director of Data Architecture . Together, these leaders will connect business priorities, enterprise architecture, and engineering execution.

Key Responsibilities

Data Engineering Leadership

• Lead, develop, and manage a team of approximately 10 data engineers responsible for building and operating the Company’s enterprise data platform.
• Establish a strong engineering culture focused on quality, accountability, scalability, automation, and predictable delivery.
• Set engineering standards for data pipelines, transformations, testing, deployment, monitoring, documentation, and production support.
• Provide technical guidance and mentorship to engineers while developing future technical and team leaders.
• Maintain sufficient technical depth to participate in design reviews, challenge engineering approaches, and help resolve complex technical issues.
• Establish clear performance expectations and continually improve the productivity and capabilities of the engineering organization.

Databricks Platform Leadership

• Serve as the primary engineering leader for the Company’s Databricks ecosystem .
• Lead engineering and platform execution across multiple Databricks environments and workspaces .
• Establish consistent standards for development, testing, promotion, security, monitoring, and production operations across environments.
• Drive platform performance, scalability, reliability, observability, and cost optimization.
• Partner with Data Architecture on the implementation and evolution of the Company’s lakehouse architecture.
• Establish reusable frameworks and engineering patterns leveraging Databricks, Spark, Delta Lake, Python, and SQL.
• Support adoption and optimization of capabilities including Unity Catalog, Delta Lake, Databricks Workflows, MLflow, and broader Databricks tooling .

Deep, production-level Databricks experience is a must-have for this position.

Engineering Delivery & Agile Leadership

A major component of this role is creating a disciplined and highly effective engineering delivery organization.

• Own the engineering team's Agile/Scrum delivery process.
• Serve as a strong Scrum leader , driving sprint planning, backlog refinement, daily stand-ups, retrospectives, capacity planning, and delivery commitments.
• Establish clear sprint goals, engineering priorities, and measurable delivery expectations.
• Partner with Data Products to translate product priorities and business requirements into executable engineering work.
• Manage engineering capacity across platform initiatives, new development, technical debt, production support, and strategic projects.
• Identify and remove blockers that impact engineering velocity.
• Create visibility into engineering progress, dependencies, risks, and delivery commitments.
• Drive continuous improvement through sprint metrics, retrospectives, and engineering performance data.

Azure DevOps (ADO)

• Own and mature the engineering team's use of Azure DevOps (ADO) for planning, execution, and delivery management.
• Establish standards for Epics, Features, User Stories, Tasks, Bugs, and engineering backlogs.
• Ensure engineering work is properly defined, estimated, prioritized, assigned, and tracked.
• Build visibility into sprint velocity, capacity, backlog health, delivery risks, and dependencies.
• Leverage ADO to improve transparency and communication across Engineering, Data Products, Data Architecture, and executive leadership.
• Partner with engineering teams on CI/CD and automated deployment practices.

Strong hands-on Azure DevOps and Agile/Scrum leadership experience is required.

Data Pipeline Engineering

• Lead the design, development, and operation of scalable pipelines ingesting and transforming clinical, financial, operational, and enterprise data.
• Establish modern engineering standards for ETL/ELT, batch and near-real-time processing, orchestration, testing, observability, and code promotion .
• Ensure pipelines meet established standards for performance, reliability, data quality, and availability.
• Build reusable engineering frameworks that accelerate the integration of new data sources.
• Ensure engineering capabilities can scale to support the Company’s continued organizational growth and acquisition activity.
• Establish production support and incident-management practices for critical data pipelines and platform services.

Partnership Across the Data Organization

Director of Data Products

Partner closely with the Director of Data Products to translate business and clinical priorities into an executable engineering roadmap.

Data Products defines what needs to be delivered and why ; Data Engineering determines how it gets built and delivered reliably at scale .

Director of Data Architecture

Partner closely with the Director of Data Architecture to implement the Company’s enterprise data architecture, governance, modeling, security, and data quality standards.

Data Architecture establishes the architectural direction and standards ; Data Engineering turns those standards into production technology and engineering practices .

VP, Enterprise Data Platforms

Provide the VP with clear visibility into engineering capacity, platform health, delivery performance, technical risks, resource needs, and major dependencies.

The Director will be the VP’s primary leader for engineering execution and Databricks platform delivery .

Platform Reliability & Engineering Excellence

• Establish clear SLAs and operational standards for critical data pipelines and platform services.
• Implement monitoring, alerting, observability, and incident-response practices.
• Drive automated testing and data validation throughout the engineering lifecycle.
• Establish CI/CD and disciplined code-promotion processes across Databricks environments.
• Ensure appropriate source control, peer review, documentation, and release-management practices.
• Continuously identify opportunities to improve platform performance and engineering productivity.
• Balance new feature delivery with technical debt, platform maintenance, and reliability.

Security, Governance & Compliance

• Partner with Data Architecture, Security, and Governance teams to ensure engineering solutions meet enterprise security and data governance standards.
• Ensure appropriate access controls, auditability, encryption, and secure development practices across Databricks environments.
• Ensure data engineering practices support HIPAA and applicable healthcare regulatory requirements.
• Implement data lineage, quality, classification, and governance standards within engineering workflows.

Required Qualifications

• 8 years of progressive data engineering experience , including significant experience building enterprise-scale data platforms.
• 3 years of engineering leadership experience , ideally managing teams of 8–15 engineers.
• Deep hands-on Databricks experience is required , including experience operating complex production environments.
• Demonstrated experience working across multiple Databricks environments/workspaces .
• Strong expertise with Python, SQL, PySpark/Spark, and Delta Lake .
• Strong experience building production-grade ETL/ELT pipelines and modern lakehouse architectures.
• Experience with Databricks Workflows and/or orchestration technologies such as Azure Data Factory or Airflow.
• Strong understanding of CI/CD, automated testing, observability, source control, and modern engineering practices.
• Heavy Azure DevOps (ADO) experience , including backlog management, sprint planning, work-item management, capacity planning, and delivery reporting.
• Demonstrated experience leading Agile/Scrum engineering teams and driving disciplined sprint execution.
• Strong experience managing technical teams while remaining meaningfully engaged with architecture and engineering decisions.
• Experience operating in Azure or another major cloud environment.

Preferred Qualifications

• Healthcare or other highly regulated industry experience.
• Experience working with clinical, EHR, claims, financial, or healthcare operational data.
• Experience with Unity Catalog and enterprise Databricks governance.
• Experience with Azure Data Factory, Azure Data Lake, Azure Functions, dbt, and related Azure data technologies.
• Experience supporting analytics, data science, machine learning, or AI workloads.
• Experience supporting enterprise growth, acquisitions, or integration of new business units and data sources.

Leadership Profile

The ideal candidate is an engineering leader first .

This person should be equally comfortable leading a team of 10 engineers, reviewing a complex Databricks design, managing competing priorities in ADO, running a disciplined Scrum process, and explaining platform risks or delivery timelines to senior leadership.

This is not primarily a BI, analytics, data governance, or data strategy position. Those capabilities will sit alongside this role within the broader data organization.

The Director of Data Engineering & Platform owns the critical middle layer between strategy and execution : the people, platform, processes, and engineering discipline required to reliably turn the Company’s data strategy into production capabilities.

Critical Must-Haves

1. Databricks: Deep, recent, production-level Databricks experience.
2. Engineering Leadership: Proven success leading and developing a data engineering team of comparable scale.
3. Platform Leadership: Experience owning engineering execution across multiple environments and complex enterprise data ecosystems.
4. Azure DevOps: Heavy hands-on ADO experience managing engineering backlogs, sprints, capacity, dependencies, and delivery.
5. Scrum: Strong Scrum leadership with the ability to create structure, accountability, and predictable engineering delivery.
6. Technical Depth: Strong Python, SQL, Spark/PySpark, pipeline, orchestration, CI/CD, and modern data engineering knowledge.
7. Execution: Ability to translate architecture and product priorities into scalable, production-ready engineering solutions.