Manager, Data Engineering

Responsibilities Lead, coach, and develop a team of Data Engineers, fostering strong technical skills and ownership Set clear priorities, manage workload, and ensure timely delivery of data engineering initiatives Establish engineering standards, code quality expectations, and best practices across the team Partner with stakeholders to translate business needs into scalable data solutions Oversee the design, development, and operation of data pipelines built in Databricks Ensure pipelines are scalable, reliable, and aligned to medallion architecture standards (bronze, silver, gold) Guide implementation of ingestion frameworks using tools like Fivetran and custom ingestion patterns Provide technical leadership for the Databricks platform, including workspace, jobs, clusters, and performance optimization Collaborate with Cloud Engineering to ensure seamless integration with AWS services (e.g., S3) Ensure alignment with enterprise architecture, including data modeling, partitioning strategies, and storage optimization Optimize compute usage for performance and cost efficiency Establish and enforce data quality standards, including testing, validation, and monitoring frameworks Ensure robust observability across pipelines (monitoring, alerting, lineage visibility) Partner with Governance teams to support metadata management and lineage through tools like Atlan Enforce security, compliance, and data access standards across all data engineering assets Own production support processes, including incident management, root cause analysis, and prevention Establish proactive monitoring and health checks for pipelines and platform performance Drive continuous improvement in automation, CI/CD, and release management practices Support and lead data migration efforts from legacy on-prem systems to cloud platforms Requirements Bachelor’s Degree or equivalent experience 7+ years of experience in data engineering or data platform roles 3+ years of experience leading and developing technical teams Experience operating in modern cloud data environments (Databricks, data lakes, or lakehouse platforms) Proven experience delivering enterprise-scale data pipelines and platforms Experience supporting cloud migration or modernization initiatives Strong expertise in Databricks (notebooks, jobs, cluster management) Proficiency in Python and PySpark for distributed data processing Advanced SQL skills and experience working with large-scale datasets Experience designing and operating cloud-based data platforms (AWS preferred) Experience with data ingestion tools (e.g., Fivetran or similar) Deep understanding of data modeling and medallion architecture patterns #J-18808-Ljbffr