Manager Data Engineer

Manager Data Engineer

Penn Foster Group is seeking an experienced Lead Data Engineer to help shape the future of our enterprise data platform and AI strategy. This is a highly visible technical leadership role responsible for architecting, building, and evolving our modern cloud data platform while establishing engineering standards, mentoring a growing team, and driving innovation across the organization.

As a Lead Data Engineer, you will serve as the technical leader for our Databricks Lakehouse platform, providing hands-on leadership across data ingestion, transformation, governance, analytics enablement, and AI-ready data products. You will partner closely with Business Intelligence, Product, Platform Engineering, Security, Data Governance, and MLOps teams to build trusted, scalable, and governed data solutions that power enterprise analytics, machine learning, and generative AI.

This role combines deep technical expertise with leadership, mentoring, and strategic influence. You'll help establish the long-term technical direction of Penn Foster Group's data ecosystem while coaching and developing engineers, driving engineering excellence, and implementing modern data engineering best practices.

Why Join Penn Foster Group?

Penn Foster Group is in the midst of an exciting enterprise-wide data platform modernization journey. We are investing in a modern cloud-native architecture built on Databricks, Azure, and AI-enabled technologies that will become the foundation for analytics, operational reporting, machine learning, and generative AI across the company.

This is a unique opportunity to join at a transformational point in our technology journey. Rather than simply maintaining existing systems, you'll help define the future of our data platform—shaping architecture, engineering standards, governance, and the adoption of emerging AI technologies that will influence how data is used across the organization for years to come. You will also be a key player in our drive for Self Service Analytics at PFG.

If you're passionate about building modern data platforms, mentoring engineers, and helping shape the future of AI-enabled analytics, this is an opportunity to make a lasting impact.

Key Responsibilities

Technical Leadership

• Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
• Mentor and develop junior and mid-level Data Engineers, fostering technical growth, knowledge sharing, and continuous learning.
• Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence.
• Lead technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices.
• Champion a culture of quality, collaboration, innovation, and continuous improvement.

Data Platform & Databricks Leadership

• Serve as the technical lead for Penn Foster Group's Databricks Lakehouse platform.
• Design, build, and support scalable enterprise data pipelines using SQL, Python, Apache Spark, and Databricks.
• Define best practices for Databricks development, including Workflows, Repos, Jobs, notebooks, reusable Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation.
• Design and optimize Delta Lake architectures using Medallion patterns, Delta optimization, Liquid Clustering, and Photon.
• Build reusable ingestion frameworks supporting batch, streaming, CDC, and API-based integration patterns.
• Optimize Spark workloads for performance, scalability, reliability, and cloud cost efficiency.

AI & Databricks Genie Enablement

• Partner with Analytics and business stakeholders to develop trusted semantic data products that power Databricks Genie.
• Design and maintain Genie Spaces and Genie Ontologies that accurately represent business entities, relationships, metrics, and terminology.
• Establish best practices for semantic modeling, governed metrics, business metadata, and AI-ready datasets.
• Evaluate and implement emerging Databricks AI capabilities to improve self-service analytics and business productivity.
• Collaborate with Data Science and MLOps teams to enable machine learning, generative AI, and advanced analytics initiatives.

Data Architecture, Governance & Operational Excellence

• Design scalable Lakehouse architectures supporting analytics, reporting, AI, and operational data products.
• Partner with Data Governance teams to implement enterprise governance using Unity Catalog, including data lineage, fine-grained security, metadata management, and access controls.
• Champion automated testing, monitoring, observability, data quality, and production reliability.
• Serve as the technical escalation point for complex production issues and lead root cause analysis and continuous improvement efforts.
• Drive platform modernization initiatives while balancing delivery, scalability, maintainability, and operational excellence.

Cross-Functional Partnership

• Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams to deliver trusted, scalable data products.
• Translate business requirements into modern technical solutions that support enterprise reporting, analytics, AI, and strategic decision-making.
• Contribute to technical roadmaps, platform strategy, and the continued evolution of Penn Foster Group's Data & Analytics capabilities.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.
• 8+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.
• 3+ years of experience leading technical initiatives and mentoring engineering teams.
• Demonstrated success developing engineers while driving engineering excellence and technical standards.
• Strong communication and collaboration skills with the ability to influence both technical and business stakeholders.

Technical Qualifications

Required

• Expert knowledge of the Databricks Lakehouse Platform, including: Apache Spark (PySpark), Delta Lake, Unity Catalog, Databricks Workflows & Jobs, Repos, SQL Warehouses, Cluster Policies, Serverless Compute, MLflow, Lakehouse Monitoring, Auto Loader, Delta Live Tables / Lakeflow
• Expert Knowledge in SQL and Python.
• Strong Experience working with BI tools such as PowerBI, Tableau or Business Objects
• Strong Microsoft Azure experience, including ADLS Gen2, Microsoft Entra ID (Azure AD), RBAC, networking, and cloud security.
• Experience implementing Medallion Architecture, dimensional modeling, and domain-oriented data products.
• Deep understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization.
• Experience implementing CI/CD pipelines, Git-based development workflows, Infrastructure as Code, and automated testing.
• Strong understanding of data governance, metadata management, data quality, observability, security, and compliance.

Preferred

• Experience implementing Databricks Genie, Genie Spaces, and Genie Ontologies.
• Experience designing semantic models and AI-ready data products.
• Experience supporting Power BI, Tableau, or other enterprise BI platforms.
• Experience with machine learning platforms, MLOps, or Generative AI applications.
• Experience with dbt, Great Expectations, or similar modern data engineering tools.
• Databricks Certified Data Engineer Professional and/or Microsoft Azure certifications.