Lead Data Engineer

Lead enterprise data engineering for a Databricks-based lakehouse and streaming analytics platform at Norfolk Southern Corp . In this hybrid role in Atlanta, GA , you will set technical direction, modernize ingestion and transformation patterns, and help guide steady-state operations while collaborating with cross-functional partners to deliver production-grade data pipelines. What you’ll do Lead the definition of data architecture with team members and stakeholders, establishing technical direction for lakehouse , ingestion , and transformation patterns. Identify and remove technical blockers by using hands-on problem solving, architectural decisions, and coordination across teams. Design and develop enterprise data models (conceptual, logical, and physical) across domains, ensuring alignment with lakehouse and pipeline architecture. Define and enforce data modeling standards , best practices, and governance processes using tools such as Erwin . Define data requirements, then gather, wrangle , and validate large-scale structured and unstructured data using data tools in the data environment. Enable data standardization, customization, and ad-hoc analysis by building mechanisms to ingest, analyze, validate, normalize, and clean data. Create data policy and develop interfaces and retention models that may require synthesizing or anonymizing data. Implement statistical data quality procedures for new data sources, using iterative analytics to support data scientists and analytics teams preparing data for visualization and insight generation. Develop and maintain data engineering best practices, contributing to insights related to data analytics and visualization . Lead a data engineering team to use state-of-the-art big data tools to build scalable architectures and high-performance pipelines for both real-time and batch data. Partner with data science and business intelligence teams to deliver models and pipelines for research, reporting, and machine learning. Build pipelines that clean, transform, and aggregate data from disparate sources. Use multiple languages and tools (for example, scripting) to integrate systems, and apply data architecture components from requirements through implementation while leading project teams. What you bring Databricks Lakehouse expertise (Required) : 10+ years in data engineering with deep hands-on experience across the Databricks ecosystem including Spark/PySpark pipelines, Delta Lake (merges, schema evolution), Delta Live Tables , Unity Catalog governance, Genie for self-service analytics, and LakeFlow Designer for visual ETL orchestration. Streaming and cloud infrastructure : 4+ years building reliable batch and streaming Spark workloads; 3+ years with Kafka (or Confluent) for high-volume event processing; and 3+ years using AWS analytics services including S3 , IAM , Glue/Lambda/MSK . Data modeling and SQL : ability to build conceptual, logical, and physical data models with strong governance, along with advanced SQL skills for business-ready datasets. Technical leadership : experience leading technical design, mentoring engineers, building architectural consensus, and unblocking complex data engineering challenges. Delivery and collaboration : experience delivering ETL/ELT pipelines end-to-end in a lakehouse environment, working within Agile frameworks such as Scrum/Kanban/SAFe to manage iterative delivery and cross-team dependencies. Tools and technologies Databricks, Spark, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, Genie, LakeFlow Designer Kafka, Confluent AWS analytics services (S3, IAM, Glue/Lambda/MSK) SQL, Erwin, Python, Scala Also listed: Snowflake, BigQuery, HBASE, Cassandra, Azure, Kinesis, TIBCO EMS, IBM MQ Series, MSK, Git, REST API, Web Services, NoSQL, ETL/ELT, and Agile frameworks (Scrum/Kanban/SAFe) Work setup Location : Atlanta, GA, US Schedule : Hybrid (two days in office, three days remote) Shift work : No On-call : Yes Weekend work : No Minimum education Bachelor’s Degree, preferably in Information Systems , Computer Science , Computer Information Systems , or a related technology field. #J-18808-Ljbffr