Data Engineer 3

Role Description

Looking for a skilled Data Engineer with expertise in SQL and Python experienced in building large-scale data pipelines using Apache Spark and designing robust data architectures on AWS. The ideal applicant will have hands-on experience and curiosity to learn and develop skills in data lake architectures, open table formats (Delta Lake), and modern data platforms delivery self-service analytics for a large number of users. If you are a problem solver, a data platform enthusiast, and someone who thrives in fast-paced environments, this is an opportunity for you!

Responsibilities

• Architect, design, and build high-performance, scalable data pipelines that support large-scale data processing needs.
• Develop complex Big Data ETL workflows using modern frameworks such as Databricks Delta Live Tables (DLT).
• Design and implement robust data lake architectures, including open table formats such as Delta Lake and Apache Iceberg.
• Define, promote, and enforce data engineering best practices to ensure data quality, integrity, reliability, and performance.
• Lead the implementation of observability and monitoring across pipelines, including data cataloging, lineage, and metadata management.
• Contribute to peer reviews, knowledge sharing, production support, incident resolution, and change management activities.
• Adhere to code management and version control best practices across all development workflows.
• Produce high-quality documentation including data flows, lineage diagrams, and architecture details, aligned to data engineering documentation standards.
• Ensure comprehensive code coverage through the creation of unit and integration tests, following test-driven development (TDD) principles.
• Own critical components of large-scale batch ETL systems and clearly articulate data models and design decisions to Product, Engineering, and Leadership stakeholders.
• Manage multiple priorities effectively, balancing long-term architectural direction with immediate business needs.
• Build advanced semantic layers and support the creation of customer-facing self-service analytics solutions.
• Exercise independent judgment and discretion in handling complex and impactful technical decisions.
• Maintain regular, consistent, and punctual attendance, with flexibility to work nights, weekends, or variable schedules as required.
• Demonstrate strong understanding of semantic layer technologies such as Looker, Tab, Snowflake, and Databricks.
• Perform additional duties and responsibilities as assigned.

Qualifications

• Experience in Data Engineering.
• Proficient in Python, PySpark.
• Good grasp on SQL.
• Hands-on experience in Databricks and/or Snowflake is preferred.
• Understanding of Cloud-based architectures (AWS).
• Problem-solving skills.
• Certifications: AWS Certified Developer Associate, Databricks Certified Data Engineer Associate, SnowPro Advanced: Data Engineer.

Requirements

• Designing and constructing data architectures and pipelines to standardize, transform, and ensure the integrity of data for business insights.
• Guaranteeing data quality throughout the ingestion, processing, and loading phases, with a focus on maintaining high standards.
• Engineering robust ingestion frameworks for a variety of data types, while monitoring and reporting on data quality.
• Developing accessible methods for data consumption, including APIs, database views, and data extracts for user applications.
• Implementing data solutions on multiple platforms such as Kubernetes, Teradata, and cloud services like AWS and Databricks.
• Selecting appropriate data storage platforms based on data sensitivity, privacy requirements, and access needs.
• Mastering data lineage and applying transformation rules for effective change management and issue resolution.
• Collaborating with cross-functional teams to refine data sourcing and processing, ensuring optimal data quality and efficiency.
• Consistent exercise of independent judgment and discretion in matters of significance.
• Regular, consistent and punctual attendance. Must be able to work nights and weekends, variable schedule(s) as necessary.
• Other duties and responsibilities as assigned.

Benefits

• Comcast provides best-in-class benefits to eligible employees.
• An array of options, expert guidance, and always-on tools to support you physically, financially, and emotionally.

Education

• Bachelor's Degree preferred, or a combination of coursework and experience, or extensive related professional experience.

Relevant Work Experience

• 5-7 Years.

Compensation

• National Pay Range: $78,966.34 USD - $185,077.36 USD.
• Illinois Pay Range: $83,901.74 USD - $162,868.08 USD.
• Colorado Pay Range: $88,837.13 USD - $170,271.17 USD.
• Hawaii Pay Range: $103,643.32 USD - $155,464.99 USD.
• Washington DC Pay Range: $113,514.12 USD - $170,271.17 USD.
• Maryland Pay Range: $93,772.53 USD - $170,271.17 USD.
• Minnesota Pay Range: $88,837.13 USD - $155,464.99 USD.
• New York Pay Range: $93,772.53 USD - $185,077.36 USD.
• Washington Pay Range: $88,837.13 USD - $177,674.27 USD.
• New Jersey Pay Range: $98,707.93 USD - $177,674.27 USD.
• Vermont Pay Range: $93,772.53 USD - $148,061.89 USD.
• Massachusetts Pay Range: $98,707.93 USD - $177,674.27 USD.
• Virginia Pay Range: $88,837.13 USD - $170,271.17 USD.
• Maine Pay Range: $88,837.13 USD - $148,061.89 USD.
• California Pay Range: $88,837.13 USD - $164,513.21 USD.