Senior Data Engineer

Senior Data Engineer (Databricks / PySpark / Python / SQL)

The Sr. Data Engineer will be responsible for the design and development of data pipelines to support System Integrations as well as data pipelines that power our Analytical Platforms. In collaboration with Product Owners and other technical teams, the Sr. Data Engineer will lead significant enterprise-wide work as part of larger cross-functional teams supporting multiple major project initiatives.

A candidate will deliver scalable, flexible, and high-quality solutions that conform to Client's data design and governance strategies, leveraging our cloud data platform built on Snowflake or Databricks along with other modern cloud-based technologies.

Responsibilities Include But Not Limited To:

• Design and develop data pipelines and integrations that are performant, scalable and flexible.
• Work with project teams to deliver system integration pipelines, ensuring a high degree of reliability and resiliency.
• Work with business users and project teams to provide SME guidance, finalize pipeline requirements and develop level of effort estimates.
• Develop automated testing and deployment scripts to support integrations and pipelines.
• Provide database support by coding utilities, responding to user questions, and resolving problems.
• Create and maintain documentation of data pipelines including data flow and data lineage documentation.
• Responsible for tier 3 support and assisting our Operations Team as required to provide a great customer experience for our users.
• Responsible for overall data quality as it relates to our Enterprise Data Warehouse and associated Data Marts. Provide data analysis as required to troubleshoot data issues.
• Assist with integration related code reviews and mentorship of junior developers.
• Implement data governance and master data management principles as part of data pipeline development and delivery.

Qualifications (Some Examples Listed Below):

• Lakehouse Architecture: Hands-on experience building lakehouse solutions on Databricks (Delta Lake, Unity Catalog) or Snowflake (Iceberg Tables, Horizon Catalog).
• Medallion Architecture: Proven ability to design Bronze/Silver/Gold layers for curated, analytics-ready data.
• Streaming or Batch Processing: Experience implementing Lambda or Kappa architectures using Databricks Structured Streaming / DLT or Snowflake Dynamic Tables / Snowpipe Streaming.
• Semantic Layer: Experience building and managing semantic models using Databricks AI/BI Genie or Unity Catalog Metrics or Snowflake Semantic Views / Cortex Analyst to enable consistent, governed metrics for BI and AI consumption.
• ETL/ELT Orchestration: Building pipelines with Airflow, Databricks Lakeflow, or Snowflake Openflow; familiarity with dbt is a plus.
• Programming: Strong PySpark or advanced SQL skills; Python for data engineering or automation.
• Data Modeling: Dimensional modeling, Data Vault, or schema design for analytical workloads.
• Performance Tuning: Cluster/warehouse sizing, partitioning, clustering keys, Z-ordering, or query optimization.
• Governance & Security: Working knowledge of Unity Catalog (Databricks) or Horizon Catalog (Snowflake) for lineage, access control, or data quality.
• CI/CD & DevOps: Git, Databricks Asset Bundles or Snowflake CLI/Schemachange, automated testing, or deployment pipelines.
• Cloud Ecosystem: AWS (S3, Glue, Kinesis), Azure, or GCP services supporting modern data platforms.
• AI/ML Enablement: Familiarity with Databricks Mosaic AI or Snowflake Cortex for GenAI or ML use cases.
• Collaboration: Strong communication skills to partner with Product Owners, Analysts, or cross-functional engineering teams.
• Education: Bachelor's degree in Computer or Information Science or related field, or equivalent combination of education and experience.
• Experience: 5–8 years creating quality data pipelines and system integrations with at least 3 years of experience in a cloud environment.

Must Have

• Strong hands-on experience building data pipelines with Databricks, Apache Spark (PySpark), and Delta Lake.
• Advanced SQL and Python, with strong data modeling skills across dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or managing data hygiene when porting old systems.
• Strong problem-solving, collaboration, and communication skills, including mentoring junior engineers and explaining data concepts to non-technical stakeholders.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or an equivalent transformation framework.
• Familiarity with secure coding standards and industry security best practices.

Nice To Have

• Experience with Infrastructure as Code (IaC) using Terraform and CI/CD using Azure Devops.
• Experience working with relational databases (specifically PostgreSQL) and data persistence concepts.
• Familiarity with logging and monitoring tools (e.g., Dynatrace, CloudWatch, Databricks system tables).