Senior Databricks Data Engineer

Databricks Data Engineer

About the Role:

We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize our next-generation data architecture. In this role, you will be instrumental in modernizing our data infrastructure, transitioning legacy ETL workloads to highly scalable, cloud-native solutions, and ensuring our data assets are secure, discoverable, and performant. You will leverage the latest capabilities of the Databricks platform—from declarative pipelines to advanced governance—to deliver high-quality data products to the business.

Key Responsibilities

• Pipeline Engineering: Design, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Spark SQL to process large volumes of structured and semi-structured data.
• Declarative Frameworks: Implement and manage Spark Declarative Pipelines (Delta Live Tables) to simplify pipeline development, automate data quality checks, and streamline operations.
• Legacy Modernization: Lead the migration of complex workloads from legacy enterprise ETL systems into modernized, scalable PySpark architectures.
• Data Governance & Security: Architect and enforce centralized data governance, access controls, auditing, and data lineage tracking across the organization utilizing Unity Catalog.
• CI/CD & Automation: Automate the deployment lifecycle of data pipelines, notebooks, and infrastructure using Databricks Asset Bundles (DABs) integrated with enterprise CI/CD workflows.
• Performance Optimization: Profile, tune, and optimize complex PySpark jobs and Spark SQL queries for maximum performance and cost-efficiency.

Required Skills & Qualifications

Experience: 5+ years of experience in Data Engineering, with a heavy focus on the Databricks Data Intelligence Platform.

Core Languages: Expert-level proficiency in Python (PySpark) and advanced Spark SQL.

Databricks Ecosystem: Deep hands-on experience with modern Databricks features, specifically Spark Declarative Pipelines and Unity Catalog for centralized governance.

DevOps / DataOps: Proven ability to implement CI/CD pipelines for data assets using Declarative Asset Bundles (DABs), Git, and automation tools (e.g., GitHub Actions, Jenkins, or GitLab CI).

Architecture & Migration: Strong understanding of distributed computing principles and experience migrating legacy on-premise ETL logic (e.g., DataStage, Informatica) to cloud-native Spark environments.

Problem Solving: Strong analytical skills with the ability to troubleshoot complex data processing issues and optimize massive data transformations.

Nice-to-Haves

• Experience with event-driven data ingestion architectures and storage-layer triggers (e.g., S3).
• Familiarity with cloud infrastructure (AWS, Azure, or GCP) and infrastructure-as-code (Terraform).
• Experience building and designing internal frameworks to accelerate data pipeline development.