Data Engineer

Databricks Data EngineerAbout the Role:We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize our next-generation data architecture. In this role, you will be instrumental in modernizing our data infrastructure, transitioning legacy ETL workloads to highly scalable, cloud-native solutions, and ensuring our data assets are secure, discoverable, and performant. You will leverage the latest capabilities of the Databricks platform—from declarative pipelines to advanced governance—to deliver high-quality data products to the business.Key ResponsibilitiesPipeline Engineering: Design, build, and maintain robust, scalable ETL/ELT pipelines using PySpark and Spark SQL to process large volumes of structured and semi-structured data.Declarative Frameworks: Implement and manage Spark Declarative Pipelines (Delta Live Tables) to simplify pipeline development, automate data quality checks, and streamline operations.Legacy Modernization: Lead the migration of complex workloads from legacy enterprise ETL systems into modernized, scalable PySpark architectures.Data Governance & Security: Architect and enforce centralized data governance, access controls, auditing, and data lineage tracking across the organization utilizing Unity Catalog.CI/CD & Automation: Automate the deployment lifecycle of data pipelines, notebooks, and infrastructure using Databricks Asset Bundles (DABs) integrated with enterprise CI/CD workflows.Performance Optimization: Profile, tune, and optimize complex PySpark jobs and Spark SQL queries for maximum performance and cost-efficiency.Required Skills & QualificationsExperience: 5+ years of experience in Data Engineering, with a heavy focus on the Databricks Data Intelligence Platform.Core Languages: Expert-level proficiency in Python (PySpark) and advanced Spark SQL.Databricks Ecosystem: Deep hands-on experience with modern Databricks features, specifically Spark Declarative Pipelines and Unity Catalog for centralized governance.DevOps / DataOps: Proven ability to implement CI/CD pipelines for data assets using Declarative Asset Bundles (DABs), Git, and automation tools (e.g., GitHub Actions, Jenkins, or GitLab CI).Architecture & Migration: Strong understanding of distributed computing principles and experience migrating legacy on-premise ETL logic (e.g., DataStage, Informatica) to cloud-native Spark environments.Problem Solving: Strong analytical skills with the ability to troubleshoot complex data processing issues and optimize massive data transformations.Nice-to-HavesExperience with event-driven data ingestion architectures and storage-layer triggers (e.g., S3).Familiarity with cloud infrastructure (AWS, Azure, or GCP) and infrastructure-as-code (Terraform).Experience building and designing internal frameworks to accelerate data pipeline development.