Sr Data Engineer

***This position is only for Local Nearby Contractor***** We are seeking a Senior Databricks Engineer with deep hands-on experience designing and implementing large-scale data solutions on Azure Databricks . The ideal candidate has real-world experience building and troubleshooting production-grade data pipelines, optimizing Spark workloads, managing Delta Lake architecture, and implementing DevOps best practices using IaC and CI/CD automation. Key Responsibilities Design, develop, and maintain data pipelines and ETL solutions in Azure Databricks using PySpark and Delta Lake. Implement data integration frameworks and API-based ingestion using tools like Apigee or Kong . Analyze, design, and deliver enterprise data architecture solutions focusing on scalability, performance, and governance. Implement automation tools and CI/CD pipelines using Jenkins, Ansible, or Terraform. Troubleshoot production failures and performance bottlenecks — fix partitioning, caching, shuffle, cluster sizing, and Z-ordering issues. Manage Unity Catalog , enforce data security (row/column-level access), and maintain data lineage. Administer Databricks clusters, jobs, and SQL warehouses , optimizing costs through auto-stop, job clusters, and Photon usage. Collaborate with cross-functional teams to drive data strategy and standards across domains. Create and maintain detailed architectural diagrams , interface specs, and data flow documentation. Mentor junior engineers on Databricks, Spark optimization, and Azure data best practices. Required Skills & Experience 5+ years of experience as a Data Engineer with strong hands-on experience in Azure Databricks and PySpark . Solid understanding of Delta Lake , Z-ordering , partitioning , OPTIMIZE , and ACID transactions . #J-18808-Ljbffr Golden Technology