Big Data Specialist

We are seeking a highly skilled and experienced Databricks SME to lead the architecture, design, and implementation of enterprise‑scale Big Data solutions on the Databricks Lakehouse Platform. The ideal candidate will have at least 12 years of hands‑on experience architecting big data solutions with Databricks, with proven expertise in migrating data and pipelines from legacy platforms like Alteryx, handling streaming data, and implementing CI/CD practices for data pipelines. This role is critical for driving data modernization initiatives, ensuring robust data governance, and enabling scalable data ingestion and transformation across multiple source systems into the Databricks ecosystem. Key Responsibilities Architect enterprise Big Data solutions on Databricks using Bronze/Silver/Gold (Medallion) layers with Delta Lake and Delta Live Tables Migrate data and pipelines from legacy platforms (Alteryx, Informatica, DataStage, Talend etc.) to Databricks, including source‑to‑target mapping and data validation Design streaming data pipelines using Structured Streaming, Auto Loader, and Delta Live Tables Implement CI/CD practices for data pipelines using Git, Azure DevOps, and GitHub Actions Driving Innovation, Empowering Insights Develop data governance framework using Unity Catalog ABAC/RBAC policies and collaborate with stakeholders to translate requirements into scalable solutions Experience with Databricks Workflows, Jobs, and Databricks Asset Bundles for deployment automation Mentor junior team members and provide technical leadership across data engineering projects Required Skills 12+ years architecting Big Data solutions with Databricks Lakehouse Platform Proven experience migrating pipelines from Alteryx or similar legacy ETL tools Hands‑on expertise with Delta Live Tables, Delta Lake, and Medallion architecture Strong experience building streaming pipelines with Structured Streaming and Auto Loader Demonstrated ability implementing CI/CD for data workloads Unity Catalog Data governance framework with ABAC / RBAC policies Proficient in Python, SQL, PySpark, and strong experience in performance tuning of large‑scale data Proven experience on at least one major cloud platform — Azure (preferred) or AWS — and its native data services Preferred Skills Unity Catalog for data governance and lineage Data quality tools: Great Expectations, Informatica DQ, Collibra Data modeling tools: Erwin, dbt, ER/Studio AI/ML experience with GenAI, LLMs, MLOps Certifications (Preferred) Databricks Certified Data Engineer Associate/Professional #J-18808-Ljbffr