Databricks Engineer (Chicago)

Position Summary

We are seeking an experienced Databricks Lead Engineer / Technical Lead to drive large-scale data platform modernization initiatives. The ideal candidate will possess deep expertise in Databricks Lakehouse architecture , Delta Lake , Databricks Workflows , and hands-on experience migrating enterprise data platforms from Azure Synapse and SQL Server to Databricks . This role requires strong technical leadership, client-facing communication skills, and the ability to guide cross-functional teams through complex data transformation projects.

Key Responsibilities

• Lead the design, development, and implementation of enterprise-scale data solutions using Databricks Lakehouse Platform .
• Design and build scalable data pipelines utilizing Databricks Workflows , Delta Lake, and Spark-based processing frameworks.
• Drive end-to-end migration of data warehouses, ETL pipelines, and reporting workloads from Azure Synapse and SQL Server to Databricks.
• Collaborate with Data Architects, Business Stakeholders, and Delivery Teams to define migration strategies and target-state architectures.
• Configure and manage Unity Catalog , including catalogs, schemas, tables, governance policies, and access controls.
• Ensure adherence to data governance, security, and compliance requirements across the Databricks ecosystem.
• Provide technical leadership, solution guidance, and mentorship to offshore and onshore development teams.
• Act as the primary technical point of contact for client stakeholders, facilitating discussions, solution reviews, and project planning.
• Conduct code reviews, performance tuning, and optimization of Databricks workloads.
• Support deployment automation and platform standardization initiatives.

Required Skills & Qualifications

Technical Skills

• Strong hands-on experience with:
• Databricks Workflows
• Databricks Lakehouse Architecture
• Delta Lake
• Apache Spark (PySpark/Scala preferred)
• Proven experience migrating:
• Azure Synapse Analytics workloads to Databricks
• SQL Server ETL processes to Databricks
• Enterprise data warehouses and data pipelines
• Expertise in designing and implementing scalable ETL/ELT frameworks.
• Hands-on experience with Unity Catalog , including:
• Catalog and schema management
• Table creation and management
• Role-based access control (RBAC)
• Data governance implementation
• Strong understanding of cloud-based data engineering principles and best practices.

Leadership & Stakeholder Management

• Experience serving as an Onshore Technical Lead for data engineering programs.
• Strong client-facing communication and stakeholder management skills.
• Ability to work closely with:
• Client Business Stakeholders
• Data Architects
• Program Managers
• Delivery Leads
• Experience leading distributed teams across onshore and offshore locations.

Good to Have Skills

• Experience with Databricks Asset Bundles (DABs) .
• Familiarity with Azure DevOps and CI/CD implementation for Databricks deployments.
• Understanding of Infrastructure-as-Code and deployment automation practices.
• Knowledge of retail, restaurant, or hospitality operational data systems such as:
• Toast
• HotSchedules
• Crunchtime
• Experience with modern data governance and data quality frameworks.

Preferred Experience

• 8+ years of overall Data Engineering experience.
• 4+ years of hands-on Databricks development and solution design experience.
• Prior experience leading cloud data migration and modernization programs.
• Experience working in Agile/Scrum delivery environments.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.