Senior Databricks Data Engineer

We're looking for a Senior Databricks Data Engineer to support a cloud modernization initiative where legacy SQL Server/SSIS pipelines are being rebuilt on a modern Databricks Lakehouse platform. This is a hands-on engineering role with the opportunity to influence standards, build net-new pipelines, and help shape a scalable cloud data environment.


Key experience we're seeking:
• Hands-on Databricks development (SQL + PySpark)
• Strong data warehousing fundamentals (facts, dimensions, modeling)
• Experience modernizing or migrating ETL pipelines
• Cloud experience (Azure or AWS)
• Comfort working in an early-phase, fast-evolving environment

Senior Data Engineer (Databricks / Cloud Migration)

Summary

This role supports an early-phase cloud modernization initiative focused on retiring legacy SQL Server/SSIS pipelines and building a new enterprise data platform in Databricks. The engineer will be hands-on in Databricks development and must have a strong understanding of data warehouse concepts, dimensional modeling, and scalable data transformation patterns. While Databricks is the primary skill, SQL and Python remain foundational components of the work.

Responsibilities

Develop, optimize, and maintain data pipelines within Databricks using SQL, PySpark, and Delta Lake


Translate existing SQL Server/SSIS logic into Databricks-based modern cloud architecture


Apply data warehouse concepts (star schemas, slowly changing dimensions, fact/dimension modeling) to pipeline and table design


Build scalable, reliable ETL/ELT processes to support analytics, reporting, and downstream systems


Participate in cloud migration activities and support integration across data, BI, and application teams


Partner with data architects and engineering leads to design and implement Lakehouse solutions


Contribute to standards for coding, documentation, testing, and performance optimization


Support data validation and QA processes to ensure accuracy during and after migration


Participate in ongoing development phases as the platform expands to new domains and workloads

Required Skills

Hands-on experience with Databricks (not just exposure)


Strong understanding of data warehousing concepts including dimensional modeling


Proficiency with SQL for transformations, performance tuning, and pipeline logic


Experience with Python and/or PySpark in a data engineering environment


Experience migrating or modernizing legacy ETL processes (SSIS experience a plus)


Background in building scalable pipelines in cloud environments (Azure or AWS)


Ability to work in an early-stage environment with evolving requirements


Strong communication skills and ability to collaborate with architecture, BI, and integration teams

Nice to Have

Experience with Delta Live Tables or Databricks Workflows


Experience with Power BI or similar BI tools


Familiarity with DevOps practices for data (CI/CD, Git, testing frameworks)


Experience supporting data validation or automated QA frameworks


Exposure to professional services, legal, or real estate environments (not required)

Critical Success Factors

Comfortable working hands-on and iteratively as the cloud platform matures


Ability to interpret legacy ETL logic and redesign it for Databricks and Spark


Strong grounding in SQL and Python as core languages within Databricks


Ability to deliver high-quality pipelines with accuracy, reliability, and performance


Collaborative mindset and willingness to work across multiple technical teams