Senior Databricks Data Engineer (Python, Spark, SQL, Delta Lake, Lakeflow, ETL/ELT)

Skills:

Databricks, Apache Spark, SQL, Python, Scala, ETL, ELT, Delta Lake, Lakehouse Architecture, Medallion Architecture, Lakeflow Declarative Pipelines (DLT), Lakeflow Jobs, Databricks Workflows, Data Warehousing, Star Schema, Snowflake Schema, Databricks SQL, Databricks Apps, Data Governance, Data Quality, Data Security, Apache Airflow, Git, CI/CD, Azure DevOps, SDLC, Agile, Cloud Data Engineering, Public Sector,

 

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark .
• Build and optimize enterprise data solutions utilizing Delta Lake and Lakehouse Architecture .
• Design and implement Bronze, Silver, and Gold (Medallion) data layers .
• Develop and maintain Lakeflow Declarative Pipelines (formerly Delta Live Tables - DLT) for production-grade data processing.
• Create, schedule, and manage workflows using Lakeflow Jobs (formerly Databricks Workflows) or comparable orchestration tools such as Apache Airflow .
• Design and maintain enterprise data warehouse solutions using dimensional modeling techniques, including Star Schema and Snowflake Schema .
• Develop high-performance SQL queries and optimize Spark jobs for large-scale data processing.
• Build dashboards and applications using Databricks SQL Dashboards and Databricks Apps .
• Implement enterprise data governance , data quality , data security , and metadata management best practices.
• Monitor and optimize data pipeline performance, scalability, and reliability.
• Implement CI/CD pipelines for data engineering solutions using Git-based workflows and DevOps practices.
• Collaborate with architects, business analysts, and stakeholders to understand business requirements and translate them into technical solutions.
• Troubleshoot production issues and provide ongoing support for enterprise data platforms.
• Document architecture, data flows, pipeline designs, and operational procedures.
• Present technical solutions and project updates to both technical and business stakeholders.
• Follow software development lifecycle (SDLC) and Agile development methodologies.

Required Qualifications

• Strong experience supporting the design, development, deployment, and delivery of enterprise technology solutions.
• Extensive experience with Databricks and Apache Spark .
• Experience building and optimizing ETL/ELT data pipelines.
• Strong experience with Delta Lake and Lakehouse Architecture .
• Experience implementing Lakeflow Declarative Pipelines (Delta Live Tables/DLT) .
• Experience creating and scheduling production jobs using Lakeflow Jobs (Databricks Workflows) or similar orchestration tools.
• Strong knowledge of SQL and Python (Scala experience is a plus).
• Experience with data warehousing and dimensional data modeling (Star and Snowflake schemas).
• Experience designing Databricks SQL Dashboards and Databricks Apps .
• Strong understanding of data governance , data quality , data security , and compliance best practices.
• Experience implementing CI/CD processes for data engineering projects using Git-based workflows.
• Excellent troubleshooting, analytical, and problem-solving skills.
• Strong verbal and written communication skills.
• Ability to collaborate effectively with cross-functional teams.

Preferred Qualifications

• Experience working in public sector or state government environments.
• Databricks Certified Data Engineer Associate or Professional Certification .
• Experience with Azure Databricks , AWS , or Google Cloud Platform (GCP) .
• Experience using orchestration tools such as Apache Airflow .
• Experience with Agile/Scrum development methodologies.
• Familiarity with enterprise data governance and regulatory compliance standards.