Azure Databricks Lead

Job Title

Responsibilities

Data Pipeline Design and Development

Designing, developing, and maintaining scalable data pipelines and ETL processes using Azure Databricks Data Factory and other Azure services

Spark Job Implementation and Optimization

Implementing and optimizing Spark jobs, data transformations, and workflows within Databricks

Data Model Development

Developing and maintaining data models and data dictionaries

Data Quality and Governance

Developing and maintaining data quality checks, governance policies, and security procedures

ETL Process Design and Maintenance

Designing and creating ETL processes to supply data to various destinations including data warehouses

Data Integration

Integrating data from various sources into Azure Databricks

Collaboration

Working with other teams including data engineers, data scientists, and analysts to ensure data quality and consistency

Monitoring and Optimization

Implementing effective monitoring processes to track performance and optimize workflows

Data Lakehouse Solutions

Contributing to the design and implementation of data lakehouse solutions using Databricks

Skills and Qualifications

Azure Databricks Proficiency Experience with Azure Databricks, PySpark, and Spark

Programming Languages Proficiency in languages like Python, SQL, and Scala

ETL and Data Warehousing Strong understanding of ETL processes, data warehousing concepts, and data modeling

Cloud Platforms Experience with cloud platforms particularly Microsoft Azure

Data Engineering Experience in data engineering, data pipelines, and data integration

Data Governance and Security Knowledge of data governance policies and procedures

Problemsolving and Debugging Excellent problemsolving and debugging skills

Communication and Teamwork Strong communication and teamwork skills