Azure Databricks Data Engineer

Job Description

Must-Have Technical and Functional Skills

● Strong working knowledge of relevant Azure services.

● Strong knowledge of data warehouse concepts.

● Strong hands-on proficiency in working with data using SQL.

● Ability to write complex queries aligned with business requirements.

● Experience developing data pipelines in Databricks.

● Prior experience with a programming language, preferably Python.

● Hands-on experience using version control systems such as Git.

● Experience creating project plans, technical design documents, and training documents,

and presenting solutions to end users.

● Strong verbal and written communication skills, with the ability to express complex

technical concepts in business terms and translate complex business rules into technical

designs.

Roles and Responsibilities

● Strong knowledge of data warehousing and RDBMS platforms such as SQL Server,

Teradata, and Oracle.

● Hands-on experience with Python, Java, Airflow, and related technologies.

● Experience building and optimizing complex data pipelines in Azure or an equivalent cloud

platform.

● Strong working knowledge of Azure components and services such as Storage Accounts,

Azure SQL Database, Azure Data Factory, Microsoft Entra ID (formerly Azure Active

Directory), and Azure Synapse Analytics.

● Experience with Azure SQL Database deployment and administration is desirable.

● Knowledge of Azure networking services.

● Knowledge of Azure infrastructure creation using ARM templates.

● Proficiency with GitLab and Azure DevOps.

● Implement data pipelines to automate the ingestion, transformation, and augmentation of

data sources, and apply best practices for pipeline operations.

● Identify downstream implications of data loads and migrations, including data quality and

regulatory considerations.

● Monitor workloads, ensure optimal utilization of SQL scripts, document database

requirements, and analyze data anomalies.

● Work directly with business users and analysts to understand business requirements.

Generic Managerial Skills

● Ability to collaborate effectively across multiple teams.