Azure Data Engineer

This will require an onsite interview or require the candidate to pick up their laptop in person


Requirements/Top Skills Needed:
ADF
Databricks
Python
SQL
Retail experience would be helpful, but not required


Job Title: Azure Data Engineer
Job Summary: We are seeking a highly skilled Azure Data Engineer to design, build, and optimize our next-generation cloud data platform. You will build scalable data pipelines, manage enterprise data warehouses, and transform massive datasets using Azure Cloud, ADF, Databricks, Spark, and PySpark.

Key Responsibilities
• Pipeline Orchestration: Design and implement scalable ETL/ELT pipelines using Azure Data Factory (ADF).
• Big Data Processing: Build high-performance, distributed data transformation workloads via Azure Databricks and Apache Spark/PySpark.
• Data Warehousing: Architect, model, and maintain cloud-native Data Warehouses and Data Lakes (ADLS Gen2).
• Architecture Implementation: Apply modern architectural patterns like the Medallion Architecture (Bronze, Silver, Gold layers) and Delta Lake formats.
• Performance Tuning: Optimize Spark jobs, cluster configurations, and SQL queries to minimize latency and cloud compute costs.
• Governance & Security: Implement security frameworks, data lineage, and user access policies using Unity Catalog.
• CI/CD Automation: Deploy, monitor, and automate workflows using Azure DevOps and automated alert configurations.

Required Technical Skills:
• Cloud Platform: Microsoft Azure Ecosystem (ADLS Gen2, Azure SQL, Synapse Analytics, Key Vault, Storages, Azure Functions, Logic Apps).
• Data Orchestration: Advanced proficiency in Azure Data Factory (ADF).
• Compute & Analytics: Deep hands-on experience with Azure Databricks.
• Big Data Frameworks: Strong mastery of Apache Spark and PySpark (including Spark SQL).
• Languages: Expert level in Python and SQL.
• Data Warehousing: Solid understanding of Data Modeling (Star/Snowflake schema) and warehousing concepts