Azure Databricks Data Engineer

Azure Databricks Data EngineerDesign and build batch and streaming data pipelines using Microsoft Azure, Azure Databricks, and Kafka. Develop scalable ETL/ELT solutions and support Data Lake/Lakehouse architectures, data migration, integration, quality, governance, and performance optimization.Key ResponsibilitiesDesign and build batch and real-time data pipelines using Azure, Databricks, and Kafka.Develop ETL/ELT solutions using Spark, SQL, Python/PySpark, or Scala.Implement and support Data Lake/Lakehouse architectures.Integrate data from legacy and modern platforms; preferred experience with PostgreSQL migration to Azure Cloud.Perform data migration using APIs.Implement data quality checks and support data governance, including metadata, lineage, and access controls.Optimize Spark jobs for performance, reliability, and cost.Collaborate with Product Owners, Admin Teams, BSAs, Scrum Masters, Project Managers, and other cross-functional teams.Required Skills & Experience8+ years of Data Engineering experience with Azure and Azure Databricks.Strong experience with Azure Data Lake, Databricks, Azure Data Factory (ADF), and PostgreSQL.Strong Big Data Engineering experience with Cloudera.Hands-on expertise in SQL, Python, and PySpark or Scala.Proven experience with batch and real-time data ingestion.Strong experience with Kafka and Big Data processing.Good to HaveInformatica PowerCenter.Informatica IDMC or equivalent cloud data integration tools.SQL Server, Oracle, Azure SQL.Cosmos DB.Unix/Linux.Maestro or similar enterprise schedulers.Precisely or equivalent data tools.Azure DevOps (AzDO) and GitHub-based CI/CD.Practical use of Copilot for data engineering and development productivity.Exposure to Generative AI/GenAI use cases in data platforms and analytics.Added AdvantageInsurance domain knowledge, including Policy, Claims, Billing, Underwriting, Actuarial, and Regulatory data.Experience with PostgreSQL migration to Azure Cloud.Experience with SAP/enterprise data integration and migration using APIs.SkillsDigital: Microsoft Azure, Azure Databricks (ADB), Azure Data Factory (ADF), Kafka, Spark, SQL, Python, PySpark, Scala, PostgreSQL, Cloudera.Experience8+ years of relevant Data Engineering experience.About the companyAvance Consulting is a global leader in innovative talent solutions for diverse industries. Since 2007, we have been serving nearly a third of Fortune 500 companies and most of the top 100 technology companies worldwide.Our team of 700+ experts offers Engineering Services, Information Technology, Digital and Executive Search solutions from our offices in North America, UK, Europe and APAC.We understand the local markets and cultures and tailor our services to meet the needs of our clients. Our clients value our flexibility and agility, which enable us to deliver high-quality results at any scale across the globe.This allows us to work with numerous industries at varying scale across the globe while ensuring the excellent service standard that our clients have come to expect.At Avance you can learn new skills and technologies, have more advancement opportunities and career growth, being a part of our productive atmosphere, and a positive reinforcement culture.