Data Engineer - Databricks (Microsoft Azure)

Required Skills & Experience Required Qualifications \n\nBachelor's degree in computer science, Information Systems, Engineering, or related field. \n\n4–6 years of experience in ETL, database development, or data engineering. \n\n4+ years of advanced SQL development and query optimization. \n\n3+ years of experience developing data pipelines using Databricks on Microsoft Azure. \n\nExperience with Databricks SQL, Spark SQL, PySpark, Delta Lake, and Azure Data Lake Storage (ADLS). \n\nExperience migrating legacy ETL processes from Oracle or other relational databases to cloud-based data platforms. \n\nExperience with Git, source control, and CI/CD deployment processes. \n\nStrong analytical, troubleshooting, and problem-solving skills. \n\nExcellent communication skills and the ability to work independently and collaboratively. Nice to Have Skills & Experience Preferred Qualifications \n\nExperience in the healthcare, biomedical research, or clinical data domain. \n\nExperience converting Oracle PL/SQL packages, procedures, and functions into Databricks pipelines. \n\nExperience with Azure Data Factory, Databricks Workflows, or similar orchestration tools. \n\nKnowledge of dimensional data modeling and enterprise data warehousing. \n\nExperience with Python and modern software engineering practices. \n\nExperience with Power BI, Tableau, or similar reporting tools. \n\nFamiliarity with machine learning or AI-enabled data engineering is a plus. Job Description Position Summary \n\nWe are seeking a highly motivated Data Engineer with experience in Databricks on Microsoft Azure to join a fast-paced team responsible for modernizing enterprise data platforms supporting healthcare research and analytics. \n\nThe primary responsibility of this role is to migrate existing Oracle PL/SQL-based ETL processes to Databricks data pipelines. The successful candidate will analyze existing ETL workflows, redesign them using modern cloud technologies, and develop scalable, high-performance data pipelines while ensuring data quality and reliability. The individual will also support existing production ETL processes during the migration. \n\nKey Responsibilities \n\nWork with business users and stakeholders to gather requirements and translate them into technical solutions. \n\nAnalyze existing Oracle SQL and PL/SQL ETL processes and migrate them to Databricks using Spark SQL and/or PySpark. \n\nDesign, develop, test, deploy, and maintain scalable data pipelines on Microsoft Azure Databricks. \n\nCreate and maintain technical design, data mapping, and system documentation. \n\nValidate migrated data with business users and subject matter experts to ensure business-level accuracy \n\nImplement incremental data processing strategies using Databricks and Delta Lake. \n\nAnalyze and convert complex Oracle PL/SQL transformation logic (packages, procedures, functions) into scalable Databricks pipelines. \n\nPerform data validation and reconciliation between source (Oracle) and target (Databricks) to ensure data accuracy and completeness during migration. \n\nProvide ongoing support and maintenance for ETL pipelines post-migration in Databricks, including monitoring, troubleshooting, and enhancements. \n\nOptimize pipeline performance, troubleshoot production issues, perform root cause analysis, and implement solutions. \n\nParticipate in source control, CI/CD, release management, and production support activities. \n\nTake ownership of assigned projects, manage priorities, and deliver high-quality solutions on schedule. Insight Global