Senior Azure Databricks & PySpark Data Engineer / Architect

Senior Azure Databricks & PySpark Data Engineer / Architect

Location: Greenfield, Indiana (Onsite)

Employment Type: Full-Time (FTE) OR (CONTRACT TO HIRE)

Experience: 10–20 Years of Overall IT Experience

Visa: No visa restrictions. Independent visa candidates preferred.

Job Overview

We are seeking an experienced Senior Azure Databricks & PySpark Data Engineer / Architect to design, develop, and implement enterprise-scale Azure Data Engineering solutions.

The ideal candidate must have strong hands-on experience with Azure Databricks, PySpark, Azure Data Factory, and Azure Cloud services, along with expertise in modern Lakehouse architecture and DevOps practices.

The candidate will be responsible for building scalable data engineering solutions, developing data pipelines, implementing cloud-native technologies, and driving engineering best practices within healthcare-focused enterprise environments.

Key Responsibilities

• Design, develop, and maintain enterprise-scale Azure Data Engineering solutions.
• Build and optimize data pipelines using Azure Databricks, PySpark, and Azure Data Factory.
• Develop and maintain modern Lakehouse architectures using Azure Synapse and Databricks.
• Implement and manage Databricks workflows and Unity Catalog.
• Build scalable data processing solutions using ADLS Gen2 and Azure SQL.
• Develop Python packages and support package builds.
• Implement DevOps practices and CI/CD processes for data engineering solutions.
• Develop and maintain data architecture and low-level design documents.
• Collaborate with internal engineering teams and customers to deliver high-quality solutions.
• Follow standard coding principles, guidelines, and engineering best practices.
• Lead technical decisions, mentor team members, and drive engineering improvements.

Mandatory Technical Skills

• 4+ years of hands-on experience with Azure Databricks and PySpark.
• 2+ years of experience with Databricks Workflows and Unity Catalog.
• 3+ years of experience with Azure Data Factory (ADF).
• 3+ years of experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• 3+ years of experience with Azure SQL.
• 2+ years of experience with Python programming and package builds.
• 5+ years of experience with the Azure Cloud platform.
• Strong understanding of Azure Synapse and Databricks Lakehouse architecture.
• Strong communication and collaboration skills.

Preferred / Nice-to-Have Skills

• Change Data Capture (CDC) solutions, preferably Debezium.
• Azure Entra ID / Active Directory.
• GitHub Actions.
• Workflow orchestration tools such as Airflow, Dagster, and Azure Logic Apps.
• Event-driven architecture using Kafka and Azure Event Hubs.
• Google Cloud Pub/Sub.
• Google Cloud Storage (GCS).
• Experience managing cloud storage solutions on Azure Data Lake Storage.

Required Qualifications

• 10–20 years of overall IT experience.
• Strong hands-on Azure Data Engineering background.
• Proven experience developing enterprise-scale data pipelines and Lakehouse solutions.
• Ability to prepare low-level design documents and follow coding standards.
• Strong technical communication and collaboration skills.
• Ability to take ownership of solution design and technical decisions.

Work Arrangement

• This is a 100% onsite position in Greenfield, Indiana.
• Candidates must be willing to work onsite.
• Relocation willingness should be confirmed before submission.

Application Instructions

Interested candidates are requested to share:

• Updated resume.
• Total IT experience.
• Exact years of experience in each mandatory technical skill.
• Current location and relocation willingness to Greenfield, IN.
• Visa / Work Authorization.
• Current salary and expected salary.
• Contact number and email address.