Senior Databricks Developer

Senior Databricks Developer

Location :Remote/Hybrid

Job Summary

We are seeking a Senior Databricks Developer with strong expertise in Databricks, Python, and cloud-based data engineering, preferably within the healthcare domain. The ideal candidate will have extensive experience building scalable data pipelines, developing cloud-native analytics solutions, and working in Agile delivery environments. Experience with AWS and healthcare data platforms is highly desirable.

Key Responsibilities

· Design, develop, and optimize large-scale data processing solutions using Databricks.
· Build and maintain robust ETL/ELT pipelines using Python and Databricks.
· Develop scalable data ingestion, transformation, and integration frameworks from multiple data sources.
· Leverage Databricks features including Delta Lake, Spark SQL, Unity Catalog, Workflows, and ML capabilities.
· Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
· Work with AWS services to support cloud-based data platforms and analytics workloads.
· Implement data quality checks, monitoring, performance tuning, and operational support processes.
· Participate in architecture discussions and contribute to best practices for data engineering and cloud adoption.
· Ensure compliance with security, governance, and data management standards.
· Actively participate in Agile ceremonies including sprint planning, backlog grooming, daily standups, sprint reviews, and retrospectives.
Required Skills & Qualifications

Mandatory Skills

· 7+ years of overall IT experience in Data Engineering and Analytics.
· Strong hands-on experience with Databricks development and administration.
· Strong Python programming skills for data processing, automation, and analytics.
· Experience with Apache Spark (PySpark) and distributed data processing.
· Strong experience in designing and developing ETL/ELT pipelines.
· Experience working with structured and unstructured data.
· Hands-on experience with Git version control and CI/CD practices.
· Strong understanding of data modeling, data warehousing, and performance optimization.
· Mandatory experience working in Agile/Scrum delivery models.
· Excellent analytical, problem-solving, and communication skills.

Preferred Skills

· Working knowledge of AWS services such as:

o S3
o Glue
o Lambda
o EC2
o IAM
o CloudWatch
o Redshift (preferred)
· Experience with healthcare data ecosystems.
· Knowledge of FHIR, HL7, Claims, Membership, Provider, or Clinical data domains is an advantage.
· Exposure to data governance, data quality, and security frameworks.

Preferred Experience
· Healthcare/Payer/Provider industry experience is a strong advantage.
· Experience supporting enterprise-scale cloud data platforms.
· Exposure to DevOps and DataOps practices.
· Experience mentoring junior developers and leading technical workstreams.

Education
· Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.

Nice to Have
· Databricks Certifications.
· AWS Cloud Certifications.
· Healthcare domain certifications or relevant project experience.