Databricks Data Engineer

Databricks Data EngineerThe Databricks Data Engineer will be part of the Data Services team and help transform the delivery of data driven insights at scale. In this role, they will design and engineer robust data pipelines using technologies like Databricks, Azure Data Factory, Apache Spark, and Delta Lake. This role will work hands on crafting healthcare data solutions - processing massive healthcare datasets, optimizing performance, and ensuring our data is accurate, secure, and accessible when it matters most.Essential FunctionsTranslate business requirements into technical specifications and document solution designs, data flows and architectureDesign, develop, and maintain ETL/ELT pipelines using Azure Data Factory, Databricks and Apache SparkImplement Delta Lake architecture for reliable data storage and processingBuild and optimize data workflows using Databricks Workflows and JobsDevelop scalable data models following medallion architecture (bronze, silver, gold layers)Implement Unity Catalog for data governance, access control, and metadata managementCreate and maintain Databricks notebooks for data transformation and analysisOptimize Spark jobs for performance and cost efficiencyImplement data quality checks and validation frameworksCollaborate with BI developers, data analysts, and data scientistsDesign and implement data orchestration workflows using Azure Data Factory to coordinate complex ETL/ELT processesDevelop and maintain CI/CD pipelines for data workflowsMonitor data pipeline performance and troubleshoot issuesDocument data processes, architectures, and best practicesEnsure compliance with data security and privacy regulationsProvide support for new and existing solutionsAbout YouKnowledge/Skills/Abilities/Expectations:Excellent problem-solving and analytical skillsStrong oral and written communication abilitiesSelf-motivated with ability to adapt to new technologies quicklyTeam player with ability to work independentlyDetail-oriented with strong organizational skillsAbility to manage multiple priorities and meet deadlinesExperience communicating technical concepts to non-technical stakeholdersTechnical Skills:Databricks Platform: Expert-level knowledge of Databricks Workspace, clusters, and notebooksDelta Lake implementation and optimizationUnity Catalog for data governance and catalogingDatabricks SQL and SQL AnalyticsDatabricks Workflows, Delta Live Tables, and job orchestrationDelta Live Tables (DLT) for pipeline orchestration and data qualityProgramming & Development: Advanced Python programming (PySpark, pandas, NumPy)Advanced SQL (query optimization, performance tuning)Scala programming (preferred)Git version control and collaborative developmentCloud Technologies: Azure DatabricksCloud storage services (ADLS Gen2, Azure Blob Storage)Azure Data Factory for pipeline orchestration and integrationExperience designing and managing Azure Data Factory pipelines, triggers, and linked servicesInfrastructure as Code (Terraform)Business Intelligence & Analytics: Experience with BI tools (Power BI, SSRS)Data warehousing and data modeling conceptsSQL Server, including SSIS (Integration Services)MLflow for ML lifecycle management (plus)Preferred Additional Skills Experience with complex data modeling including dimensional modeling, star/snowflake schemasExperience with medallion architecture (bronze/silver/gold layers)Data quality and validation framework implementationCI/CD pipeline development for data workflows (Azure DevOps)Performance tuning and cost optimizationDataOps and DevOps practicesEducation/Experience:Bachelor's degree in Computer Science, Information Technology or related field5+ years of progressive experience in data engineering, analytics, or software development3+ years of hands-on experience with Databricks platformStrong experience with Apache Spark and PySparkHealthcare IT or healthcare data experience preferredLicenses/Certification:Databricks Certified Data Engineer Associate (strongly preferred)Databricks Certified Data Engineer ProfessionalDatabricks Lakehouse FundamentalsAzure Data Engineer Associate (DP-203)Apache Spark certificationsWe OfferComprehensive Benefits Package: Health Insurance, 401k Plan, Tuition Reimbursement, PTOOpportunity to participate in a Fleet ProgramCompetitive SalariesMileage ReimbursementProfessional growth and development opportunitiesLegaleseThis is a safety-sensitive positionEmployee must meet minimum requirements to be eligible for benefitsWhere applicable, employee must meet state specific requirementsWe are proud to be an EEO employerWe maintain a drug-free workplaceLocationGentivaOur CompanyAt Gentiva, it is our privilege to offer compassionate care in the comfort of wherever our patients call home. We are a national leader in hospice care, palliative care, home health care, and advanced illness management, with nearly 600 locations and thousands of dedicated clinicians across 38 states.Our place is by the side of those who need us – from helping people recover from illness, injury, or surgery in the comfort of their homes to guiding patients and their families through the physical, emotional, and spiritual effects of a serious illness or terminal diagnosis.Our nationwide reach is powered by a family of trusted brands that include:Hospice care: Gentiva Hospice, Emerald Coast Hospice Care, Heartland Hospice, Hospice Plus, New Century Hospice, Regency SouthernCare, SouthernCare Hospice Services, SouthernCare New BeaconPalliative care: Empatia Palliative Care, Emerald Coast Palliative CareHome health care: Heartland Home HealthAdvanced illness management: Illumia HealthWith corporate headquarters in Atlanta, Georgia, and providers delivering care across the U.S., we are proud to offer rewarding careers in a collaborative environment where inspiring achievements are recognized – and kindness is celebrated.