Senior Azure Databricks Engineer

Job Description

Job Description

Pay Range: $51.40hr - $66.71hr

Job Overview:

The candidate will collaborate with the Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.

Roles & Responsibilities:

• Collaborate with the Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, and related Azure services.
• Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads.
• Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance.
• Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns.
• Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities.
• Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices.
• Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables.
• Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation.
• Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture.
• Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.

Requirements:

• At least 8-10 years of experience in designing and implementing enterprise data platforms.
• At least 5 years of hands-on Azure Databricks experience in enterprise environments.
• Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake.
• Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security.
• Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL).
• Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage, and fine-grained access controls.
• Experience designing and working with enterprise security architectures including RBAC, Managed Identities, Private Endpoints, Key Vault integration, encryption, and network security.
• Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards.
• Strong understanding of data modeling techniques including dimensional modeling, Data Vault, medallion architecture, and enterprise data warehouse concepts.
• Strong knowledge of Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking.
• Strong SQL programming skills and proficiency in Python and PySpark development.
• Strong experience implementing CI/CD pipelines utilizing Azure DevOps, automated testing, release management, and deployment automation.

Benefits:

Our Benefits Include:

• Medical, Dental, and Vision Insurance
• 401(k) Retirement Plan
• Health Savings Account (HSA)
• Disability Insurance (Short-Term and Long-Term)
• Life and AD&D Insurance
• Paid Sick Leave (where required by applicable state or local law)
• Supplemental Insurance Plans
• Identity Theft Protection
• Pet Insurance
• Employee Wellness Programs
• Employee Assistance Program (EAP)
• Career Growth and Professional Development Opportunities

Disclaimer : Benefits eligibility, accrual rates, and usage limits may vary based on employment status, length of service, and work location. Paid Sick Leave is provided in strict accordance with applicable state and municipal mandates. Cynet Systems Inc. reserves the right to modify, amend, or terminate any benefit plans at any time in accordance with applicable laws.

About Cynet Systems

Founded in 2010 and headquartered in the Washington, DC metro area, Cynet Systems Inc. is a leading technology staffing and workforce solutions company serving Fortune 500 companies, government agencies, and enterprise organizations across the United States and Canada. We deliver agile, scalable talent solutions across IT, engineering, life sciences, clinical, and professional staffing, powered by a high-performing recruitment engine operating across North America and Asia.

As a nationally and locally certified Minority Business Enterprise (MBE), Cynet Systems is committed to helping organizations build high-performing teams while empowering professionals to grow rewarding careers. Our organization is certified to ISO 9001, ISO 14001, ISO 27001, and SOC 2 Type II standards, reflecting our commitment to quality, security, operational excellence, and customer success.