Databricks DevOps Lead

Role: Databricks DevOps Lead
Location: Deerfield, IL
Job Type: Full time
Salary: $100,000-$130,000 a year

Job description
Mandatory skills: Databricks, Spark, IAAS, Azure, Python, Cosmos DB, Azure DevOps (GitHub, CI/CD pipelines, Boards etc.), Docker & Azure Kubernetes Service, Graffana, JUnit, Postman, SonarQube
Additional Skills: 5+ years of DevOps/SRE experience supporting data platforms or distributed systems.
Strong experience with CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components. Hands-on experience with infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault). Solid understanding of build and release strategies for notebooks, JARs, wheels, and configuration artifacts. Experience setting up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools.
Strong scripting skills (Python, PowerShell, or Bash) for automation and tooling.
Good to have skills: Experience with containerization and orchestration (Docker, Kubernetes, AKS) for supporting ancillary services. Knowledge of security best practices (managed identities, secrets management, RBAC) for Azure data services. Exposure to reliability engineering concepts (error budgets, SLOs, SLI dashboards) for data workloads.
Ability to create/maintain CI/CD pipelines (Azure DevOps/GitHub Actions) for Databricks, Spark jobs, and Azure data components.
Develop infrastructure as code (Terraform/Bicep/ARM) for provisioning Azure resources (Databricks, ADLS Gen2, Event Hubs, Key Vault).
Create scripts to manage release strategies for notebooks, JARs, wheels, and configuration artifacts.
Set up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.