Azure Devops - Databricks platform Engineer

Key Responsibilities

Platform & Infrastructure

Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production) Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources Establish and maintain security best practices, including network isolation, RBAC, and data governance Configure and optimize cluster management, including auto-scaling policies and cost optimization strategies Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls) DevOps & Automation

Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations Implement automated testing and validation of data pipelines Create self-service deployment frameworks and templates for development teams Establish monitoring, alerting, and logging solutions for Databricks environments Automate workspace provisioning and deprovisioning processes Collaboration & Enablement

Partner with data engineers and analysts to understand platform requirements Provide technical guidance and documentation for platform usage Develop runbooks and operational procedures for incident response Mentor team members on Databricks best practices and Azure DevOps tools Governance & Compliance

Implement access controls and audit logging across Databricks workspaces Ensure compliance with organizational security and data governance policies Monitor and manage workspace usage, costs, and resource utilization Document architecture decisions and maintain platform documentation Required Qualifications

7+ years of experience in platform engineering, infrastructure

7+ engineering, or DevOps

2+ years of hands-on experience with Databricks infrastructure setup (or

2+ Apache Spark/Delta Lake environments)

3+ years of experience with Azure cloud services and infrastructure

2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts)

Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise) Solid understanding of CI/CD principles and implementation Experience with Linux/Unix environments and scripting (Bash, Python, or similar) Familiarity with containerization technologies (Docker, Kubernetes) Strong knowledge of networking, security, and identity management in cloud environments Preferred Qualifications

Experience with Unity Catalog and Databricks governance features Knowledge of Apache Spark architecture and optimization Experience with other IaC tools (Terraform) Familiarity with cost optimization strategies in cloud environments Azure certifications (AZ-900, AZ-104, AZ-305) Databricks Certification Experience with Git and version control workflows Knowledge of data governance and data mesh architectures Experience with monitoring tools (Datadog, New Relic, Azure Monitor)