DevOps Engineer-Databricks

Overview:

Senior Databricks DevOps Engineer (Azure & CI/CD)

Experience Required

7+ Years

Job Summary

We are seeking an experienced Databricks DevOps Engineer with strong expertise in Azure Cloud, Azure Databricks, CI/CD automation, Infrastructure as Code (IaC), and cloud-native application deployment. The ideal candidate will have hands-on experience in Azure platform services, Databricks administration, Terraform module development, API integrations, and DevOps automation.

The candidate will play a key role in designing, automating, deploying, and supporting scalable cloud and data platform solutions while enabling continuous delivery and operational excellence across enterprise environments.

Key Responsibilities

• Design, develop, and maintain Azure Databricks environments and cloud platform infrastructure.
• Build and manage CI/CD pipelines using Jenkins, GitLab CI/CD, and DevOps best practices.
• Develop reusable Infrastructure as Code (IaC) solutions using Terraform and Terraform modules.
• Automate deployment, monitoring, and operational processes across Azure and Databricks environments.
• Design and implement secure authentication and authorization solutions using Azure AD, OAuth, JWT, and Azure AD B2C.
• Develop and integrate REST APIs and third-party APIs with enterprise applications.
• Manage Azure application registrations, service principals, and identity integrations.
• Deploy and maintain cloud-native applications on Azure Kubernetes Service (AKS).
• Build and support Microsoft Teams applications using Microsoft Teams SDK and Bot Framework.
• Implement real-time communication solutions using Azure SignalR Services.
• Configure monitoring, logging, and observability using Azure Application Insights and Azure monitoring tools.
• Collaborate with Data Engineering teams to support ETL/ELT workloads and data platform operations.
• Ensure security, scalability, reliability, and performance of cloud-based solutions.
• Participate in troubleshooting, incident resolution, and root cause analysis activities.
• Support Agile development and DevOps delivery processes.

Required Skills

Cloud & Platform Technologies

• Microsoft Azure
• Azure Databricks
• Azure Active Directory (Azure AD)
• Azure AD B2C
• Azure Kubernetes Service (AKS)
• Azure SignalR
• Azure Application Insights

DevOps & Automation

• CI/CD Pipelines
• Jenkins
• GitLab CI/CD
• Infrastructure as Code (IaC)
• Terraform
• Terraform Module Development
• Git
• GitHub
• GitLab

Security & Identity

• OAuth 2.0
• JWT Authentication
• Azure App Registrations
• Service Principals
• Identity & Access Management

Development & Integration

• REST APIs
• Third-Party API Integrations
• SQL Server
• Oracle SQL
• Python
• Bash Scripting
• PowerShell

Collaboration Platforms

• Microsoft Teams SDK
• Teams Application Development
• Azure Bot Framework

Nice to Have Skills

Data Engineering

• Data Engineering Concepts
• ETL / ELT Development
• Data Warehousing
• Data Modeling
• Big Data Technologies
• Data Pipeline Automation

Additional Skills

• Workflow Automation
• Monitoring & Observability
• Cloud Governance
• Security Best Practices

Preferred Qualifications

• Experience supporting enterprise-scale Azure Databricks environments.
• Hands-on experience with Terraform module development and cloud automation.
• Experience integrating Azure services with enterprise applications.
• Exposure to Data Engineering and modern data platforms.
• Knowledge of DevSecOps practices.

Soft Skills

• Strong analytical and troubleshooting abilities.
• Excellent communication and stakeholder management skills.
• Ability to work independently in fast-paced environments.
• Strong collaboration and problem-solving mindset.