Databricks Administrator

Job Summary We are seeking an experienced Databricks Administrator to manage, secure, optimize, and support enterprise Databricks environments. The ideal candidate will have expertise in Databricks workspace administration, Unity Catalog, cluster management, security, performance optimization, automation, and cloud platforms such as Azure, AWS, or GCP. The role involves working closely with data engineers, data scientists, architects, security teams, and cloud administrators to deliver a secure and scalable data analytics platform. Key Responsibilities Databricks Administration Install, configure, and administer Databricks workspaces. Manage multiple Databricks environments (Development, QA, UAT, Production). Configure and maintain clusters and cluster policies. Manage SQL Warehouses and Databricks Jobs. Monitor workspace health and performance. Upgrade and maintain Databricks Runtime versions. Optimize cluster utilization and reduce infrastructure costs. Security & Governance Implement Unity Catalog governance. Configure Role-Based Access Control (RBAC). Manage users, groups, service principals, and permissions. Integrate Databricks with Azure AD, AWS IAM, or Okta. Configure Single Sign-On (SSO). Manage secrets using Databricks Secret Scopes or cloud key management services. Ensure compliance with organizational security policies. Workspace Administration Create and manage workspaces. Configure notebooks and repositories. Manage Git integration. Administer workspace permissions. Manage cluster access policies. Configure job permissions. Manage libraries and dependencies. Infrastructure Management Deploy Databricks using Terraform. Automate infrastructure provisioning. Configure networking: VPC/VNet Private Link Secure Cluster Connectivity NSG/Security Groups Firewalls. Configure storage accounts (ADLS, S3). Manage DBFS and external storage. Performance Optimization Tune Spark clusters. Optimize autoscaling. Configure instance pools. Monitor cluster performance. Troubleshoot slow jobs. Optimize notebook execution. Manage caching strategies. Monitoring & Troubleshooting Monitor Databricks clusters. Configure alerts. Review audit logs. Troubleshoot: Cluster failures Job failures Authentication issues Permission issues Connectivity issues Library conflicts Performance bottlenecks Automation Automate workspace deployment. Automate cluster creation. Automate user provisioning. Automate permission management. Automate backup processes. Automate scheduled jobs using Terraform, Azure DevOps, GitHub Actions, or Jenkins.

CI/CD

Implement CI/CD pipelines. Git integration. Azure DevOps Pipelines. GitHub Actions. Jenkins. Deployment automation. Release management. Data Governance Unity Catalog Data Lineage Catalog Management Schema Management Access Policies Table Permissions Row-Level Security Column-Level Security Data Masking Backup & Disaster Recovery Backup workspace configurations. Backup notebooks. Backup cluster configurations. Disaster recovery planning. Workspace migration. Restore procedures. Required Skills Databricks Databricks Workspace Administration Unity Catalog Cluster Administration SQL Warehouses Jobs Notebooks Repos DBFS Delta Lake Delta Live Tables MLflow Apache Spark Spark Administration Spark SQL Performance Tuning Cluster Optimization Cloud Platforms Azure AWS GCP Infrastructure as Code Terraform ARM Templates CloudFormation Scripting Python Bash PowerShell DevOps Git Azure DevOps Jenkins GitHub Actions CI/CD Monitoring Azure Monitor CloudWatch Datadog Prometheus Grafana Splunk Security Azure AD

AWS IAM

OAuth SAML LDAP RBAC Secret Scopes Key Vault AWS Secrets Manager Databases SQL Server Snowflake Oracle PostgreSQL MySQL Preferred Qualifications Databricks Certified Data Engineer Associate/Professional Databricks Certified Administrator (where applicable) Azure Administrator Associate AWS Solutions Architect Associate Experience with Lakehouse Architecture Experience with Unity Catalog Governance Experience with Delta Lake Optimization Knowledge of Apache Iceberg Experience supporting enterprise data platforms Soft Skills Strong analytical and troubleshooting skills Excellent verbal and written communication Ability to work with cross-functional teams Strong documentation skills Ability to support production environments Customer-focused mindset Experience working in Agile/Scrum environments Nice-to-Have Skills Azure Synapse Analytics Microsoft Fabric Kafka Airflow dbt Kubernetes Docker Python-based automation AI/ML platform administration Cost optimization and FinOps experience #J-18808-Ljbffr CloudIngest