Databricks Platform Engineer (AWS)

Location: New York(Hybrid)



Employment Type: Contract



Experience: 5 10 Years



Platform Experience: 3 5+ Years Hands-On Databricks Administration on AWS Job Summary

We are seeking a Databricks Platform Manager / Engineer to design, administer, secure, monitor, and optimize an enterprise Databricks platform running on AWS. This role focuses on platform engineering and administration , ensuring scalability, security, governance, reliability, automation, and cost efficiency for data engineering, analytics, and AI teams.

Key Responsibilities

• Administer Databricks workspaces across Development, Test, UAT, and Production.
• Establish workspace standards, networking, environment isolation, and platform configurations.
• Manage Databricks Runtime upgrades and platform lifecycle activities.
• Configure cluster policies, instance pools, job clusters, and compute standards.
• Integrate Databricks with AWS IAM and manage authentication and authorization.
• Administer Unity Catalog , catalogs, schemas, tables, permissions, and RBAC.
• Manage service principals, PATs, secret scopes, and enterprise access controls.
• Configure Git integration and support CI/CD for Databricks notebooks, workflows, and infrastructure.
• Implement Infrastructure as Code using Terraform where applicable.
• Monitor workspace health, jobs, clusters, platform performance, and operational alerts.
• Troubleshoot production issues, perform RCA, and support platform releases.
• Drive Databricks compute optimization, auto-scaling, job scheduling, and cloud cost reduction.
• Maintain audit logging, security controls, operational documentation, and platform standards.

Required Qualifications

• 5 10 years of overall IT experience.
• 3 5+ years of hands-on Databricks platform administration/engineering in production AWS environments.
• Strong experience managing enterprise Databricks workspaces.
• Experience with cluster management, cluster policies, instance pools, and job clusters .
• Hands-on experience with Unity Catalog and RBAC .
• Strong experience with AWS IAM, S3, VPC, EC2, CloudWatch, KMS, and Security Groups .
• Experience with Databricks Runtime management and platform upgrades.
• Experience with Git, CI/CD, Databricks Repos, and Terraform/IaC .
• Experience with platform monitoring, incident management, troubleshooting, and production support.
• Experience with secrets management and audit logging.
• Strong understanding of Databricks security, governance, and cost optimization.

Preferred Qualifications

• Experience managing large-scale Databricks environments with 100+ users.
• Experience supporting multiple business units or enterprise data platforms.
• Strong hands-on Terraform experience.
• Knowledge of Lakehouse Architecture and Delta Lake .
• Experience with FinOps and cloud cost optimization.
• Knowledge of enterprise data governance and security frameworks.
• Experience with AWS Organizations, Control Tower, PrivateLink, or VPC endpoints.
• Experience with CloudWatch, Datadog, or Splunk.
• Exposure to Apache Spark internals and AI/ML workloads on Databricks.

Key Skills

Databricks Platform Administration, Databricks AWS, Workspace Administration, Cluster Management, Cluster Policies, Unity Catalog, AWS IAM, Databricks Runtime, Databricks Repos, CI/CD, Git, Terraform, Infrastructure as Code, Production Support, Platform Monitoring, Security, Governance, Cost Optimization

One-Line Summary

Databricks Platform Engineering AWS Unity Catalog Cluster Management AWS IAM Terraform CI/CD Git Platform Security Monitoring Production Support Cost Optimization