AWS Databricks Platform Administrator

Role: AWS Databricks Platform Administrator

Location: Raritan, NJ

Job Summary

We are seeking an experienced AWS Databricks Platform Administrator with 9 years of overall IT experience including substantial handson expertise in administering securing optimizing and supporting Databricks on AWS The role will own platform reliability governance security cost management automation and operational excellence across enterprise data and analytics workloads The ideal candidate combines strong Databricks administration skills with deep knowledge of AWS services infrastructure automation cloud security and production support

Skills

Mandatory Skills : AWS S3, Databricks, Redshift

Key Responsibilities

• Administer and support Databricks accounts workspaces Unity Catalog metastores clusters SQL warehouses jobs instance pools repositories libraries secrets and access controls on AWS
• Design and maintain secure scalable and highly available Databricks platform architecture aligned with enterprise cloud standards
• Manage identity and access through IAM roles and policies service principals single signon SCIM provisioning and rolebased access control
• Implement data governance using Unity Catalog including catalogs schemas external locations storage credentials privileges lineage and auditability
• Configure and troubleshoot AWS integrations such as S3 IAM VPC PrivateLink KMS CloudWatch CloudTrail Route 53 and Secrets Manager
• Establish cluster policies approved runtime standards library controls tagging autotermination serverless usage guidelines and workload isolation
• Monitor platform health performance job reliability capacity and usage proactively identify and resolve bottlenecks and recurring issues
• Drive FinOps practices by analyzing DBU and AWS consumption enforcing tagging and budgets optimizing compute configurations and reducing idle or inefficient resources
• Automate workspace provisioning configuration policy enforcement and operational tasks using Terraform Databricks Asset Bundles Databricks CLISDK Python and CICD pipelines
• Lead incident problem change release vulnerability and servicerequest management for the platform while meeting agreed SLAs and operational KPIs
• Plan and execute runtime upgrades platform migrations disasterrecovery tests backup validation and businesscontinuity activities
• Partner with data engineering analytics security networking cloud engineering and application teams to onboard workloads and resolve complex technical issues
• Maintain architecture diagrams runbooks standard operating procedures support documentation compliance evidence and knowledge articles
• Provide technical leadership mentor administrators and engineers conduct reviews and define platform standards and product roadmaps
• Coordinate with Databricks and AWS support teams for critical incidents product defects and service improvements

Required Experience and Qualifications

• 9 years of overall experience in cloud infrastructure data platforms DevOps platform engineering or production administration
• 5 years of handson Databricks administration experience preferably with enterprise deployments on AWS
• Strong experience with Databricks account console workspace administration Unity Catalog cluster policies SQL warehouses jobs Delta Lake system tables and audit logs
• Strong knowledge of AWS networking security storage encryption monitoring and identity services particularly VPC S3 IAM KMS PrivateLink CloudWatch and CloudTrail
• Handson experience with infrastructure as code and automation using Terraform Python shell scripting REST APIs Databricks CLISDK and CICD tools
• Proven experience supporting businesscritical platforms in production including incident response rootcause analysis change management capacity planning and SLA governance
• Working knowledge of Linux administration networking DNS firewalls certificates proxies and secure connectivity patterns
• Good understanding of Spark architecture Delta Lake workload performance data lakehouse patterns and troubleshooting distributed data workloads
• Experience implementing security controls leastprivilege access secrets management encryption audit logging vulnerability remediation and compliance requirements
• Bachelors degree in Computer Science Information Technology Engineering or a related discipline or equivalent relevant experience

Technical Competencies

• Databricks Account and workspace administration Unity Catalog Delta Lake SQL warehouses clusters jobs notebooks repos instance pools secrets system tables audit logs and cost controls
• AWS S3 IAM VPC EC2 KMS PrivateLink CloudWatch CloudTrail Route 53 Secrets Manager Lambda and AWS Organizations
• Operations Monitoring observability performance tuning capacity management FinOps compliance disaster recovery