Databricks Data Platform Architect

Databricks Data Platform Architect (DevOps & Infrastructure Focus)

Location: Alameda, CA – Onsite (5 days/week)

About the Role

We are looking for a hands‑on Databricks Data Platform Architect to design, build, and operate our enterprise lakehouse platform on AWS.

This role combines DevOps, Infrastructure Engineering, and Data Engineering architecture.

You will own the reliability, scalability, security, and performance of our Databricks environment and enable data teams to build high-quality, production‑grade pipelines.

If you enjoy building platforms from the ground up, automating everything, and solving performance problems at scale, this role is for you.

What You’ll Own

Platform & Infrastructure Architecture (Core Focus)

• Architect and manage Databricks on AWS end-to-end
• Implement CI/CD for notebooks, jobs, and pipelines
• Define cluster policies, autoscaling, and environment isolation (dev/stage/prod)
• Set up monitoring, logging, and alerting (CloudWatch/Datadog)
• Drive cost optimization and compute efficiency strategies
• Ensure high availability, reliability, and production uptime
• Design lakehouse architecture (bronze/silver/gold)
• Optimize Spark/Delta workloads for performance and scalability
• Establish data engineering standards and best practices
• Support batch and streaming workloads
• Troubleshoot performance bottlenecks and production incidents

Security, Governance & Compliance

• Implement role-based access control using Unity Catalog
• Secure S3, IAM, and external locations
• Enforce data access policies and PII protection
• Enable audit logging and compliance guardrails
• Partner with security and governance teams to ensure best practices

Required Qualifications

• 8+ years in Data Engineering / Platform Engineering / Cloud Infrastructure
• 4+ years hands‑on with Databricks in production
• Strong expertise in Apache Spark, Python, and SQL
• Deep knowledge of AWS (S3, IAM, VPC, CloudWatch, networking)
• Experience with Terraform or other IaC tools
• CI/CD implementation experience for data platforms
• Strong troubleshooting and performance tuning skills

Preferred

• Experience with Delta Lake optimization (ZORDER, OPTIMIZE, partitioning)
• Workflow orchestration tools (Airflow/Databricks Workflows)
• Experience with enterprise data governance tools (Unity Catalog/Atlan/Collibra)
• Knowledge of GDPR/HIPAA or regulated environments
• Prior experience building platforms for multiple teams

What Success Looks Like

• Clear environment separation and automation
• Self‑service data platform for engineers and analysts
• Strong security and governance with minimal friction