Senior SRE - Platform Engineering

Senior Platform Engineer

Fulltime
We are seeking a Senior Platform Engineer to build, administer, automate, secure, and operate enterprise Databricks and cloud data-platform environments. The role is responsible for Databricks workspace administration, Unity Catalog governance, identity and access management, compute and cluster policies, infrastructure automation, CI/CD enablement, monitoring, reliability, cost management, and production support.

The ideal candidate combines hands-on Databricks administration with strong AWS or Azure platform engineering, Terraform, Python, Kubernetes, CI/CD, security, networking, observability, and Site Reliability Engineering practices.

Databricks Administration and Platform Engineering

• Administer Databricks accounts, workspaces, metastores, catalogs, schemas, external locations, storage credentials, connections, shares, recipients, and platform configurations.
• Configure workspace settings, repositories, jobs, notebooks, SQL warehouses, instance pools, job clusters, all-purpose compute, and serverless capabilities.
• Define and enforce cluster policies, approved runtime versions, libraries, init scripts, autoscaling, tagging, and compute-usage standards.
• Manage Databricks Runtime and platform upgrades, compatibility testing, release planning, maintenance windows, and rollback procedures.
• Support Databricks Workflows, Delta Live Tables or Lakeflow Declarative Pipelines, Databricks SQL, MLflow, model registry, feature engineering, and structured-streaming services.
• Troubleshoot workspace, permissions, connectivity, compute, storage, job execution, library, runtime, and performance issues.
• Maintain administration standards, platform documentation, knowledge articles, support procedures, and operational runbooks.

Design and manage Unity Catalog metastores, catalogs, schemas, managed and external tables, volumes, external locations, storage credentials, and grants.

• Administer account-level and workspace-level users, groups, service principals, permissions, entitlements, and access-control models.
• Implement role-based and attribute-based access controls, least-privilege permissions, separation of duties, and privileged-access procedures.
• Configure secure access patterns for secrets, tokens, credentials, service principals, private endpoints, storage, and external systems.
• Enable audit logging, lineage, system tables, tagging, data classification, row-level security, column masking, and compliance reporting.
• Partner with security and governance teams on encrypti on, key management, network controls, data loss prevention, retention, auditability, and regulatory requirements.
• Review access, monitor privileged activities, remediate policy violations, and support internal and external audits.

Cloud Infrastructure and Networking

• Build and operate Databricks on AWS or Azure, including secure integration with cloud storage, identity, networking, encryption, and monitoring services.
• On AWS, work with S3, IAM, KMS, VPC, PrivateLink, security groups, Route 53, CloudWatch, Secrets Manager, and related services.
• On Azure, work with ADLS Gen2, Microsoft Entra ID, managed identities, Key Vault, virtual networks, private endpoints, network security groups, Azure Monitor, and related services.
• Configure control-plane and data-plane connectivity, private networking, DNS, routing, firewall, proxy, and egress controls.
• Integrate Databricks with cloud data lakes, APIs, databases, message platforms, and enterprise applications.
• Support Kubernetes, Docker, EKS or AKS, and container-based platform services where required.
• Contribute to capacity planning, disaster recovery, high availability, backup, restoration, and business-continuity exercises.

Build and maintain reusable Terraform modules using cloud and Databricks providers.

• Automate workspace, network, Unity Catalog, storage, identity, compute-policy, cluster, job, permission, and monitoring configurations.
• Manage Terraform state, workspaces, variables, modules, versioning, policy checks, drift detection, and controlled promotion across environments.
• Use Python, shell scripting, Databricks CLI, REST APIs, and SDKs to automate administrative and operational tasks.
• Maintain configuration standards and reduce manual administration through repeatable automation.

DevOps, CI/CD, and Release Engineering

• Design and support CI/CD pipelines using GitHub Actions, GitLab CI/CD, Jenkins, Azure DevOps, or Harness.
• Automate deployment of notebooks, jobs, workflows, libraries, policies, infrastructure, and platform configuration.
• Support Databricks Asset Bundles, Git integration, artifact management, environment promotion, testing, approvals, and rollback.
• Integrate security scanning, policy validation, infrastructure testing, and release evidence into delivery pipelines.
• Enable engineering teams through templates, reusable pipelines, documentation, and self-service platform capabilities.
• Partner with application, data-engineering, and DevOps teams to ensure deploy ment standards are consistent and supportable.

Reliability, Monitoring, Operations, and FinOps

• Establish monitoring, alerting, dashboards, logs, metrics, traces, and health checks for Databricks and connected cloud services.
• Use CloudWatch, Azure Monitor, Datadog, Splunk, New Relic, or similar platforms to monitor availability, compute utilization, failures, security events, and cost.
• Define platform service-level indicators, service-level objectives, operational metrics, and error budgets.
• Manage vulnerability remediation, runtime patching, dependency updates, security exceptions, and platform lifecycle activities.
• Optimize cluster sizing, autoscaling, pools, SQL warehouses, job concurrency, serverless usage, storage, and workload scheduling.
• Participate in operational support rotations and maintain escalation paths with Databricks and cloud providers.
• Coordinate platform upgrades, disaster-recovery tests, security reviews, and production-readiness assessments.

Partner with architecture, data engineering, security, cloud, network, governance, FinOps, and service-management teams.

• Advise engineering teams on Databricks platform standards, secure patterns, deployment models, performance, and cost.
• Mentor platform engineers and administrators and lead knowledge-transfer sessions.
• Communicate platform health, risks, dependencies, incidents, and improvement roadmaps to technical and business stakeholders.
• Drive continuous improvement in automation, reliability, security, developer experience, and operational efficiency.

Typically 7-10 years of cloud, DevOps, Site Reliability Engineering, infrastructure, or platform-engineering experience.

• At least 3 years of hands-on Databricks platform administration in an enterprise environment.
• Strong experience administering Databricks workspaces, Unity Catalog, compute, cluster policies, jobs, SQL warehouses, permissions, and service principals.
• Strong experience with AWS or Azure infrastructure, identity, storage, networking, encryption, monitoring, and private connectivity.
• Experience with Python, shell script