Senior Platform Engineer

Job Title: Senior Platform Engineer
Location: Irvine, CA
Duration: Fulltime


Skills: Data Integration
Salary: 130K-150K/Year


Roles and Responsibilities:
• We are seeking a Senior Platform Engineer to build, administer, automate, secure, and operate enterprise Databricks and cloud data-platform environments. The role is responsible for Databricks workspace administration, Unity Catalog governance, identity and access management, compute and cluster policies, infrastructure automation, CI/CD enablement, monitoring, reliability, cost management, and production support.
• The ideal candidate combines hands-on Databricks administration with strong AWS or Azure platform engineering, Terraform, Python, Kubernetes, CI/CD, security, networking, observability, and Site Reliability Engineering practices.


Key Responsibilities
Databricks Administration and Platform Engineering
• dminister Databricks accounts, workspaces, metastores, catalogs, schemas, external locations, storage credentials, connections, shares, recipients, and platform configurations.
• Provision and manage development, test, staging, and production workspaces using standardized, repeatable patterns.
• Configure workspace settings, repositories, jobs, notebooks, SQL warehouses, instance pools, job clusters, all-purpose compute, and serverless capabilities.
• Define and enforce cluster policies, approved runtime versions, libraries, init scripts, autoscaling, tagging, and compute-usage standards.
• Manage Databricks Runtime and platform upgrades, compatibility testing, release planning, maintenance windows, and rollback procedures.
• Support Databricks Workflows, Delta Live Tables or Lakeflow Declarative Pipelines, Databricks SQL, MLflow, model registry, feature engineering, and structured-streaming services.
• Troubleshoot workspace, permissions, connectivity, compute, storage, job execution, library, runtime, and performance issues.
• Maintain administration standards, platform documentation, knowledge articles, support procedures, and operational runbooks.


Unity Catalog, Identity, Security, and Governance
• Design and manage Unity Catalog metastores, catalogs, schemas, managed and external tables, volumes, external locations, storage credentials, and grants.
• Implement user and group provisioning through single sign-on, SCIM, identity-provider integration, and enterprise directory services.
• dminister account-level and workspace-level users, groups, service principals, permissions, entitlements, and access-control models.
• Implement role-based and attribute-based access controls, least-privilege permissions, separation of duties, and privileged-access procedures.
• Configure secure access patterns for secrets, tokens, credentials, service principals, private endpoints, storage, and external systems.
• Enable audit logging, lineage, system tables, tagging, data classification, row-level security, column masking, and compliance reporting.
• Partner with security and governance teams on encrypti on, key management, network controls, data loss prevention, retention, auditability, and regulatory requirements.
• Review access, monitor privileged activities, remediate policy violations, and support internal and external audits.


Cloud Infrastructure and Networking
• Build and operate Databricks on AWS or Azure, including secure integration with cloud storage, identity, networking, encryption, and monitoring services.
• On AWS, work with S3, IAM, KMS, VPC, PrivateLink, security groups, Route 53, CloudWatch, Secrets Manager, and related services.
• On Azure, work with ADLS Gen2, Microsoft Entra ID, managed identities, Key Vault, virtual networks, private endpoints, network security groups, Azure Monitor, and related services.
• Configure control-plane and data-plane connectivity, private networking, DNS, routing, firewall, proxy, and egress controls.
• Integrate Databricks with cloud data lakes, APIs, databases, message platforms, and enterprise applications.
• Support Kubernetes, Docker, EKS or AKS, and container-based platform services where required.
• Contribute to capacity planning, disaster recovery, high availability, backup, restoration, and business-continuity exercises.


Infrastructure as Code and Automation
• Build and maintain reusable Terraform modules using cloud and Databricks providers.
• utomate workspace, network, Unity Catalog, storage, identity, compute-policy, cluster, job, permission, and monitoring configurations.
• Manage Terraform state, workspaces, variables, modules, versioning, policy checks, drift detection, and controlled promotion across environments.
• Use Python, shell scripting, Databricks CLI, REST APIs, and SDKs to automate administrative and operational tasks.
• Implement self-service workspace, catalog, schema, access, and compute vending with appropriate approval and governance controls.
• Maintain configuration standards and reduce manual administration through repeatable automation.


DevOps, CI/CD, and Release Engineering
• Design and support CI/CD pipelines using GitHub Actions, GitLab CI/CD, Jenkins, Azure DevOps, or Harness.
• utomate deployment of notebooks, jobs, workflows, libraries, policies, infrastructure, and platform configuration.
• Support Databricks Asset Bundles, Git integration, artifact management, environment promotion, testing, approvals, and rollback.
• Integrate security scanning, policy validation, infrastructure testing, and release evidence into delivery pipelines.
• Enable engineering teams through templates, reusable pipelines, documentation, and self-service platform capabilities.
• Partner with application, data-engineering, and DevOps teams to ensure deploy ment standards are consistent and supportable.


Reliability, Monitoring, Operations, and FinOps
• Establish monitoring, alerting, dashboards, logs, metrics, traces, and health checks for Databricks and connected cloud services.
• Use CloudWatch, Azure Monitor, Datadog, Splunk, New Relic, or similar platforms to monitor availability, compute utilization, failures, security events, and cost.
• Define platform service-level indicators, service-level objectives, operational metrics, and error budgets.
• Lead incident response, problem management, root-cause analysis, corrective actions, and post-incident reviews.
• Manage vulnerability remediation, runtime patching, dependency updates, security exceptions, and platform lifecycle activities.
• Optimize cluster sizing, autoscaling, pools, SQL warehouses, job concurrency, serverless usage, storage, and workload scheduling.
• Implement budget controls, chargeback or showback tagging, utilization reporting, anomaly detection, and cost-optimization recommendations.
• Participate in operational support rotations and maintain escalation paths with Databricks and cloud providers.
• Coordinate platform upgrades, disaster-recovery tests, security reviews, and production-readiness assessments.


Collaboration and Technical Leadership
• Partner with architecture, data engineering, security, cloud, network, governance, FinOps, and service-management teams.
• dvise engineering teams on Databricks platform standards, secure patterns, deployment models, performance, and cost.
• Conduct technical reviews and ensure solutions meet enterprise architecture and operational-support requirements.
• Mentor platform engineers and administrators and lead knowledge-transfer sessions.
• Communicate platform health, risks, dependencies, incidents, and improvement roadmaps to technical and business stakeholders.
• Drive continuous improvement in automation, reliability, security, developer experience, and operational efficiency.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.