Databricks Platform Lead / Technical Delivery Lead

Job Description

Job Title: Databricks Platform Engineering Lead / Technical Delivery Lead

\n

Work Location: Warren, NJ (Hybrid)

\n

Employment Type: Fulltime/Permanent

\n

\n

Need Independent Candidates Only | Local or Nearby

\n

Minimum Years of Experience: 10+ Years

\n

\n

Need Someone who has experience in Python/PySpark and AWS Glue, EMR, S3

\n

Develop and optimize analytical datasets and workloads involving Redshift and Athena

\n

\n

Project Overview

\n

We are seeking an experienced Databricks Platform Engineering Led to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.

\n

The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.

\n

The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.

\n

Technical and Delivery Expertise

\n

\n
• 10+ years of overall technology experience.
\n
• 5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.
\n
• 3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
\n
• Strong hands-on experience with:
\n
• Azure Databricks
\n
• Databricks account and workspace administration
\n
• Unity Catalog
\n
• Apache Spark and PySpark
\n
• Python and SQL
\n
• Delta Lake and Lakehouse architecture
\n
• Azure Data Lake Storage
\n
• Azure Data Factory or comparable integration tools
\n
• Databricks Workflows and job orchestration
\n
• Infrastructure as code using Terraform
\n
• CI/CD and automated deployment practices
\n
• Strong understanding of:
\n
• Databricks workspace provisioning and administration
\n
• Development, test, and production environment management
\n
• Compute provisioning and cluster configuration
\n
• Cluster policies, pools, runtimes, and serverless compute
\n
• Platform security, identity, and access management
\n
• Service principals, managed identities, and secrets management
\n
• Azure networking, private endpoints, and secure connectivity
\n
• Unity Catalog architecture, permissions, lineage, and governance
\n
• Platform monitoring, logging, auditability, and Observability.
\n
• Databricks performance, capacity, and cost optimization
\n
• Production support and platform-incident management
\n
• Good understanding of data-engineering practices, including:
\n
• Batch and streaming data ingestion
\n
• ETL/ELT pipeline development
\n
• Medallion architecture
\n
• Data transformations and enrichment
\n
• Data quality, validation, and reconciliation
\n
• Schema management and data lineage
\n
• Spark workload and query optimization
\n
• Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.
\n
• Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.
\n
• Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.
\n

\n

Preferred Qualifications

\n

\n
• Prior experience implementing a Databricks-enabled pricing, underwriting, actuarial, insurance analytics, or financial-services application.
\n
• Experience working with global insurance or reinsurance carriers.
\n
• Understanding of reinsurance pricing, underwriting, treaty, and facultative business processes.
\n
• Experience integrating Databricks with:
\n
• Pricing engines
\n
• SQL Server applications
\n
• Policy, claims, and accounting systems
\n
• Data warehouses and reporting platforms
\n
• Internal and third-party data sources
\n
• Experience establishing reusable Databricks platform standards, engineering frameworks, and reference architectures.
\n
• Experience implementing a hybrid solution of Databricks and SQL
\n
• Experience defining the division of responsibility among platform engineering, data engineering, application development, DevOps, data science, and production-support teams.
\n
• Familiarity with MLflow, model lifecycle management, model serving, or actuarial-model integration.
\n
• Experience implementing data governance, metadata management, lineage, role-based access, audit controls, and regulatory controls.
\n
• Familiarity with Azure DevOps, Terraform, Power BI, Microsoft Purview, Azure Monitor, or comparable tools.
\n
• Databricks and Microsoft Azure certifications are preferred.
\n
• Experience supporting business-critical production platforms with defined availability, recovery, security, and performance requirements.
\n

\n

The Resource Will Be Expected To:

\n

\n
• Own the engineering and operational management of the Databricks platform supporting the enterprise pricing application.
\n
• Lead end-to-end technical delivery of platform enhancements, environment changes, integrations, releases, and production improvements.
\n
• Provision and administer Databricks accounts, workspaces, compute resources, clusters, policies, and supporting Azure services.
\n
• Define and enforce platform standards covering architecture, security, governance, compute, deployment, monitoring, and cost management.
\n
• Design and maintain appropriate separation across development, test, UAT, and production environments.
\n
• Implement infrastructure as code and automated deployment practices for Databricks and supporting cloud resources.
\n
• Establish secure connectivity between Databricks, SQL Server, cloud storage, the pricing engine, the underwriting workbench, and other enterprise systems.
\n
• Configure and maintain identity, access, security, secrets, networking, and governance controls.
\n
• Define Unity Catalog structures, access models, data ownership, lineage, and auditability standards.
\n
• Establish cluster policies, runtime standards, compute controls, and workload-management practices.
\n
• Monitor and optimize platform availability, reliability, performance, capacity, and cost.
\n
• Troubleshoot workspace, authentication, networking, compute, storage, access, and platform-performance issues.
\n
• Lead root-cause analysis and corrective actions for platform incidents and recurring operational problems.
\n
• Provide technical oversight to data engineers building ingestion pipelines, transformations, curated datasets, and pricing data products.
\n
• Review pipeline designs, Spark workloads, data-processing patterns, deployment approaches, and production-readiness plans.
\n
• Ensure that data-engineering solutions comply with platform, security, governance, quality, and performance standards.
\n
• Coordinate platform engineering, data engineering, application development, QA, DevOps, infrastructure, security, and production-support teams.
\n
• Independently identify technical and delivery impacts across Databricks, the SQL Server pricing engine, the underwriting workbench, and integrated systems.
\n
• Support successful SIT, UAT, regression testing, release validation, deployment, cutover, and post-production stabilization.
\n
• Communicate technical risks, delivery dependencies, architectural decisions, and recovery plans to business and technology stakeholders.
\n
• Serve as the primary technical point of contact for the Databricks platform and its integration with the enterprise pricing application.
\n
• Serve as a subject matter expert on Databricks platform engineering, administration, governance, security, reliability, and operational best practices.
\n

\n