Senior Databricks Data Engineer

Senior Databricks Data EngineerThis role leads the technical design and hands-on development of an enterprise-grade Azure Databricks Lakehouse platform, delivering scalable data infrastructure and governance capabilities across the organization.•Build and evolve an internal Azure Databricks Lakehouse platform using Delta Lake and standardized architectural patterns such as medallion architecture.•Develop scalable ingestion and transformation frameworks for multiple source systems using Auto Loader, CDC incremental patterns, and Delta best practices.•Implement curated data products with clear contracts covering schemas, SLAs, data quality rules, ownership, and documentation.•Operationalize pipelines using Databricks Workflows, including scheduling, dependencies, retries, alerting, and run-time observability.•Own performance and cost optimization across workloads, including cluster policies, partitioning strategies, file sizing, and caching.•Implement and administer data governance and security using Unity Catalog, including least-privilege access patterns and audit standards.•Mentor junior and mid-level engineers, lead architecture and design reviews, and drive adoption of platform standards and best practices.•8 years of data engineering experience, including 3 years of recent, hands-on Azure Databricks experience in production environments.•Strong proficiency in Python (PySpark) and Spark SQL for large-scale data transformation.•Strong experience with Delta Lake, including MERGE/UPSERT, schema evolution, time travel, ACID transactions, and incremental processing.•Solid Azure fundamentals relevant to a data platform, including ADLS Gen2, service principals, managed identities, and Azure Key Vault secrets management.•Demonstrated ability to work cross-functionally and communicate technical decisions clearly to non-engineering stakeholders.•Experience with financial datasets such as general ledger, subledger, and accounting line items.•Databricks Certified Data Engineer Professional certification (Associate certification also considered).•Experience implementing Unity Catalog in an enterprise environment, including RBAC model design and audit patterns.•Experience with CI/CD for Databricks assets using Azure DevOps or GitHub Actions, and familiarity with Terraform or asset-bundle deployment.•Experience with streaming or near-real-time ingestion using Structured Streaming, Event Hubs, or Delta Live Tables.•Private equity, credit, or alternative asset management industry experience.•Bachelor's degree in computer science, data engineering, mathematics, or a related field (preferred but not required).