Senior Databricks Data Engineer

Job Description

Job Description

Job Title: Senior Databricks Data Engineer Location-Type: Onsite, Santa Monica, CA (office relocation to Century City anticipated within approximately one year) Start Date: ASAP Duration: Permanent Compensation Range: $200,000 - $225,000 Benefits: Medical premiums 100% covered, Dental and Vision premiums 100% covered, 401K with employer match up to 6%, 14 days PTO, 9 sick days annually, 11 paid holidays, $20K lifetime fertility benefit, daily catered lunch, weekly carwash service, monthly cellphone reimbursement Visa Sponsorship: Not eligible for visa sponsorship Job Description: This role leads the technical design and hands-on development of an enterprise-grade Azure Databricks Lakehouse platform, delivering scalable data infrastructure and governance capabilities across the organization. Job Summary •Build and evolve an internal Azure Databricks Lakehouse platform using Delta Lake and standardized architectural patterns such as medallion architecture. •Develop scalable ingestion and transformation frameworks for multiple source systems using Auto Loader, CDC incremental patterns, and Delta best practices. •Implement curated data products with clear contracts covering schemas, SLAs, data quality rules, ownership, and documentation. •Operationalize pipelines using Databricks Workflows, including scheduling, dependencies, retries, alerting, and run-time observability. •Own performance and cost optimization across workloads, including cluster policies, partitioning strategies, file sizing, and caching. •Implement and administer data governance and security using Unity Catalog, including least-privilege access patterns and audit standards. •Mentor junior and mid-level engineers, lead architecture and design reviews, and drive adoption of platform standards and best practices. Minimum Requirements: •8 years of data engineering experience, including 3 years of recent, hands-on Azure Databricks experience in production environments. •Strong proficiency in Python (PySpark) and Spark SQL for large-scale data transformation. •Strong experience with Delta Lake, including MERGE/UPSERT, schema evolution, time travel, ACID transactions, and incremental processing. •Solid Azure fundamentals relevant to a data platform, including ADLS Gen2, service principals, managed identities, and Azure Key Vault secrets management. •Demonstrated ability to work cross-functionally and communicate technical decisions clearly to non-engineering stakeholders. •Experience with financial datasets such as general ledger, subledger, and accounting line items. Preferred Qualifications: •Databricks Certified Data Engineer Professional certification (Associate certification also considered). •Experience implementing Unity Catalog in an enterprise environment, including RBAC model design and audit patterns. •Experience with CI/CD for Databricks assets using Azure DevOps or GitHub Actions, and familiarity with Terraform or asset-bundle deployment. •Experience with streaming or near-real-time ingestion using Structured Streaming, Event Hubs, or Delta Live Tables. •Private equity, credit, or alternative asset management industry experience. •Bachelor's degree in computer science, data engineering, mathematics, or a related field (preferred but not required).