Lead Data Engineer

Lead Data Engineer

Location:- Irvine, CA

Duration:- Full Time

Job Description

Must Have Technical/Functional Skills

Data Bricks , EBT ,Airflow , Asset Management exp

Roles & Responsibilities

• We are seeking a highly skilled EDO Platform Engineering Lead to architect, build, and manage enterprise-scale cloud-native data platforms supporting Asset Management business functions. The role will lead platform engineering,
• infrastructure automation, DevOps enablement, platform reliability, security, and operational excellence across the Enterprise Data Office (EDO).
• The ideal candidate will have deep expertise in Databricks, DBT, AWS Cloud, Terraform, Python, DevOps, Platform Engineering, and Data Operations, with strong experience supporting investment management organizations.
• This role is responsible for establishing and managing a scalable, secure, and highly available data platform that supports analytics, reporting, regulatory compliance, portfolio management, risk management, and investment operations.

Key Responsibilities

• Lead the design, implementation, and management of enterprise data platforms.
• Define platform engineering strategy aligned with EDO and enterprise architecture standards.
• Establish reusable platform frameworks, accelerators, and engineering best practices.
• Drive cloud-native modernization initiatives across data and analytics ecosystems.
• Own platform lifecycle management, scalability, availability, and reliability.
• Lead enterprise-wide Databricks platform architecture and administration.
• Design and implement Lakehouse architecture leveraging Delta Lake.
• Manage: Databricks Workflows ,Delta Live Tables (DLT) ,Unity Catalog ,
• MLflow ,Structured Streaming ,Cluster Policies ,Security and Access Controls
• Optimize platform performance and cost efficiency.
• Build governance controls across development, test, and production environments
• Lead cloud engineering activities using AWS services:S3 ,ECS,EKS,Lamda .Glue ,Cloudwatch
• Infrastructure as a Code (IaC)
• Build reusable Terraform modules.
• Automate environment provisioning.
• Implement infrastructure standardization and governance.
• Maintain infrastructure version control.
• Ensure compliance through Infrastructure-as-Code practices.
• Design and implement CI/CD pipelines.
• Automate deployment and release management processes.
• Enable Infrastructure-as-Code and GitOps methodologies.
• Improve deployment velocity and reliability.
• Support engineering teams with self-service platform capabilities.
• GitHub Actions ,GitLab CI/CD ,Jenkins ,Azure DevOps ,Terraform ,Docker ,Kubernetes
• Support API integrations and operational workflows.
• Establish Site Reliability Engineering (SRE) practices.
• Define platform SLAs, SLOs, and operational metrics.
• Lead platform monitoring, observability, and incident management.
• Drive root cause analysis and preventive actions