Consulting Partner - Data Engineering & Modern Data Platforms

Job Title: Consulting Partner - Data Engineering & Modern Data Platforms

Location: Irvine, CA

Full-Time




Job Description

Required Experience

15+ years of experience in data engineering, data platforms, analytics, or technology architecture.

10+ years of deep hands-on technical experience in data engineering and data platform technologies.

Demonstrated experience architecting and delivering complex enterprise-scale data platforms.

Strong hands-on experience with Spark/PySpark and SQL.

Strong practical experience with at least one modern cloud data platform such as Databricks or Snowflake.

Experience with modern data transformation frameworks such as dbt.

Experience with workflow orchestration technologies such as Airflow.

Strong understanding of distributed computing, data processing performance, and scalability.

Experience leading large-scale data modernization or migration programs.

Experience managing technical teams distributed across multiple locations.

Ability to engage directly with engineers, architects, platform teams, and executive stakeholders.

Preferred Experience

Cloud-to-cloud data platform migration.

Migration of large PySpark estates.

Legacy ETL modernization.

Stored procedure modernization to dbt or modern transformation frameworks.

Databricks platform engineering.

Data platform governance and security.

Large-scale data migration involving hundreds of terabytes or petabyte-scale environments.

AI-assisted engineering and migration accelerators.

Building reusable engineering frameworks and accelerators.

Experience with platform reliability, FinOps, and performance optimization.

Roles & Responsibilities

We are seeking a highly accomplished Consulting Partner with deep hands-on Data Engineering and Modern Data Platform expertise to lead complex enterprise data transformation programs.

This is a technical leadership role, not a purely advisory or relationship-management position. The successful candidate will combine executive consulting capabilities with strong architecture and engineering expertise and must have demonstrated hands-on experience designing, building, modernizing, and troubleshooting large-scale data platforms and pipelines.

The Consulting Partner will work with client executives to shape transformation strategies while remaining sufficiently close to technology and engineering teams to influence architecture, solve complex technical problems, review implementations, and guide critical delivery decisions.

The role requires an individual who can move seamlessly from an executive discussion with a CIO or CDO to a deep technical discussion involving Databricks, Spark/PySpark, dbt, SQL, cloud architecture, data pipelines, orchestration, performance optimization, and large-scale data migration.

Key Responsibilities

1. Technical Consulting & Architecture Leadership

Serve as the senior technical advisor for complex data engineering and data platform transformation programs.

Define enterprise data platform, lakehouse, and data engineering architectures.

Lead architecture assessments and develop modernization strategies for legacy and cloud-based data ecosystems.

Design scalable architectures covering data ingestion, transformation, storage, orchestration, governance, security, and consumption.

Review and challenge architecture decisions made by engineering and solution teams.

Establish technology standards, engineering principles, and reference architectures.

Identify technical risks and scalability challenges early in the program lifecycle.

Provide hands-on technical guidance for complex architecture and engineering issues.

2. Hands-On Data Engineering Leadership

The Consulting Partner must maintain strong hands-on technical depth and be capable of working directly with engineering teams when required.

Responsibilities include:

Reviewing and optimizing PySpark and Spark applications.

Analyzing query and pipeline performance issues.

Reviewing complex SQL transformations.

Designing and reviewing dbt models, transformation frameworks, and testing strategies.

Troubleshooting complex data processing and distributed computing issues.

Reviewing data partitioning, clustering, file sizing, and storage optimization strategies.

Providing technical direction on Delta Lake, table design, and data lifecycle management.

Reviewing Airflow DAGs and orchestration architecture.

Helping teams resolve critical production issues and performance bottlenecks.

Conducting architecture and code reviews for high-risk or complex components.

The role does not require day-to-day individual contributor coding but requires the ability to get hands-on with technology when necessary and independently validate engineering decisions.

3. Complex Program Technical Leadership

Lead the technical direction of large-scale data engineering and modernization programs.

Oversee programs involving hundreds or thousands of pipelines, jobs, tables, and datasets.

Define technical migration strategies for:

o Legacy ETL to modern ELT

o Stored procedures to dbt

o Legacy Spark platforms to Databricks

o Cloud-to-cloud data platform migration

o Autosys and legacy scheduling to Airflow

o Data warehouse modernization

o Data lake and lakehouse implementation

Define migration patterns, factory approaches, automation strategies, and validation frameworks.

Establish technical governance across multiple engineering teams and workstreams.

Resolve cross-platform and cross-domain technical dependencies.

Define technical KPIs for scalability, reliability, performance, data quality, and cost efficiency.

4. Modern Data Platform Expertise

The candidate should have strong practical experience across several of the following technologies:

Data Engineering

Apache Spark

PySpark

SQL

Python

Distributed data processing

Data pipeline design and optimization

Modern Data Platforms

Databricks

Delta Lake

Snowflake

Data lakehouse architectures

Data warehouse modernization

Transformation & Orchestration

dbt

Apache Airflow

Astronomer

Enterprise workload orchestration

CI/CD for data engineering

Cloud

AWS, Azure and/or GCP

Cloud storage and data services

IAM and security architecture

Networking for data platforms

Infrastructure as Code

Terraform

Governance & Reliability

Unity Catalog

Data catalog and metadata management

Data lineage

Data quality

Data observability

DataOps

5. Executive Consulting & Client Leadership

Act as a trusted technical advisor to CIOs, CDOs, CTOs, and senior Data and Analytics executives.

Translate complex technical issues into business implications and executive decisions.