Data Engineering Technical Lead

job summary:

We are seeking a hands-on Data Engineering Technical Lead to help shape, modernize, and scale our enterprise data platform. This role is central to our mission of transforming legacy data systems into a modern, cloud-native Lakehouse environment that powers analytics, AI, and business intelligence across the organization. As a technical lead, you will design and deliver scalable data pipelines, define data design patterns, enforce engineering standards, and leverage AI-assisted tools to accelerate modernization, improve productivity, and reduce technical debt. You will drive proofs of concept (POCs) and points of view (POVs) to evaluate emerging technologies and frameworks, ensuring that the platform remains innovative, cost-efficient, and future-ready.

location: San Diego, California

job type: Permanent

salary: $130,000 - 150,000 per year

work hours: 8am to 5pm

education: Bachelors

responsibilities:

Lead the transformation of SSIS/SSRS workloads into modular, high-performance pipelines using Databricks, dbt, Fivetran, and Airflow.

Define and implement standardized frameworks for ingestion, transformation, curation, and consumption layers across the Lakehouse.

Experiment with new technologies (e.g., Delta Live Tables, Iceberg, streaming ingestion, AI-driven observability) to validate architecture choices and influence the enterprise roadmap.

Use AI-enabled tools like Databricks Assistant, Cursor AI, GitHub Copilot, and dbt Mesh AI tests for code generation, automated testing, documentation, and pipeline optimization.

Integrate predictive models to detect pipeline anomalies, data drift, and optimize compute and scheduling.

Drive CI/CD, version control, peer reviews, and observability practices across the data platform.

Partner with data architects, platform engineers, analysts, and business product owners to translate business needs into technical solutions.

Provide technical guidance, foster continuous learning, and help the team adopt modern data engineering best practices.

Continuously tune Spark workloads, storage tiers, and orchestration logic across Azure and Google Cloud Platform environments.

qualifications:

Bachelor's degree in Computer Science, Data Engineering, or a related technical field.

8+ years of experience in data engineering or related technical fields, with at least 3+ years in a lead or senior role.

Proven experience designing and implementing data design patterns (e.g., CDC, SCD, Medallion, Data Vault, streaming, and batch patterns).

Deep expertise with Databricks, Apache Spark, dbt, Fivetran, Census, Airflow, and Kafka. Solid experience across Azure and/or Google Cloud Platform (e.g., Synapse, Data Factory, BigQuery, Pub/Sub).

Hands-on experience modernizing legacy ETL (SSIS/SSRS) workloads into cloud-native pipelines.

Demonstrated ability to build POCs and POVs that validate new tools, frameworks, or architectures.

Working knowledge of AI-assisted engineering tools for development, observability, or optimization.

Proficiency in SQL and at least one programming language (Python, Scala, or Java).

Strong problem-solving, architectural thinking, and collaboration skills. Excellent communicator with the ability to translate technical topics to business stakeholders.

skills:

Airflow,Kafka,Spark,Apache Spark,AI,AI-enabled,automated testing,BigQuery,cloud-native,code generation,CI/CD,Cursor,analytics,data design,data platform,scalable data pipelines,streaming,data systems,Data Vault,Databricks,ETL pipelines,ETL,GitHub Copilot,Data Engineering,Java,Azure,Python,Proficiency in SQL,SSIS,SSRS,engineering tools,version control,technical debt,Excellent communicator,Strong problem-solving,high-performance,architecture,business intelligence,business needs,Census,continuous learning,curation,dbt,Experiment,innovative,observability practices,Mentor,operational intelligence,predictive models,collaboration skills,Pub/Sub,scheduling,documentation

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact


Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.

Qualified applicants in San Francisco with criminal histories will be considered for employment in accordance with the San Francisco Fair Chance Ordinance.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.

We will consider for employment all qualified Applicants, including those with criminal histories, in a manner consistent with the requirements of applicable state and local laws, including the City of Los Angeles' Fair Chance Initiative for Hiring Ordinance.