Lead Data Engineer

VBeyond

Role: Lead Data Engineer (Priority 1)

Location: San Diego( Onsite only )

Job Type Contrct/Full Time

Lead Data Engineer JD:

Role Summary

We are seeking a highly skilled Lead Data Engineer with deep, hands-on experience in building large-scale data ingestion pipelines, real-time streaming solutions, and high-performance data processing systems. The ideal candidate excels at writing clean, efficient code, refactoring complex systems, improving scalability and performance, addressing production issues, and delivering reliable data solutions across cloud platforms such as AWS, Azure, or GCP .

Key Responsibilities

Design, build, and maintain high-volume data ingestion and processing pipelines for batch and real-time workloads.
Implement and optimize real-time streaming pipelines using platforms such as Kafka .
Develop scalable data solutions using Databricks, PySpark, Python, and SQL .
Perform core refactoring to modernize and optimize existing pipelines and data services.
Build robust, fault-tolerant pipelines capable of large-scale, high-throughput data processing.
Write unit tests , automate validation, and ensure high code quality and reliability.
Integrate pipelines into CI/CD workflows to streamline and automate deployment processes.
Identify, troubleshoot, and fix production issues , ensuring system reliability and stability.
Address performance bottlenecks and implement improvements for scalability, throughput, and efficiency.
Work extensively across AWS, Azure, or GCP cloud environments and cloud-native data services.
Design and orchestrate end-to-end pipelines using workflow and orchestration tools.
Collaborate with Data Scientists and BI Engineers to deliver clean, analytics-ready datasets.
Communicate complex technical topics clearly to non-technical stakeholders.

Required Qualifications

8+ years of hands-on data engineering experience building and maintaining large-scale data systems.
Proven experience with high-volume data ingestion , ETL/ELT, and real-time data processing.
Strong expertise with Kafka or similar streaming technologies.
Advanced proficiency in Databricks, PySpark, Python, and SQL .
Experience with core refactoring , improving code maintainability and system performance.
Demonstrated ability to design and build scalable, low-latency data pipelines .
Strong skills in debugging, performance optimization , and pipeline tuning.
Hands-on experience with at least one major cloud platform: AWS, Azure, or GCP .
Experience writing unit tests and integrating solutions with CI/CD pipelines .
Strong problem-solving skills and excellent communication abilities.

Qualification: BE/BTech/MS in Computer Science or related field

Apply

Vacancy posted 4 days ago

Similar jobs that could be interesting for youBased on the Lead Data Engineer in San Diego, CA vacancy

Data Engineer with streaming
...Data Engineer Location: MTV/San Diego CA (100% Onsite) Job Description: Expertise in Apache Flink or streaming exp Strong programming skills in Java Knowledge of stream processing and batch processing Experience working with Apache Kafka. Proven...
Suggested
United IT
San Diego, CA
4 days ago
Data Ops Engineer
...Data Scientist Xenith Solutions is a small family focused business where we focus on... ...of the Navy, DoD, and Intel communities engineering and technology needs. You should possess... ...libraries, and visualization/BI tools to lead advanced analytics initiatives supporting...
Suggested
For contractors
Local area
Xenith Solutions
San Diego, CA
3 days ago
Data Engineer
$61.9k - $141k
...Data Engineer The Opportunity: Ever-expanding technology like IoT, machine learning, and artificial intelligence means that there's more structured and unstructured data available today than ever before. As a data engineer, you know that organizing data can yield pivotal...
Suggested
Full time
Contract work
Part time
Local area
Navstar
San Diego, CA
4 days ago
Senior Data Engineer (Labs)
...Senior Data Engineer (Labs) Ignite Digital enables national security agencies to accelerate decisions, elevate operational outcomes and... ...partnership beyond presentations to inspire the confidence to lead in a digital-forward world. Ignite Digital. The Edge from Within...
Suggested
For contractors
Work at office
1 day per week
Ignite Digital Services
San Diego, CA
4 days ago
Data Engineer
...staffing and recruiting solutions, has an immediate opening for a Data Engineer. Data Warehouse. Ideally Cloud & Google BigQuery, but any... ...can do the work as needed, but importantly we are looking for a lead type person that can be relied on and leveraged to help create...
Suggested
For contractors
Local area
Immediate start
KORE1 Technologies
San Diego, CA
2 days ago
Data Engineer
$84.9k - $110.4k
...passionate about serving their local communities. Join our team: Apply to our open position(s) below. OPEN POSITION ANNOUNCEMENT! Data Engineer Open until filled. Posted 4/29/2026 San Diego County - hybrid work schedule set by SDCP About the role : The San...
Interim role
Work at office
Local area
San Diego Community Power
San Diego, CA
4 days ago
Data Engineer
...Data Engineer The Data Engineer independently designs, builds, and maintains the data infrastructure that enables data-driven decision-making across the Naval Aviation Enterprise. Working within the NAAC/FRAG team, this individual operationalizes raw data from NAE source...
For contractors
For subcontractor
T3W Business Solutions
San Diego, CA
4 days ago
Sr. Data Engineer
...Position: Sr. Data Engineer Type: Remote Duration: Long Term Requirement: Candidate must have: Proficiency in Python, SQL, GCP tools, clean rooms (Google, Databricks), Kafka or similar streaming tools, and strong understanding of retail media...
Local area
Remote work
E-Solutions
La Jolla, CA
2 days ago
Data Engineer
...currently 250 downstream applications that depend on the existing student information system, so these apps need to connect to a new data warehouse once the new system goes live. To support this effort, the SQL developer will extract data from the existing data...
Insight Global
San Diego, CA
1 day ago
Data Engineer
$61.9k - $141k
...Data Engineer The Opportunity: Ever-expanding technology like IoT, machine learning, and artifi cia l intelligence means that there's more structured and unstructured data available today than ever before. As a data engineer, you know that organizing data can yield...
Full time
Contract work
Part time
Work at office
Local area
Remote work
BOOZ, ALLEN & HAMILTON, INC.
San Diego, CA
3 days ago
Data Engineer
$175k - $225k
...The Marlin Alliance is seeking a forward-thinking Data Engineer in San Diego, CAto provide client support to our Navy client. This is an on-site role and applicants must have active DoD Secret Clearance. Incorporated in 2002, The Marlin Alliance is a digital transformation...
Contract work
Work at office
The Marlin Alliance
San Diego, CA
3 days ago
Senior Data Engineers - Remote
...Position : Senior Data Engineers - Remote Location : San Diego, CA Duration : 06 Months Total Hours/week : 40.00 1st Shift Client... ...Temp to perm As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining...
Permanent employment
Contract work
Temporary work
Work experience placement
Remote work
Day shift
MILLENNIUMSOFT
San Diego, CA
1 day ago
Data Ops Engineer VFDE
$200k - $240k
...Description We are seeking a Data Operations Engineer to design, build, and maintain real-time data ingestion pipelines. In this role, you will be responsible for the reliable flow of streaming data from a wide range of sources into our data platform, ensuring data...
SAIC
San Diego, CA
5 days ago
Information Assurance Engineer II
$93.9k - $156.8k
...Information Assurance Engineer II Location US-CA-San Diego ID 2026-4027 Category IT / Cyber Security / Network Systems Position Type Full-Time Remote No Clearance Required Secret Overview...
Full time
For contractors
Remote work
American Systems
San Diego, CA
1 day ago
Data Ops Engineer VFDE
$200k - $240k
...Job Description Description We are seeking a Data Operations Engineer to design, build, and maintain real-time data ingestion pipelines. In this role, you will be responsible for the reliable flow of streaming data from a wide range of sources into our...
Science Applications International Corporation
San Diego, CA
8 hours ago
Databricks Data Engineer - Senior - Consulting - Miami
$106.9k - $176.5k
...help to build a better working world. Technology – Data and Decision Science – Data Engineering – Senior We are seeking a highly skilled Senior Consultant... ...solutions that meet their business objectives. Lead end-to-end data pipeline development, including data...
Summer holiday
Flexible hours
EY
San Diego, CA
8 hours ago
Senior Scientific Data Engineer
$101k - $172k
...eradicate the disease and the suffering it causes. As a Senior Data Engineer, this seasoned professional will demonstrate competence and... ...in a wide range of technical areas. This role will have a lead role in the design, development, and testing of software applications...
Full time
Part time
For contractors
Local area
Work visa
Night shift
Weekend work
Exact Sciences
San Diego, CA
4 days ago
Data Engineer
$122k - $130k
...Data Engineer The Data Engineer at Bumble Bee Seafoods is responsible for designing, building, and operating the enterprise data platform... ...in the U.S., Canada and over 50 markets globally under leading brands including Bumble Bee®, Brunswick®, Clover Leaf®, Snow's...
Full time
Local area
Flexible hours
Bumble Bee Foods
San Diego, CA
3 days ago
Healthcare Data Engineer
$110k - $160k
...recruiter to learn more.Base pay range$110,000.00/yr - $160,000.00/yrThis company is a value based care provider that manages a number of data ingestion points for third party API integration. Great work life balance and remote work!This Jobot Job is hosted by Alex...
Local area
Remote work
Jobot
San Diego, CA
3 days ago
Senior Data Engineer (Remote)
...About the job Senior Data Engineer (Remote) Technology is the next frontier of change in Agriculture and will be central to delivering... ...and operational efficiency of data and analytical solutions Lead the evaluation, implementation, and deployment of emerging tools...
Remote work
Military, Veterans and Diverse Job Seekers
San Diego, CA
2 days ago
Remote Senior Data Engineer - Data Pipelines & Analytics
A leading agriculture technology firm is seeking a Senior Data Engineer to join their digital team. This role involves designing and maintaining data pipelines that support data-driven decision-making for farmers. The ideal candidate should have strong experience in data...
Remote work
Military, Veterans and Diverse Job Seekers
San Diego, CA
2 days ago
Data Engineer-San Diego, or San Jose, CA
...Data Engineer Location – San Diego, or San Jose, CA Duration – 6 months Rate – DOE $/hr. on w2 US Citizen, Green Card, TN, only Primary... ...~ Designing data models ~ Proficient understanding and tech-lead level experience in one or more programming languages (Java,...
Staffing the Universe
San Diego, CA
4 days ago
Data Engineer II
$105k - $125k
...Overview SteerBridge seeks a highly skilled and motivated Data Engineer II to join our team supporting the F-35 AI/ML Spares Project.... ...analysts, and operational partners. Key Responsibilities • Lead end-to-end data pipeline operations - design, develop, and...
SteerBridge
San Diego, CA
2 days ago
Senior Data Engineer
...Senior Data Engineer AArete is one-of-a-kind when it comes to consulting firm culture. We're a global, innovative management and technology consulting firm with offices in the U.S., India, and Europe. Our name comes from the Greek word for excellence: "Areté." And...
Contract work
Work at office
Flexible hours
Night shift
AArete
San Diego, CA
2 days ago
Data Engineer
...will collaborate in the design, creation, and enhancement of our data infrastructure, which underpins the productization of our... ...numerous other devices. We are seeking a highly skilled Data Software Engineer to design, develop, and maintain scalable data infrastructure and...
Apple
San Diego, CA
3 days ago
Data Engineer
...duties may be assigned. The Database Engineer III is responsible for key areas which includes... ...activities including production data modification requests Gains a thorough... ...with team members and technical project lead to ensure component design fits in with the...
Local area
Remote work
Monday to Friday
Night shift
Weekend work
My3Tech Inc
San Diego, CA
3 days ago
Data Engineer
$105k - $130k
...* *We are unable to sponsor or take over sponsorship of an employment visa at this time* RIVO is seeking an experienced Data Engineer to join our Data team and help build upon the infrastructure that drives analytics, reporting, and data-driven decision-making...
Remote work
Work visa
Weekend work
Rivo Holdings
San Diego, CA
4 days ago
AWS Data Sr Engineers
$60 - $65 per hour
...AWS Data Sr Engineers Client: INSULET Location: San Diego, CA or Acton, MA (Onsite only) Rate: $60-65/hr. Mandatory Skills:... ...Kafka, Kinesis, Lambda, Spring Boot Responsibilities: # Lead and manage a team of engineers and Sr engineers by providing...
Kasmo Global
San Diego, CA
4 days ago
Big Data Engineer
...Big Data Engineer CA – Los Angeles, CA - Los Alamitos, CA - Norwalk, CA - El Segundo, CA - Sacramento, CA - San Diego, CA - San Francisco... ...Advanced Technology & Architecture (AT&A) practice, you will lead technology innovation for our clients through robust delivery...
Work experience placement
ClifyX
San Diego, CA
2 days ago
Senior Data Engineer, Solutions Architecture
$110k - $145k
...What The Role Is We are seeking a talented Senior Data Engineer to design, build, and maintain our data infrastructure supporting mission... ...monitoring, and incident response CI/CD & DevOps Lead continuous integration and deployment initiatives for Dagster...
H1b
Work at office
Local area
Relocation
Visa sponsorship
Flexible hours
Clearway Energy, Inc.
San Diego, CA
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!