Lead Data Engineer
VBeyond
Role: Lead Data Engineer (Priority 1)
Location: San Diego( Onsite only )
Job Type Contrct/Full Time Lead Data Engineer JD:
Role Summary
We are seeking a highly skilled Lead Data Engineer with deep, hands-on experience in building large-scale data ingestion pipelines, real-time streaming solutions, and high-performance data processing systems. The ideal candidate excels at writing clean, efficient code, refactoring complex systems, improving scalability and performance, addressing production issues, and delivering reliable data solutions across cloud platforms such as AWS, Azure, or GCP .
Key Responsibilities
Qualification: BE/BTech/MS in Computer Science or related field
Location: San Diego( Onsite only )
Job Type Contrct/Full Time Lead Data Engineer JD:
Role Summary
We are seeking a highly skilled Lead Data Engineer with deep, hands-on experience in building large-scale data ingestion pipelines, real-time streaming solutions, and high-performance data processing systems. The ideal candidate excels at writing clean, efficient code, refactoring complex systems, improving scalability and performance, addressing production issues, and delivering reliable data solutions across cloud platforms such as AWS, Azure, or GCP .
Key Responsibilities
- Design, build, and maintain high-volume data ingestion and processing pipelines for batch and real-time workloads.
- Implement and optimize real-time streaming pipelines using platforms such as Kafka .
- Develop scalable data solutions using Databricks, PySpark, Python, and SQL .
- Perform core refactoring to modernize and optimize existing pipelines and data services.
- Build robust, fault-tolerant pipelines capable of large-scale, high-throughput data processing.
- Write unit tests , automate validation, and ensure high code quality and reliability.
- Integrate pipelines into CI/CD workflows to streamline and automate deployment processes.
- Identify, troubleshoot, and fix production issues , ensuring system reliability and stability.
- Address performance bottlenecks and implement improvements for scalability, throughput, and efficiency.
- Work extensively across AWS, Azure, or GCP cloud environments and cloud-native data services.
- Design and orchestrate end-to-end pipelines using workflow and orchestration tools.
- Collaborate with Data Scientists and BI Engineers to deliver clean, analytics-ready datasets.
- Communicate complex technical topics clearly to non-technical stakeholders.
- 8+ years of hands-on data engineering experience building and maintaining large-scale data systems.
- Proven experience with high-volume data ingestion , ETL/ELT, and real-time data processing.
- Strong expertise with Kafka or similar streaming technologies.
- Advanced proficiency in Databricks, PySpark, Python, and SQL .
- Experience with core refactoring , improving code maintainability and system performance.
- Demonstrated ability to design and build scalable, low-latency data pipelines .
- Strong skills in debugging, performance optimization , and pipeline tuning.
- Hands-on experience with at least one major cloud platform: AWS, Azure, or GCP .
- Experience writing unit tests and integrating solutions with CI/CD pipelines .
- Strong problem-solving skills and excellent communication abilities.
Qualification: BE/BTech/MS in Computer Science or related field
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in San Diego, CA vacancy
- ...will collaborate in the design, creation, and enhancement of our data infrastructure, which underpins the productization of our... ...numerous other devices. We are seeking a highly skilled Data Software Engineer to design, develop, and maintain scalable data infrastructure and...Suggested
Apple
San Diego, CA3 days ago - ...duties may be assigned. The Database Engineer III is responsible for key areas which includes... ...activities including production data modification requests Gains a thorough... ...with team members and technical project lead to ensure component design fits in with the...SuggestedLocal areaRemote workMonday to FridayNight shiftWeekend work
My3Tech Inc
San Diego, CA3 days ago $105k - $130k
...* *We are unable to sponsor or take over sponsorship of an employment visa at this time* RIVO is seeking an experienced Data Engineer to join our Data team and help build upon the infrastructure that drives analytics, reporting, and data-driven decision-making...SuggestedRemote workWork visaWeekend workRivo Holdings
San Diego, CA4 days ago- ...Flink Data Engineer Location: MTV/San Diego CA Experience: 8-10 years of experience Must have: Expertise in Apache Flink Strong programming skills in Java or Scala and SQL Knowledge of stream processing and batch processing Experience working with...Suggested
United IT
San Diego, CA4 days ago - ...Position : Senior Data Engineer – Remote Location : San Diego, CA Duration : 5 Months Total Hours/week : 40.00 1st Shift Client... ...About the role: As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and...SuggestedContract workWork experience placementRemote workDay shift
MILLENNIUMSOFT
San Diego, CA1 day ago $60 - $65 per hour
...AWS Data Sr Engineers Client: INSULET Location: San Diego, CA or Acton, MA (Onsite only) Rate: $60-65/hr. Mandatory Skills:... ...Kafka, Kinesis, Lambda, Spring Boot Responsibilities: # Lead and manage a team of engineers and Sr engineers by providing...Kasmo Global
San Diego, CA4 days ago- ...Big Data Engineer CA – Los Angeles, CA - Los Alamitos, CA - Norwalk, CA - El Segundo, CA - Sacramento, CA - San Diego, CA - San Francisco... ...Advanced Technology & Architecture (AT&A) practice, you will lead technology innovation for our clients through robust delivery...Work experience placement
ClifyX
San Diego, CA2 days ago $110k - $145k
...What The Role Is We are seeking a talented Senior Data Engineer to design, build, and maintain our data infrastructure supporting mission... ...monitoring, and incident response CI/CD & DevOps Lead continuous integration and deployment initiatives for Dagster...H1bWork at officeLocal areaRelocationVisa sponsorshipFlexible hoursClearway Energy, Inc.
San Diego, CA2 days ago- ...staffing and recruiting solutions, has an immediate opening for a Data Engineer. Data Warehouse. Ideally Cloud & Google BigQuery, but any... ...can do the work as needed, but importantly we are looking for a lead type person that can be relied on and leveraged to help create...For contractorsLocal areaImmediate start
KORE1 Technologies
San Diego, CA2 days ago - ...Senior Data Engineer (Labs) Ignite Digital enables national security agencies to accelerate decisions, elevate operational outcomes and... ...partnership beyond presentations to inspire the confidence to lead in a digital-forward world. Ignite Digital. The Edge from Within...For contractorsWork at office1 day per week
Ignite Digital Services
San Diego, CA4 days ago - ...Data Engineer Location – San Diego, or San Jose, CA Duration – 6 months Rate – DOE $/hr. on w2 US Citizen, Green Card, TN, only Primary... ...~ Designing data models ~ Proficient understanding and tech-lead level experience in one or more programming languages (Java,...
Staffing the Universe
San Diego, CA4 days ago - ...Senior Data Engineer AArete is one-of-a-kind when it comes to consulting firm culture. We're a global, innovative management and technology consulting firm with offices in the U.S., India, and Europe. Our name comes from the Greek word for excellence: "Areté." And...Contract workWork at officeFlexible hoursNight shift
AArete
San Diego, CA2 days ago $105k - $125k
...Overview SteerBridge seeks a highly skilled and motivated Data Engineer II to join our team supporting the F-35 AI/ML Spares Project.... ...analysts, and operational partners. Key Responsibilities • Lead end-to-end data pipeline operations - design, develop, and...SteerBridge
San Diego, CA2 days ago- ...is seeking a highly skilled and motivated Data Scientist to join our team in developing... ...of the Navy, DoD, and Intel communities engineering and technology needs. You should possess... ...libraries, and visualization/BI tools to lead advanced analytics initiatives supporting...For contractorsLocal area
Xenith Solutions LLC
San Diego, CA12 hours ago $61.9k - $141k
...Data Engineer The Opportunity: Ever-expanding technology like IoT, machine learning, and artificial intelligence means that there's more structured and unstructured data available today than ever before. As a data engineer, you know that organizing data can yield pivotal...Full timeContract workPart timeLocal areaNavstar
San Diego, CA4 days ago$122k - $130k
...Data Engineer The Data Engineer at Bumble Bee Seafoods is responsible for designing, building, and operating the enterprise data platform... ...in the U.S., Canada and over 50 markets globally under leading brands including Bumble Bee®, Brunswick®, Clover Leaf®, Snow's...Full timeLocal areaFlexible hoursBumble Bee Foods
San Diego, CA3 days ago$110k - $160k
...recruiter to learn more.Base pay range$110,000.00/yr - $160,000.00/yrThis company is a value based care provider that manages a number of data ingestion points for third party API integration. Great work life balance and remote work!This Jobot Job is hosted by Alex...Local areaRemote workJobot
San Diego, CA3 days ago- A leading agriculture technology firm is seeking a Senior Data Engineer to join their digital team. This role involves designing and maintaining data pipelines that support data-driven decision-making for farmers. The ideal candidate should have strong experience in data...Remote work
Military, Veterans and Diverse Job Seekers
San Diego, CA2 days ago - ...currently 250 downstream applications that depend on the existing student information system, so these apps need to connect to a new data warehouse once the new system goes live. To support this effort, the SQL developer will extract data from the existing data...
Insight Global
San Diego, CA1 day ago - ...About the job Senior Data Engineer (Remote) Technology is the next frontier of change in Agriculture and will be central to delivering... ...and operational efficiency of data and analytical solutions Lead the evaluation, implementation, and deployment of emerging tools...Remote work
Military, Veterans and Diverse Job Seekers
San Diego, CA2 days ago $200k - $240k
...Description We are seeking a Data Operations Engineer to design, build, and maintain real-time data ingestion pipelines. In this role, you will be responsible for the reliable flow of streaming data from a wide range of sources into our data platform, ensuring data...SAIC
San Diego, CA5 days ago$93.9k - $156.8k
...Information Assurance Engineer II Location US-CA-San Diego ID 2026-4027 Category IT / Cyber Security / Network Systems Position Type Full-Time Remote No Clearance Required Secret Overview...Full timeFor contractorsRemote workAmerican Systems
San Diego, CA1 day ago$200k - $240k
...Job Description Description We are seeking a Data Operations Engineer to design, build, and maintain real-time data ingestion pipelines. In this role, you will be responsible for the reliable flow of streaming data from a wide range of sources into our...Science Applications International Corporation
San Diego, CA12 hours ago$106.9k - $176.5k
...help to build a better working world. Technology – Data and Decision Science – Data Engineering – Senior We are seeking a highly skilled Senior Consultant... ...solutions that meet their business objectives. Lead end-to-end data pipeline development, including data...Summer holidayFlexible hoursEY
San Diego, CA12 hours ago$101k - $172k
...eradicate the disease and the suffering it causes. As a Senior Data Engineer, this seasoned professional will demonstrate competence and... ...in a wide range of technical areas. This role will have a lead role in the design, development, and testing of software applications...Full timePart timeFor contractorsLocal areaWork visaNight shiftWeekend workExact Sciences
San Diego, CA4 days ago$84.9k - $110.4k
...passionate about serving their local communities. Join our team: Apply to our open position(s) below. OPEN POSITION ANNOUNCEMENT! Data Engineer Open until filled. Posted 4/29/2026 San Diego County - hybrid work schedule set by SDCP About the role : The San...Interim roleWork at officeLocal areaSan Diego Community Power
San Diego, CA4 days ago- ...Data Engineer The Data Engineer independently designs, builds, and maintains the data infrastructure that enables data-driven decision-making across the Naval Aviation Enterprise. Working within the NAAC/FRAG team, this individual operationalizes raw data from NAE source...For contractorsFor subcontractor
T3W Business Solutions
San Diego, CA4 days ago - ...Position: Sr. Data Engineer Type: Remote Duration: Long Term Requirement: Candidate must have: Proficiency in Python, SQL, GCP tools, clean rooms (Google, Databricks), Kafka or similar streaming tools, and strong understanding of retail media...Local areaRemote work
E-Solutions
La Jolla, CA2 days ago $61.9k - $141k
...Data Engineer The Opportunity: Ever-expanding technology like IoT, machine learning, and artifi cia l intelligence means that there's more structured and unstructured data available today than ever before. As a data engineer, you know that organizing data can yield...Full timeContract workPart timeWork at officeLocal areaRemote workBOOZ, ALLEN & HAMILTON, INC.
San Diego, CA3 days ago$175k - $225k
...The Marlin Alliance is seeking a forward-thinking Data Engineer in San Diego, CAto provide client support to our Navy client. This is an on-site role and applicants must have active DoD Secret Clearance. Incorporated in 2002, The Marlin Alliance is a digital transformation...Contract workWork at officeThe Marlin Alliance
San Diego, CA3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
Related searches
- clinical data San Diego, CA
- master data coordinator San Diego, CA
- clinical data coordinator remote San Diego, CA
- data intern San Diego, CA
- data cabling installation San Diego, CA
- data collection researcher San Diego, CA
- data technician San Diego, CA
- data mining San Diego, CA
- minimum data set coordinator San Diego, CA
- data reviewer San Diego, CA