Lead Data Engineer
VBeyond
Role: Lead Data Engineer (Priority 1)
Location: San Diego( Onsite only )
Job Type Contrct/Full Time Lead Data Engineer JD:
Role Summary
We are seeking a highly skilled Lead Data Engineer with deep, hands-on experience in building large-scale data ingestion pipelines, real-time streaming solutions, and high-performance data processing systems. The ideal candidate excels at writing clean, efficient code, refactoring complex systems, improving scalability and performance, addressing production issues, and delivering reliable data solutions across cloud platforms such as AWS, Azure, or GCP .
Key Responsibilities
Qualification: BE/BTech/MS in Computer Science or related field
Location: San Diego( Onsite only )
Job Type Contrct/Full Time Lead Data Engineer JD:
Role Summary
We are seeking a highly skilled Lead Data Engineer with deep, hands-on experience in building large-scale data ingestion pipelines, real-time streaming solutions, and high-performance data processing systems. The ideal candidate excels at writing clean, efficient code, refactoring complex systems, improving scalability and performance, addressing production issues, and delivering reliable data solutions across cloud platforms such as AWS, Azure, or GCP .
Key Responsibilities
- Design, build, and maintain high-volume data ingestion and processing pipelines for batch and real-time workloads.
- Implement and optimize real-time streaming pipelines using platforms such as Kafka .
- Develop scalable data solutions using Databricks, PySpark, Python, and SQL .
- Perform core refactoring to modernize and optimize existing pipelines and data services.
- Build robust, fault-tolerant pipelines capable of large-scale, high-throughput data processing.
- Write unit tests , automate validation, and ensure high code quality and reliability.
- Integrate pipelines into CI/CD workflows to streamline and automate deployment processes.
- Identify, troubleshoot, and fix production issues , ensuring system reliability and stability.
- Address performance bottlenecks and implement improvements for scalability, throughput, and efficiency.
- Work extensively across AWS, Azure, or GCP cloud environments and cloud-native data services.
- Design and orchestrate end-to-end pipelines using workflow and orchestration tools.
- Collaborate with Data Scientists and BI Engineers to deliver clean, analytics-ready datasets.
- Communicate complex technical topics clearly to non-technical stakeholders.
- 8+ years of hands-on data engineering experience building and maintaining large-scale data systems.
- Proven experience with high-volume data ingestion , ETL/ELT, and real-time data processing.
- Strong expertise with Kafka or similar streaming technologies.
- Advanced proficiency in Databricks, PySpark, Python, and SQL .
- Experience with core refactoring , improving code maintainability and system performance.
- Demonstrated ability to design and build scalable, low-latency data pipelines .
- Strong skills in debugging, performance optimization , and pipeline tuning.
- Hands-on experience with at least one major cloud platform: AWS, Azure, or GCP .
- Experience writing unit tests and integrating solutions with CI/CD pipelines .
- Strong problem-solving skills and excellent communication abilities.
Qualification: BE/BTech/MS in Computer Science or related field
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in San Diego, CA vacancy
- ...Data Center Engineer The engineer in this position should have a wealth of experience engineering/installing datacenter equipment. The engineer... ..." such as field engineers, installation supervisors, lead installers, and installation techs. Guides detail engineer through...SuggestedFor contractorsWork at officeLocal area
ADEX
San Diego, CA6 days ago - ...Data Engineer Big Data & AI Bootcamp · Atlanta, GA · Full-Time W2 Employment Stop waiting for the right opportunity. In 9 weeks, go from data-aware developer to a fully deployed Data Engineer — paid from day one, trained on real tools, and placed on Fortune 1000...SuggestedFull timeInternshipRelocation
HirePower Staffing Solution
San Diego, CA3 hours ago $115k
...Senior Data Engineer (Labs) Ignite Digital enables national security agencies to accelerate decisions, elevate operational outcomes and... ...partnership beyond presentations to inspire the confidence to lead in a digital-forward world. Ignite Digital. The Edge from Within...SuggestedWork at office1 day per weekIgnite Digital Services
San Diego, CA6 days ago- ...is seeking a highly skilled and motivated Data Scientist to join our team in developing... ...of the Navy, DoD, and Intel communities engineering and technology needs. You should possess... ...libraries, and visualization/BI tools to lead advanced analytics initiatives supporting...SuggestedFor contractorsLocal area
Xenith Solutions LLC
San Diego, CA2 days ago $200k - $240k
...Description We are seeking a Data Operations Engineer to design, build, and maintain real-time data ingestion pipelines. In this role, you will be responsible for the reliable flow of streaming data from a wide range of sources into our data platform, ensuring data quality...SuggestedSAIC
San Diego, CA3 days ago$183.36k - $205k
...Sr. Data Engineer III SHEIN TECHNOLOGY LLC is seeking a Sr. Data Engineer III in San Diego, CA to collaborate with global teams across data, security, infrastructure, and business functions to analyze data requirements and design scalable data engineering solutions....SHEIN
San Diego, CA1 day ago- ...more about why you want to be here! PURPOSE OF THE JOB The Data Engineer III will design, develop, and implement data pipelines, data... ...for enterprise-wide digital solutions and decision making. Leads the design and build of scalable data pipelines and solutions...Full timeWork at officeLocal area
ICW Group
San Diego, CA5 days ago $110k - $145k
What The Role Is We are seeking a talented Senior Data Engineer to design, build, and maintain our data infrastructure supporting mission... ...handling, SLA monitoring, and incident response CI/CD & DevOps Lead continuous integration and deployment initiatives for Dagster...H1bWork at officeLocal areaRelocationVisa sponsorshipFlexible hoursClearway Energy, Inc.
San Diego, CA4 days ago$175k - $225k
...The Marlin Alliance is seeking a forward-thinking Data Engineer in San Diego, CAto provide client support to our Navy client. This is an on-site role and applicants must have active DoD Secret Clearance. Incorporated in 2002, The Marlin Alliance is a digital transformation...Contract workWork at officeThe Marlin Alliance
San Diego, CA5 days ago$124k - $190k
...Senior Data Engineer Job Description Overview CoStar Group (NASDAQ: CSGP) is a leading global provider of commercial and residential real estate information, analytics, and online marketplaces. Included in the S&P 500 Index, CoStar Group is on a mission...Full timeWork from homeCoStar Group
San Diego, CA2 days ago- ...Xenith Solutions is looking for a skilled Data Scientist to create innovative solutions for the Navy and DoD. The role involves designing... ..., supporting continuous data feeds, and collaborating with engineers. A Bachelor’s degree in relevant fields and U.S. citizenship...
Xenith Solutions
San Diego, CA4 days ago - ...Bumble Bee Foods in San Diego is seeking a Data Engineer responsible for designing and building an enterprise data platform powering analytics and reporting. This role requires strong expertise in data pipelines, modeling, and Microsoft Fabric, focusing on delivering scalable...
Bumble Bee Foods
San Diego, CA1 day ago $122k - $130k
...Data Engineer The Data Engineer at Bumble Bee Seafoods is responsible for designing, building, and operating the enterprise data platform... ...in the U.S., Canada and over 50 markets globally under leading brands including Bumble Bee®, Brunswick®, Clover Leaf®, Snow's...Full timeLocal areaFlexible hoursBumble Bee Foods
San Diego, CA5 days ago$90k - $110k
...Information Assurance Engineer II Location US-CA-San Diego ID 2026-4520 Category IT / Cyber Security... ...cybersecurity compliance requirements and best practices. ~ May lead and direct the work of others and provide status updates to...Full timeRemote workAmerican Systems
San Diego, CA3 days ago- ...Data Engineer The Data Engineer independently designs, builds, and maintains the data infrastructure that enables data-driven decision-making across the Naval Aviation Enterprise. Working within the NAAC/FRAG team, this individual operationalizes raw data from NAE source...For contractorsFor subcontractor
T3W Business Solutions
San Diego, CA1 day ago $105k - $125k
...Overview SteerBridge seeks a highly skilled and motivated Data Engineer II to join our team supporting the F-35 AI/ML Spares Project.... ...analysts, and operational partners. Key Responsibilities • Lead end-to-end data pipeline operations - design, develop, and...SteerBridge
San Diego, CA4 days ago$84.9k - $110.4k
...passionate about serving their local communities. Join our team: Apply to our open position(s) below. OPEN POSITION ANNOUNCEMENT! Data Engineer Open until filled. Posted 4/29/2026 San Diego County - hybrid work schedule set by SDCP About the role : The San...Interim roleWork at officeLocal areaSan Diego Community Power
San Diego, CA1 day ago$200k - $240k
...Description We are seeking a Data Operations Engineer to design, build, and maintain real-time data ingestion pipelines. In this role, you will be responsible for the reliable flow of streaming data from a wide range of sources into our data platform, ensuring data...SAIC
San Diego, CA2 days ago$106.9k - $176.5k
...working world. We are seeking a highly skilled Senior Consultant Data Engineer with expertise in cloud data engineering, specifically... ...provide tailored solutions that meet their business objectives. Lead end-to-end data pipeline development, including data ingestion...Summer holidayFlexible hoursEY
San Diego, CA4 days ago$95.81k - $130.95k
...Position Summary Under the supervision of the CFO, the Data Engineer is responsible for designing, building, maintaining, and... ...dashboards, reports, visualizations, and decision-support tools. # Lead data migration and system transition efforts, including...Work at officeLocal areaSan Diego Workforce Partnership
San Diego, CA3 days ago$110k - $180k
...Senior Data Engineer (OpAI) The Marlin Alliance, Inc. is seeking a Senior Data Engineer (OpAI) to design, build, and operationalize advanced... ..., collaborate with cross-functional engineering teams, and lead data architecture improvements aligned with operational Navy...Contract workWork at officeThe Marlin Alliance
San Diego, CA1 day ago- ...Job Description Description: We are seeking a Senior Data Visualization & Analytics Developer to support a high-visibility... ...decision-making. You will work closely with a data engineering / structuring lead who prepares the backend datasets, while you focus on designing...Temporary workRemote workFlexible hours
Loch Harbour Group Inc
San Diego, CA19 days ago $124k - $190k
CoStar Group, Inc. is seeking a Senior Data Engineer in San Diego, California. In this role, you will design and build scalable data platforms crucial for high-volume applications and collaborate with engineering teams. Ideal candidates should have over 5 years of data...$6,492 per month
...Position Summary San Diego State University is hiring a Data Scientist to join Analytic Studies & Institutional Research (ASIR), a... ...solving data problems, designing scalable analytics workflows, engineering reliable datasets, and developing analytical products that people...Full timeContract workH1bSDSU Alumni
San Diego, CA3 days ago- ...A leading healthcare technology company is seeking a Staff Data Engineer to lead in the design and development of complex software applications. You will leverage your expertise in database architecture and ETL processes while working collaboratively to drive significant...Remote work
Exact Sciences
San Diego, CA3 days ago $77k - $202k
...Specialty/Competency: Data, Analytics & AI Industry/Sector: Not Applicable Time... ...At PwC, our people in data and analytics engineering focus on leveraging advanced technologies... ..., knowledge, and experiences you need to lead and deliver value at this level include but...Full timeH1bPwC
San Diego, CA5 days ago$154.4k - $231.6k
...eradicate the disease and the suffering it causes. As a Staff Data Engineer, this extremely seasoned professional will demonstrate... ...creativity in a wide range of technical areas. This role will have a lead role in the design, development, and testing of multiple...Full timeFor contractorsLocal areaNight shiftWeekend workExact Sciences
San Diego, CA4 days ago- Turquoise is looking for a Data Science Engineer to join their remote team in the United States. This role involves building data models, pipelines, and conducting analyses to support product decisions at Turquoise. Key responsibilities include partnering with cross-functional...Remote jobFlexible hours
- Booz Allen Hamilton is seeking a Data Visualization Analyst and Qlik Developer to transform complex data sets into insightful visualizations. You will work closely with clients to understand their needs and utilize your analytical skills to derive impactful solutions....Remote job
- LMI Government Consulting is seeking an AWS Cloud Data Engineer to support a Department of Defense data modernization effort. This role involves designing and implementing cloud-based data ingestion pipelines, requiring a strong foundation in AWS services and security compliance...Remote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
Related searches