Lead Data Engineer - Real-Time Spark & GCP Pipelines
UST Inc
UST is seeking a Senior Lead I Data Engineer to design, develop, and maintain real-time and batch data pipelines using Spark and Kafka, with cloud services (GCP) and data warehousing in BigQuery. You will optimize performance and ensure data quality across scalable architectures. The role requires 8+ years IT experience, 4+ years GCP, and a BS in CS/IT/Engineering. Sunnyvale, CA-based, full-time with competitive compensation and comprehensive benefits. #J-18808-Ljbffr UST Inc
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer - Real-Time Spark & GCP Pipelines in Sunnyvale, CA vacancy
- Compunnel, Inc. is seeking a Data Engineer in Sunnyvale, California to build scalable, real-time data pipelines. The ideal candidate will work with technologies like Spark, Kafka, and GCP to facilitate data processing for analytics and machine learning. Key responsibilities...Pipeline
- ...in Sunnyvale, CA seeks a Sr. Big Data Engineer to design, build and operate petabyte... ...systems, with a strong focus on real-time pipelines and ML infrastructure. You will collaborate... .... 6-8 years of experience, Hadoop/Spark/Kafka and cloud tech (GCP/Azure) are required. #J-18808-...Pipeline
- SpaceXAI is seeking an Analytics Engineer in Mountain View, CA, to design and maintain scalable data systems that enable high-impact quantitative analysis and... ...decision-making. You will implement end-to-end data pipelines using Spark, Flink, and related technologies, develop...Pipeline
- NVIDIA Corporation is seeking a System Software Engineer for Vision AI to craft high‑performance pipelines that process video, image, and 3D data in real‑time across edge and cloud environments. You will collaborate with perception, simulation, and platform teams to translate...Pipeline
- ...are looking for a highly skilled GCP Data Engineer with strong expertise in Scala, Apache Spark, and Google Cloud Platform (GCP... ...building cloud-native data pipelines, processing big data workloads,... ...Spark.Build and optimize batch and real-time data processing solutions on...PipelineRemote work
- A leading data firm located in California is seeking a Data and Business Intelligence Associate. The role involves designing and maintaining data tools and analytics dashboards for real-time insights. Candidates should have at least 4 years of experience in SQL and business...Pipeline
- ...Mountain View, CA is seeking a Backend Engineer with strong Go skills to design and operate... ...‑functional teams to deliver reliable real‑time experiences. You’ll work on Go‑based backend... ...partner APIs, and build telemetry pipelines that surface live vehicle #J-18808-Ljbffr...Pipeline
- ...Sunnyvale, CA is seeking an experienced engineer to unify computer vision... ...firmware, and low-level software for real-time, on-device perception across... .... The ideal candidate will lead ISP and CV feature integration, optimize pipelines, and translate high-level product...Pipeline
- ...expression and facial animation. You will contribute to next-generation interactive computing using Maya within real-time pipelines. Collaborate with engineers, designers and directors while exploring new technologies to push animation excellence and deliver polished...Pipeline
- Nuro in Mountain View is seeking a backend engineer to design and operate the real-time, distributed systems powering our autonomous driving platform.... ...partner APIs, surface live vehicle state via telemetry pipelines, and contribute to end-to-end product ownership from...Pipeline
- ...Data Engineer GCP Sunnyvale, California, United States Job Title: Data Engineer Location: Sunnyvale Ca Visa... ...Details: 5 years of SQL queries Tech stack Airflow pipeline development GCP dataproc/ BQ Expertise Hive/Spark Python/Java expertise (scala) GoogleAds...PipelineH1b
Interaction24 LLC
Sunnyvale, CA1 day ago - ...Big Data Engineer: Must Have: • Proficiency... ...Hadoop, Apache Spark (Scala preferred),... ...on the cloud (GCP preferred, AWS,... ...build batch data pipelines with strong focus... ...with minimal ramp up time will be crucial to... ...building complex near real time (NRT)...Pipeline
Trinity Technology Solutions
Sunnyvale, CA1 day ago $139k - $257.55k
...Operations and Engineering team in Growth... ...(GMI) handles data architecture, transformation... ...an Engineering Lead to manage the... ...develop data pipelines, refine ETL... ...SQL, Python, Spark, and data... ...platforms (AWS, Azure, GCP) and cloud-... ...least the date and time stated above in...PipelineTemporary workLocal areaAdobe
San Jose, CA4 days ago- ...DIRECT client Title:Sr. Big Data Engineer Location: Sunnyvale, CA... ...scale. You will be focusing on real-time data pipelines, streaming analytics,... ...Hadoop, Hive, Kafka, Presto, Spark, just to name a few Highly... ...with Cloud Technologies ( GCP, Azure) Experience with Relational...Pipeline
Redolent Infotech Pvt. Ltd.
Sunnyvale, CA2 days ago - ...Principal Software Engineer who is a self-... ...and reliable APIs, data-pipelines, services, and applications... ...you'll do: • Lead and mentor a cross... ...batch or real-time customer data integrations... ...(Azure or GCP) and distributed systems (Hadoop, Spark, Kafka) In depth...PipelineLocal area
My3Tech Inc
Sunnyvale, CA3 days ago $143k - $286k
...summary: Staff Data Engineer designs and... ...across streaming, near-real-time, micro-batch, and... .... Modernizes pipelines supporting reporting... ...markets. Optimize Spark, Kafka, BigQuery,... ...incident response. Lead domain design... ...on experience with GCP or equivalent cloud...PipelineFull timeTemporary workPart time$110k - $220k
.../yearType: Full time / Regular/PermanentCompany... ...:At Walmart, data is at the heart... ...As a Staff Data Engineer, you'll be a... ...platforms and pipelines that power [Customer... ...batch and real-time workloads.Develop... ...as Airflow, GCP, BigQuery, Hive,... ...frameworks such as Spark, Flink, or Beam....PipelineFull timeTemporary workPart timeWorldwide- ...frameworks and libraries to support data processing, quality, lineage... ...scalable and efficient data pipelines on GCP. Implement robust monitoring... ...Contribute to shared Data Engineering tooling & standards to... ...such as TensorFlow, Apache Spark, and scikit-learn. Bachelor’...PipelinePermanent employmentContract workLocal area
Robotics Prcocess Automation, LLC
Mountain View, CA1 day ago $143k - $286k
...yearType: Full time / Regular/PermanentCompany... ...do...Principal Data Engineer - Agentic Data... ...that enable real-time decision-... ...across domains.Lead Agentic Data... ...-time feature pipelines, vector... ...native ecosystems (GCP or Azure preferred... ...(Kafka, Spark Structured Streaming...PipelineFull timeTemporary workPart time$180k
...based in Palo Alto is seeking to hire an experienced engineer to work on multimodal AI systems. The ideal... ...optimizing systems for multimodal learning, developing data pipelines, and advancing capabilities in real-time interactions. The role offers competitive compensation...Pipeline$150k - $220k
...company that's making a real difference in how... ...a highly skilled Data Engineer to design, build,... ...-grade data pipelines that process and transform... ...Dask or Apache Spark. Maintain... ...in Go or Rust. ~ GCP Proficiency: Hands... ...covered. Paid Time Off (PTO): Enjoy a...PipelineWork at officeRemote workFlexible hoursVersa Networks
Santa Clara, CA2 days ago$72k - $108k
...Role description Data Engineer Lead I - Data Engineering Who We Are: Born digital,... ...searching for a Data Engineer with Spark & Streaming skills builds real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and...PipelineFull timeTemporary workPart timeWork at officeLocal areaFlexible hoursUST Inc
Sunnyvale, CA3 days ago- A leader in AI testing is looking for a skilled Data Engineer (AI Pipelines) for a 12-month remote contract. You will build scalable ETL/ELT pipelines... ...of Data Engineering experience, with expertise in SQL, Spark, Python, and tools like Airflow and Snowflake. This role demands...PipelineRemote jobContract work
- Strong expertise in data modeling (dimensional modeling... ...We are seeking a Lead Data Engineer to architect, build,... ...optimization of scalable data pipelines supporting ingestion,... ...; experience with Spark or distributed... ...Mobility Standing 20% of time Sitting 70% of time...Pipeline
$152k - $230k
...the GPU in 1999 sparked the growth of... ...Design-for-Test Engineering at NVIDIA works... ...and post-silicon data analytics -... ...uphold sturdy data pipelines and ETL... ...platforms (AWS, Azure, GCP)Design and... ...architecture for real-world systemsExperience... ...US, CA, Santa ClaraType: Full timePipelineFull time- Job Summary Seeking a Data Engineer with Spark & Streaming skills to build real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML. Key Responsibilities Design, develop, and...Pipeline
$204k - $252k
...navigate and respond to dynamic real-world disruptions (e.g., road... ...events) Develop automated pipelines and operational tooling to scalably... ...observations and third-party data (including contributions from... ...weather data or handling real-time routing optimization AV...PipelineLocal area- ...assistant trustworthy and debuggable —in real time and post-call. This engineer builds the visibility layer across... ...automated triage + notification pipelines that surface issues to the right module... ...Comfortable designing both: Backend data pipelines Frontend dashboards in...PipelineImmediate start
$192.2k - $260k
...research and development of real-time multimodal conversational AI.... ...conversational AI, with the compute, data, and runway to pursue... ...work closely with inference engineers to ensure your models are designed... ...parts of the post-training pipeline from SFT through RL alignment...PipelineLocal areaFlexible hours- ...Senior Data ScientistThis role has... ...development and engineering duties related... ...classify them in real-time. The Senior... ...such as Apache Spark, Apache Storm and... ...data processing pipeline required efficacy... ...deploying and leading complete ML platforms in AWS/GCP/Azure.What We...PipelineWork at office
Hewlett Packard Enterprise Development LP
Sunnyvale, CA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer - Real-Time Spark & GCP Pipelines. Be the first to apply!
Related searches
- data loss prevention analyst Sunnyvale, CA
- data labeling Sunnyvale, CA
- data collection Sunnyvale, CA
- data technician Sunnyvale, CA
- data governance Sunnyvale, CA
- data visualization Sunnyvale, CA
- test data management Sunnyvale, CA
- data processor Sunnyvale, CA
- clinical data Sunnyvale, CA
- data tech Sunnyvale, CA