Senior Data Engineer
Compunnel
Job Summary
We are seeking an experienced Senior Data Engineer with strong Spark and streaming expertise to build real-time, scalable data pipelines using technologies such as Spark, Kafka, and cloud services. The ideal candidate will have experience designing and optimizing data solutions on GCP to support analytics and machine learning workloads.
Key Responsibilities
• Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies such as Kafka and Flink.
• Build and optimize scalable data architectures, including data lakes, data warehouses such as BigQuery, and streaming platforms.
• Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness.
• Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.
• Develop and maintain scalable data processing solutions supporting analytics and machine learning workloads.
Required Qualifications
• Minimum 8 years of total IT experience.
• Minimum 4 years of recent experience working with Google Cloud Platform (GCP).
• Strong proficiency in Python and SQL, with experience in Scala and/or Java preferred.
• Strong expertise in Apache Spark, including Spark SQL, DataFrames, and Spark Streaming.
• Hands-on experience with streaming technologies such as Apache Kafka, Flink, or Pub/Sub.
• Experience designing and implementing batch and real-time data pipelines.
• Experience with GCP data services and cloud-based data engineering solutions.
• Knowledge of data warehousing technologies such as Snowflake and Redshift.
• Knowledge of NoSQL databases.
• Strong understanding of data quality, monitoring, and alerting practices.
Preferred Qualifications
• Experience with Azure data services.
• Experience with Apache Airflow.
• Experience with Databricks.
• Experience with Docker and Kubernetes.
• Experience optimizing data pipelines and distributed processing workloads for performance and cost efficiency.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer in Sunnyvale, CA vacancy
$185k - $230k
The OpportunityWe are looking for a Senior Data Engineer to join our Data Platform team and build the core data foundations that power analytics, experimentation, and decision-making across the company. In this role, you will design and own foundational data models, pipelines...SeniorPermanent employment$175k - $221k
...feed ranking decision, every recommendation, every A/B test, and every ML model we ship runs on the data infrastructure you'd own. We're looking for a senior engineer to build and scale the batch and streaming pipelines at the core of that platform — someone who takes...SeniorFull timeLocal areaWork from home$152k - $253k
...deepen our efforts in the autonomous driving field and strive to bring sustained growth to our customers.We're looking for a Senior Data Analysis Engineer, AD critical metrics with strong technical ability to deeply understand architectures and strong scripting experience to...SeniorFull time$168k - $270.25k
The NVIDIA Operations organization is seeking an experienced data engineering professional for the position of Senior Data Engineer, Engineering Data Analytics. As a member of our team, you will be an integral part of building cloud-based data platforms that support engineering...SeniorFull time$130k - $200k
Santa Clara, CAData Engineering - Data Infrastructure /Full-time /HybridPlusAI is a Physical AI company pioneering AI-based virtual driver software for factory-built autonomous trucks. Headquartered in Silicon Valley with operations in the United States and Europe, Plus...SeniorFull time- DescriptionGeneral information:Hitachi Energy is seeking for a Senior Data Management Engineer for it's San Jose, CA location.Your Responsibilities:Analyze and understand customer engineering, business, and software requirements, propose technically feasible solutions,...Senior
$117k - $234k
...time / Regular/PermanentCompany: WalmartBusiness Segment: Home OfficePosition Summary...What you'll do...Role summary: As a Senior Data Engineer, you will design, develop, and maintain scalable data pipelines and models that support critical business decisions....SeniorFull timeTemporary workPart time$184k - $287.5k
Senior Data Center Performance Engineer - Benchmarking and Optimization page is loaded## Senior Data Center Performance Engineer - Benchmarking and Optimizationlocations: US, CA, Santa Clara: US, Remotetime type: Full timeposted on: Posted Yesterdayjob requisition id: JR...SeniorNVIDIA Corporation
Santa Clara, CA1 day ago$174.72k - $295.68k
...transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.We are seeking a Machine Learning Data Curation Engineer to spearhead the data pipeline development and dataset management for our core AI initiatives. You will bridge the gap between...SeniorFull time$180k - $258.75k
...project is harmonious with TRI’s robotics divisions' efforts in Diffusion Policy and Large Behavior Models.We are looking for a Senior Data Engineer to design and build the foundational data infrastructure and tools that power our autonomy research and development...SeniorFull timeLocal areaShift work$174.72k - $295.68k
...future of transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.We are looking for a Data Infrastructure Engineer with engineering background and development skills. As a Data Infrastructure Engineer, your role is to identify corner cases...SeniorFull time- ...ID.me is seeking a Senior Analytics Engineer to support the data infrastructure in Mountain View, CA. This role involves leading efforts to develop analytics and dashboards using Snowflake and dbt. The ideal candidate has 5+ years of experience in data analytics and SQL...Senior
ID.me Inc
Mountain View, CA3 days ago - ...Senior Data Engineer - GCP Location: Mountain View CA (Onsite) What You’ll Do Develop and enhance Python frameworks and libraries to support data processing, quality, lineage, governance, analysis, and machine learning operations. Design, build, and maintain...Senior
United IT
Mountain View, CA1 day ago - ...39 delivery centers worldwide, gaining exposure to complex engineering challenges that redefine industrial standards. If you are a... ...this is where you belong. We are currently searching for a Senior AWS Data Engineer: The Challenge (Responsibilities) Design,...SeniorRemote workWorldwide
Sequoia Connect
Mountain View, CA5 days ago - Apple Inc. is seeking a Data Engineer for the Find My team in Cupertino, CA. You will design analytics pipelines, translate needs from multiple partners into high-quality data artifacts, and build scalable solutions powering millions of users. You will collaborate with...Senior
- ...millions of patients worldwide.We’re a team of engineers, clinicians, and innovators united by one... ...in robotic surgery requires high-quality data across the robotic platform, the surgical... ...and the operating room environment. The Senior AI/Data Science Engineer will advance the...SeniorLocal areaWorldwideFlexible hours
- Yahoo is seeking a Senior Data Engineer for the Mail Analytics Infrastructure. You will define the data ontology, partner with Data Science and ML teams, and lead core dataset roadmaps for Yahoo Mail. You will design and maintain high-reliability data pipelines across...Senior
- CoreWeave, a leading AI cloud provider, seeks a Senior Data Engineer to own the data lake and analytics stack powering observability for its global hardware fleet. You will maintain Iceberg, Trino, Airflow, Spark, and Superset, while delivering ETL pipelines, data models...Senior
- NVIDIA Corporation in Santa Clara, CA is seeking a Senior Software Engineer for At Scale Compute Analysis to study large-scale datacenter workloads on GPU-accelerated clusters and turn telemetry data into actionable insights. You will work with OS, container, GPU, and...SeniorRemote job
- Samsung Electronics Perú is seeking a Sr. Data Engineer to contribute to scalable data pipelines and support the Performance & Reach engineering organization. You will work with large advertising datasets and collaborate with cross-functional teams to deliver data solutions...Senior
- Socket.dev is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and data platforms. You will enable reliable analytics, ML, and business intelligence across the organization, leveraging Python and distributed systems in a modern data...Senior
- brevian-careers is looking for an experienced Data Engineer to join our Engineering team in Sunnyvale, California. The ideal candidate will have a strong background in developing scalable, high-performance data systems, with a particular focus on enterprise data systems...Senior
- Bright Vision Technologies is seeking an experienced Big Data Engineer to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. This remote, full-time role supports enterprise analytics and ML workloads. Ideal candidate has 6...SeniorRemote jobFull time
- Apple is seeking a senior data engineer to design and maintain a massive real-time data platform that converts multimodal data into a searchable foundation. You will build end-to-end pipelines, craft analytics schemas, and enable ML/LLM inference in production, enabling...Senior
- ...virtual driver software for factory-built autonomous trucks, seeks engineers to own the loop end-to-end — metrics, pipelines, and autonomy judgments. You will work primarily in Python across large-scale data processing, geospatial analytics, evaluation frameworks, and LLM...Senior
- Redolent Infotech Pvt. Ltd. in Sunnyvale, CA seeks a Sr. Big Data Engineer to design, build and operate petabyte-scale data systems, with a strong focus on real-time pipelines and ML infrastructure. You will collaborate with engineers, product managers, BI developers and...Senior
$117k - $234k
...OfficePosition Summary...What you'll do...Role summary:The Senior Software Engineer will lead the delivery of scoped features and models through... ...’s mission and operational goals.About the team:The MarTech Data Engineering team develops scalable, multi-tenant engineering...SeniorFull timeTemporary workPart time- A leading technology firm based in Mountain View, California is seeking a Data Engineer to develop and enhance Python frameworks for data processing and build scalable data pipelines on Google Cloud Platform. The ideal candidate should have extensive experience in GCP services...Senior
- Apple Inc. in Cupertino, CA is seeking a senior software and data engineer to build and operate large-scale ML data pipelines. You will acquire, annotate, and ensure quality of multimodal datasets powering features across iPhone, iPad, Mac, and more. The role emphasizes...Senior
- Redolent Infotech Pvt. Ltd. is seeking a Sr. Data Engineer in Sunnyvale, CA to design and implement data pipelines using SQL and Hadoop ecosystems. The role focuses on large data sets, distributed computing, and real-time ingestion with Kafka, with development in Java,...SeniorContract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer. Be the first to apply!
Related searches
- aws data engineer Sunnyvale, CA
- big data devops engineer Sunnyvale, CA
- big data developer Sunnyvale, CA
- software data engineer Sunnyvale, CA
- data center engineer Sunnyvale, CA
- big data cloud engineer Sunnyvale, CA
- director data engineering Sunnyvale, CA
- hadoop big data developer Sunnyvale, CA
- big data engineer Sunnyvale, CA
- senior data integration developer Sunnyvale, CA