Lead Data Engineer

Veridic Solutions

Key Responsibilities
• Lead the architecture, design, and implementation of large-scale distributed data platforms.
• Build and optimize high-performance data pipelines using PySpark and distributed computing frameworks.
• Design and manage Data Lakehouse solutions using Apache Iceberg for schema evolution, time travel, partition optimization, and data governance.
• rchitect and optimize federated query solutions using Trino across multiple data sources.
• Drive enterprise data migration and modernization initiatives from traditional warehouses to Lakehouse architectures.
• Partner with business stakeholders, product owners, architects, and customer teams to translate business requirements into scalable technical solutions.
• Establish best practices for data modelling, performance tuning, security, governance, and observability.
• Mentor a team of data engineers and provide technical leadership across delivery streams.
• Evaluate and incorporate emerging technologies in Data Engineering, Analytics, and AI.
• Support pre-sales discussions, solution proposals, estimations, and customer presentations.

Technical Skills
• 10+ years of experience in Data Engineering and Big Data ecosystems.
• Expert knowledge of PySpark and Spark SQL.
• Strong hands-on experience with Apache Iceberg.
• Strong experience with Trino (Presto) query engine.
• Experience building large-scale batch and near-real-time pipelines.
• Strong SQL skills and query optimization expertise.
• Experience with Data Lake technologies and cloud-based analytics platforms.
• Knowledge of data modelling and distributed storage concepts.
• Experience with orchestration tools such as Airflow or equivalent.
• Experience working with file formats such as Parquet, ORC, and Avro.
• Exposure to CI/CD, Git, DevOps, and Infrastructure as Code practices.

Experience in one or more cloud platforms:
• WS (EMR, Glue, S3, Athena, Lake Formation)
• zure (Databricks, Data Factory, ADLS)
• GCP (Dataproc, BigQuery, GCS)

Leadership & Strategic Expectations
• bility to engage with senior customer stakeholders.
• Drive technical roadmaps and platform modernization strategies.
• Lead architecture reviews and governance forums.
• Identify opportunities for automation, optimization, and AI-driven solutions.
• Strong communication and presentation skills.
• bility to influence decisions across engineering, product, and business teams.

Apply

Vacancy posted 2 days ago

Similar jobs that could be interesting for youBased on the Lead Data Engineer in Sunnyvale, CA vacancy

Lead Data Engineer
$251k - $310k
...across 15+ U.S. states. This role follows a hybrid work schedule and reports to Sr. Staff TLM. You will: Lead, mentor, and grow a team of data engineers responsible for the Waymo Commercialization Data Lake and associated data pipelines. Define and drive the...
Suggested
Full time
Remote work
Waymo
Mountain View, CA
13 hours ago
Lead Data Engineer
...Role: Lead Data Engineer Location: Mountain View, CA / Dallas, TX (Onsite) About the Role Data team builds and operates the data marts and pipelines that power HR/Workforce reporting and analytics across Intuit. We're looking for a hands-on Data Engineer...
Suggested
United IT Solutions
Mountain View, CA
4 days ago
Lead Data Engineer
Job Description RadiumOne is seeking a Lead Data Engineer to join our Data Technology team with the responsibility of designing and implementing scalable data infrastructure, as well as implementing data transformations developed by data scientists as scalable and robust...
Suggested
Stepping Up Solutions
Sunnyvale, CA
3 days ago
Lead Data Engineer
Lead Data Engineer We are seeking a Lead Data Engineer to architect, build, and lead the development of scalable, cloud‑based data platforms that support enterprise analytics, operational reporting, and advanced data use cases. This role provides technical leadership in...
Suggested
Qcells North America
Santa Clara, CA
1 day ago
Data Engineer
$138.7k - $206.95k
...Databricks-based pipelines, medallion-layer data products, and enterprise integrations... ...closely with product owners, architects, data engineers, report and analytics teams, and source-... ...and we embrace the responsibility to lead the change that will make our world better...
Suggested
Full time
H1b
Local area
Work from home
Relocation package
Flexible hours
General Motors
Mountain View, CA
22 hours ago
Lead Data Engineer
A leading digital financial platform in Palo Alto is seeking a Machine Learning Engineer to design scalable data ingestion pipelines and monitor data quality. The role requires over six years of experience in machine learning and data engineering, with expertise in Python...
Nubank
Palo Alto, CA
12 hours ago
Lead Data Engineer
Obsidian is seeking a Lead Analytics Engineer to own the data warehouse and analytics foundation. You will own the dbt project, the warehouse architecture, and the semantic layer that supports executive dashboards, GTM workflows, and internal AI agents. You will report...
Obsidian
Palo Alto, CA
13 hours ago
Lead Data Engineer
$139k - $257.55k
...yours! The Opportunity The Marketing Operations and Engineering team in Growth Marketing and Insights (GMI) handles data architecture, transformation, and engineering for... ..., and lifetime value. We want an Engineering Lead to manage the data enabling the team’s work. You...
Temporary work
Local area
Adobe
San Jose, CA
3 days ago
(USA) Senior, Data Scientist
$117k - $234k
Position Summary... What you'll do... Role summary: As a Senior Data Scientist at Walmart, you will leverage advanced analytical... ...AI, the team combines innovative AI research with scalable engineering. Their efforts enhance customer engagement, conversion rates, and...
Full time
Temporary work
Part time
Walmart
Sunnyvale, CA
2 days ago
Data Scientist
$119.5k - $164.5k
...Materials is a global leader in materials engineering solutions used to produce virtually... ...while learning every day in a supportive leading global company. Visit our Careers website... ...analysis and deliver recommendations in a data-driven manner Lead thoughtful and...
Full time
Work experience placement
Relocation
Applied Materials
Santa Clara, CA
1 day ago
Senior Data Scientist
$121k - $198k
...transform the way the world works. Job Description LinkedIn's Data Science team leverages big data to empower business decisions and... ...programs, LinkedIn offers countless ways for an ambitious data engineer to have an impact and transform your career. We are now looking...
Full time
For contractors
Work experience placement
Work at office
Flexible hours
LinkedIn
Mountain View, CA
2 days ago
Data Scientist
$104k - $160.8k
...money from the bank, we're involved. If you want to make an impact on a global scale, come make a difference at Fiserv. Job Title Data Scientist What does a great Data Scientist do? The Risk Analysts in EPayments are the data scientists responsible for the identification...
Full time
Fiserv
Sunnyvale, CA
2 days ago
Senior Data Center Performance Engineer - Benchmarking and Optimization
$184k - $287.5k
Senior Data Center Performance Engineer - Benchmarking and Optimization page is loaded## Senior Data Center Performance Engineer - Benchmarking and Optimizationlocations... .... We are searching for a highly motivated engineer to lead performance benchmarking and optimization efforts for our...
NVIDIA Corporation
Santa Clara, CA
5 hours ago
Data Center Engineer
...Data Center Engineering Lead (Contract) Department : Infrastructure, Planning & Process (IPP) Company : NVIDIA Job Summary The IPP division at NVIDIA designs, scales, and optimizes the global data center footprint powering our AI platforms and cloud environments. We are...
Contract work
SproutsAI
Santa Clara, CA
2 days ago
Data Center Engineer III
$72.96k - $115.2k
...What this Job Entails: The Data Center Engineer III will be heavily involved in the design of the datacenter collocations, infrastructure hardware, and provide technical consultations. The incumbent will build/create all drawings for datacenter POP’s and develop/write...
Full time
Temporary work
Work at office
Flexible hours
Astreya
Cupertino, CA
2 days ago
Data Center Engineer
...A global technology services leader is seeking a Data Center Service Delivery Field Support Engineer (L2) to provide second-line technical support and manage incidents to ensure zero missed SLA conditions. Candidates should possess a Bachelor's degree in IT or a related...
NTT DATA
Santa Clara, CA
12 hours ago
Data Engineer
...Data Engineer Location: Mountainview, CA (Hybrid 3 days' work from office) C2C pay rate: $70-75/hr for both the roles. Role Summary: Responsible for designing and building graph data models, knowledge graphs, and data pipelines optimized for GraphRAG and reasoning...
Work at office
Kasmo Global
Mountain View, CA
5 days ago
Data Engineer
...test and deploy cutting edge solutions at scale, impacting millions of customers worldwide drive value from data at Walmart Scale Interact with Walmart engineering teams across geographies to leverage expertise and contribute to the tech community. Engage with...
Worldwide
Software Technology Inc
Sunnyvale, CA
5 days ago
Sr. Data Engineer
$170k - $210k
...Sr Data Engineer – Samsung Ads Samsung Ads is a fast-growing, advanced advertising technology business that helps brands connect with audiences across Samsung Smart TVs and digital ecosystems. By leveraging the industry's most comprehensive data and world-class technology...
Hourly pay
Full time
Internship
Samsung
Mountain View, CA
12 hours ago
Data Engineer
...Expertise in SQL: You possess a deep understanding of SQL for data manipulation, querying, and performance optimization across various... ...track record of successfully contributing to and potentially leading data-centric projects, understanding the full lifecycle from data...
Omni Inclusive
Sunnyvale, CA
3 days ago
Data Engineer
$73 per hour
...Data Engineer MAHIN-JOB-35556 We are looking for Data Engineer in Sunnyvale,CA Day 1 onsite ( 3 days a week hybrid) FTE- 130k per annum + benefits Subcon-73 $ per hour ( no layers are accepted) Job Description: Strong Python development and software...
Hourly pay
3 days per week
1 day per week
Keylent Inc
Sunnyvale, CA
2 days ago
Data Engineer
...Location: Sunnyvale, CA (5x/ week onsite) Duration: 6 months Data Engineer Must-haves: Must have Snowflake, Advanced SQL, Dimensional... ...interacting with business and independently develop and lead data projects. Collaborating with Offshore and owning overall...
Q1 Technologies
Sunnyvale, CA
2 days ago
Data Engineer
$150k
...Data Engineer Specializing In Natural Language Processing (NLP) We are a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative...
Visa sponsorship
Institute of Foundation Models
Sunnyvale, CA
5 days ago
Data Engineer GCP
...Data Engineer GCP Sunnyvale, California, United States Job Title: Data Engineer Location: Sunnyvale Ca Visa: USC/GC/H1b Pay rate: W2 only no C2C Details: 5 years of SQL queries Tech stack Airflow pipeline development GCP dataproc/ BQ Expertise Hive/Spark Python/Java...
H1b
Interaction24 LLC
Sunnyvale, CA
2 days ago
Data Engineer
PlusAI is seeking a data engineer to create and maintain optimal data processing pipeline architecture in Santa Clara, California. The role involves building infrastructure for data extraction, transforming, and loading, while also automating manual processes and optimizing...
Medium
Santa Clara, CA
13 hours ago
Data Engineer
NewRocket is looking for an AI Data Intelligence Lead who will design AI-powered capabilities for modernizing enterprise data. This role entails... ...machine learning, generative AI, and advanced data engineering to enrich and optimize enterprise data. Ideal candidates will...
Remote work
NewRocket Inc
Santa Clara, CA
13 hours ago
Senior Data Engineer
Samsung Electronics Perú is seeking a Sr. Data Engineer to contribute to scalable data pipelines and support the Performance & Reach engineering organization. You will work with large advertising datasets and collaborate with cross-functional teams to deliver data solutions...
Samsung Electronics Perú
Mountain View, CA
12 hours ago
Big Data Engineer
$100k - $150k
...Big Data Engineer – Remote Bright Vision Technologies is a technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the United States. This is a fantastic opportunity to join an established and well-respected...
Full time
H1b
Local area
Immediate start
Remote work
Visa sponsorship
Bright Vision Technologies
Sunnyvale, CA
7 days ago
Senior Data Engineer
Job Summary Seeking a Data Engineer with Spark & Streaming skills to build real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML. Key Responsibilities Design, develop,...
Compunnel
Sunnyvale, CA
3 days ago
Snowflake Data Engineer
...Role: Snowflake Data Engineer Location: Santa Cla, CA (5 days Onsite) Job Description Data Engineer - Key Requirements: Proven hands-on expertise in data engineering with Snowflake and Databricks , including building and optimizing...
Inficare
Santa Clara, CA
4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!