Big Data Engineer
Purple Drive
Overview: Key Responsibilities
- Design, develop, and maintain big data pipelines leveraging Hadoop, Spark, and Kafka .
- Develop and optimize J2EE backend services for large-scale data processing and integration.
- Program in Python, Scala, and Java to build robust data and backend solutions.
- Architect and implement cloud-native solutions on GCP and Azure platforms.
- Design and maintain data models, data warehousing solutions, and BI tools to enable data-driven decision making.
- Implement and manage workflow orchestration frameworks such as Automic and Airflow for data pipeline automation.
- Collaborate with cross-functional teams including data scientists, analysts, and DevOps engineers to deliver end-to-end data solutions.
- Ensure data quality, security, and scalability across all platforms.
- Troubleshoot and optimize performance of large-scale distributed systems.
- Bachelor's/Master's degree in Computer Science, Engineering, or related field (or equivalent experience).
- 6-8years of experience in big data engineering and backend development .
- Expertise in Hadoop ecosystem (HDFS, MapReduce, Hive, Spark) and Kafka .
- Strong programming skills in Python, Scala, and Java .
- Hands-on experience with J2EE backend development .
- Proven experience with cloud services (GCP, Azure) for data engineering and application deployment.
- Proficiency in data modeling, warehousing concepts, and BI tools .
- Experience with workflow orchestration tools such as Airflow, Automic .
- Strong problem-solving and debugging skills in large-scale distributed environments.
- Familiarity with containerization and orchestration (Docker, Kubernetes).
- Knowledge of streaming frameworks (Spark Streaming, Flink, Storm).
- Exposure to CI/CD pipelines and DevOps practices.
- Experience in Agile/Scrum development environments .
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Big Data Engineer in Sunnyvale, CA vacancy
- ...Principal Big Data Engineer As a Principal Big Data Engineer, you will be an integral member of our data ingestion and processing platform team responsible for architecture, design and development. Having the dynamic ability to adapt to conventional big-data frameworks...Big data
Omni Inclusive
Santa Clara, CA5 days ago $50 - $60 per hour
...Bigdata Engineer Location: Phoenix, AZ Duration: 12+ Months Client: Client / Impetus Pay Rate: $50 to $60 Max depends on experience... ...year Bachelors Year and university Total Exp US Exp. Big Data Exp. Current Location Open for Travel Open for...Big dataRelocationOmega Solutions
Santa Clara, CA2 days ago- ...Big Data Engineer: Must Have: • Proficiency in managing and manipulating huge datasets in the order of terabytes (TB) is essential • Expertise in big data technologies like Hadoop, Apache Spark (Scala preferred), Apache Hive, or similar frameworks on the...Big data
Texas State Library and Archives Commision
Sunnyvale, CA1 day ago $90k - $110k
...Big Data Engineer – Remote Bright Vision Technologies is a technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the United States. This is a fantastic opportunity to join an established and well-respected...Big dataFull timeH1bLocal areaImmediate startRemote workVisa sponsorshipBright Vision Technologies
Santa Clara, CA4 days ago$175k - $221k
...content intelligence.If you’re inspired to dream big, innovate fast, and make a difference, we’d love... ...A/B test, and every ML model we ship runs on the data infrastructure you'd own. We're looking for a senior engineer to build and scale the batch and streaming pipelines...Big dataFull timeLocal areaWork from home$174.72k - $295.68k
...learning, and smart connectivity.We are seeking a Machine Learning Data Curation Engineer to spearhead the data pipeline development and dataset... ...in programming languages like Python or SQL.Experience with Big Data tools and cloud platforms (e.g., AWS, GCP, BigQuery)....Big dataFull time$135k - $180k
Santa Clara, CAData Engineering - Backend Engineering /Full-time /HybridPlusAI is a Physical AI... ...: Create and maintain optimal data processing pipeline architectureBuild the... ...transformation, and loading of data using SQL and big data technologies.Identify, design, and implement...Big dataFull time$202.5k - $274k
...the QuickBooks Capital and Lending Platform as a Staff Software Engineer in Data. You will join an innovative and passionate team of engineers... ...technologies like Python, PySpark, Spark, SQL, Data Bricks, Big Data fundamentals, AWS, Elastic Search, Kafka, to build enterprise...Big dataWorldwide$117k - $234k
...OfficePosition Summary...What you'll do...Role summary: As a Senior Data Engineer, you will design, develop, and maintain scalable data pipelines... ...optimize data accessibility and reliability within cloud and big data environments. This role demands a strategic mindset to...Big dataFull timeTemporary workPart time- ...impacting millions of customers worldwide drive value from data at Walmart Scale Interact with Walmart engineering teams across geographies to leverage expertise and... .... Your Qualifications Min 6-10 years of Big data development experience Demonstrates up-to-...Big dataWorldwide
Software Technology Inc
Sunnyvale, CA4 days ago - ...professional IT Consulting services in the US. We are actively seeking Data Engineer for one of our client, Please share your resume with current... ...building and optimizing data pipelines and integrating with big data platforms like Apache Spark and Hadoop. Develop efficient,...Big data
Rootshell Enterprise Technologies
Santa Clara, CA3 days ago - ...software development or related 2. 3+ years of solid Java and 2+ years experience in design, implementation, and support of solutions big data solution in Hadoop using Hive, Spark, Drill, Impala, HBase 3. Hands on experience with Unix, Teradata and other relational...Big dataFull time
Redolent
Sunnyvale, CA2 days ago $117k - $234k
...drive critical business initiatives with Data. You love writing and owning codes and enjoy... ...and act on this data. Partner with engineering, AI/ML, and product teams to ensure data... ...you'll bring:4-6 years of experience in Big Data development with a focus on scalable...Big dataFull timeTemporary workPart time- ...Senior Data Engineer Location: Sunnyvale CA (Onsite) Duration: Long Term Job Description: GCP Experience • 4+ years of recent GCP... ...: As a Senior Data Engineer, you will • Design and develop big data applications using the latest open source technologies. •...Big data
JConnect Infotech
Sunnyvale, CA4 days ago - ...for your reference. Recent update from client for this data engineer role / Request-ID: 13145-1 _ Bentonville, AR position is no... ...As a Senior Data Engineer, you will • Design and develop big data applications using the latest open source technologies....Big data
Texas State Library and Archives Commision
Sunnyvale, CA1 day ago - ...Data Strategy Understands, articulates, and applies principles of the defined strategy to routine business problems that involve a... ...mathematical solutions. Identifies what methods (for example, analytics, big data analytics, automation) would provide a solution for the...Big data
Diverse Lynx
Sunnyvale, CA4 days ago $80 - $95 per hour
This range is provided by OSI Engineering. Your actual pay will be based on your skills and experience... ...Mountain View, CA is seeking a talented Data Engineer to help build secure, scalable... ...engineers with strong expertise in Big Data infrastructure and data warehousing...Big dataFull timeContract work- ...accomplish. We are seeking an exceptional engineer with a background in large scale software... ...using cutting-edge technologies to build data intensive systems, designing and optimizing... ...Java or Scala ~5+ years experience with big data platforms ~ Familiarity with ML/AI...Big data
Apple
Sunnyvale, CA4 days ago - ...implementation of large-scale distributed data platforms. • Build and optimize high-... ...observability. • Mentor a team of data engineers and provide technical leadership across... ...years of experience in Data Engineering and Big Data ecosystems. • Expert knowledge of...Big data
Veridic Solutions
Sunnyvale, CA1 day ago - ...Job Title: Data Engineer . Location: Sunnyvale, CA. Direct Client: Req. Key Qualifications: ~ Data Engineer... ...software. ~ Experience in Dimensional Modeling (DM) is a big plus. ~ Ability to learn and adapt to new tools and...Big data
Numeric
Sunnyvale, CA5 days ago - ...train AI agents to automate tasks, analyze data, and enhance team productivity. Our... ...safety and compliance. Job title Data Engineer Position overview We are looking for... ...on cloud technologies, especially AWS, and big data systems, your role will be pivotal in...Big dataFlexible hours
Brevian.ai
Sunnyvale, CA5 days ago - ...Data Engineer We are seeking a highly skilled Data Engineer to join our team, focusing on building, maintaining, and optimizing our data... ...and reporting. This role requires expertise in ETL processes, big data technologies, and cloud data platforms. Key Responsibilities...Big data
Omni Inclusive
Santa Clara, CA5 days ago - ...fundamental and applied research in intelligent data analytics, hardware/co-processor... ...interested in all components of a large scale Big Data Analytics system and AI Platforms,... ..., etc. We are leading researchers and engineers from worldwide to create adaptive and high...Big dataFor contractorsWorldwide
Netpace
Santa Clara, CA4 days ago - ...W2 Position - Local to CA preffer Position: Big Data Java Engineer Location: California Job Summary: Seeking a strong Big Data Java Engineer with hands-on experience in Java, Spark, GCP , and distributed systems. The ideal candidate...Big dataLocal area
Octans Group LLC
Sunnyvale, CA1 day ago - ...Sr. Data Engineer Mountain View, CA / Denver, CO / Remote Full-Time Position Summary: We are looking for a Sr. Data Engineer to... ...Interest in pulling data from many sources Experience in big data, data mining and statistical analysis Cloud computing,...Big dataFull timeRemote work
1872 Consulting
Mountain View, CA3 days ago $70 - $95 per hour
...Job Description Job Description Data Engineer The Cloud services group is looking for world class engineers to join our technology innovation... .... Looking for world class server software engineers with Big Data Infrastructure and Data warehousing experience to join our...Big dataHourly payCypress HCM
Mountain View, CAmore than 2 months ago- ...Data Pipeline Engineer ByteDance is a technology company operating a range of content platforms that inform, educate, entertain and inspire... ...Enterprise Administration, etc. Taking advantage of cutting-edge big data and AI technologies, we are aiming at building a more...Big data
Adapt Talent
Mountain View, CA4 days ago - ...Lead Data Engineer RadiumOne is seeking a Lead Data Engineer to join our Data Technology team with the responsibility of designing and implementing... ...experience ~ Knowledge and expertise of No SQL and Big Data technologies ~ Strong knowledge of Linux ~ Demonstrated...Big data
Stepping Up Solutions
Sunnyvale, CA4 days ago - ...Data Engineer Sam's Cash Core Team is looking for a savvy Data Engineer to join our growing team of analytics experts. The role will be... ...python programming skills, expert level on using Python to process Big Data; ~ Advanced working SQL knowledge and experience working...Big data
My3Tech Inc
Sunnyvale, CA2 days ago - ...centers worldwide, gaining exposure to complex engineering challenges that redefine industrial... ...are currently searching for a Senior AWS Data Engineer: The Challenge (... ...lakes, data warehouses, data marts, and big data environments. ~ Strong understanding...Big dataRemote workWorldwide
Sequoia Connect
Mountain View, CA3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!
Related searches
- aws data engineer Sunnyvale, CA
- big data devops engineer Sunnyvale, CA
- big data developer Sunnyvale, CA
- software data engineer Sunnyvale, CA
- data center engineer Sunnyvale, CA
- big data cloud engineer Sunnyvale, CA
- big data engineer Sunnyvale, CA
- senior data integration developer Sunnyvale, CA
- data developer Sunnyvale, CA
- data science developer Sunnyvale, CA