Senior Data Engineer - Spark/Flink/Scala
Omega Solutions
Scala Engineer
We are looking for Scala Engineers with experience with batch and/or streaming jobs. We utilize Spark for batch jobs and Flink for real-time streaming jobs. Experience with Hadoop, Hive, AWS S3 is also an asset. Responsibilities include creating and maintaining Spark and Flink jobs written in Scala, producing unit and system tests for all code, participating in design discussions to improve existing frameworks, defining scalable calculation logic for interactive and batch use cases, and interacting with infrastructure and data teams to produce complex analysis across data.
Required Qualifications:
- A minimum of 2 years of experience with Scala and/or Java
- A minimum of 5 years of programming experience
- Required experience with Hadoop, Spark
- Knowledge and experience with cloud-based technologies
- Experience in batch or real-time data streaming
- Ability to dynamically adapt to conventional big-data frameworks and open source tools if project demands
- Knowledge of design strategies for developing scalable, resilient, always-on data lake
- Strong development/automation skills
- Comfortable with reading and writing Scala code
- Aptitude for analytical problem solving
- Deep knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance
- Good understanding/knowledge of HDFS architecture and various components such as Job Tracker, Task Tracker, Name Node, Data Node, HDFS high availability (HA) and Map Reduce programming paradigm
- Experienced working with various Hadoop Distributions (Cloudera, Hortonworks, MapR, Amazon EMR) to fully implement and leverage new Hadoop features
- Experience in developing Spark Applications using Spark RDD, Spark-SQL, Spark -Yarn, Spark Mlib and Data frame APIs
- Experience with real-time data processing and streaming techniques using Spark streaming and Kafka, moving data in and out HDFS and RDBMS
- Familiarity with open source configuration management and development tools
- Hands on experience and production use of Hadoop/Cassandra, Spark, Flink and other distributed technologies would be a plus
- Other Technologies Scalatest Gradle/Maven Airflow SQL AWS
Preferred Qualifications:
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer - Spark/Flink/Scala in Mountain View, CA vacancy
- ...in design, implementation, and support of solutions big data solution in Hadoop using Hive, Spark, Drill, Impala, HBase. Hands on experience with Unix,... ...Teradata and other relational databases. Experience with Scala a plus. Strong communication and problem-solving skills...SuggestedWork at office2 days per week
Samprasoft
Sunnyvale, CA1 day ago - ...in Sunnyvale, California, seeks a skilled Back-End Engineer to design and maintain scalable data pipelines. The ideal candidate will have over 5... ...experience in back-end development and proficiency in Spark, Python, Scala, and Java. The role also involves collaborating...Senior
- ...Senior Data Engineer (Spark, Python, AWS) Location: Mountain View California (100% Onsite) Duration: 6+ Months Job Description: Skills: Spark, Python, SQL, DWH / ETL Concepts, AWS Responsibilities: Design, build and maintain data infrastructure that...Senior
Zortech Solutions
Mountain View, CA1 day ago - ...Sr Data Engineer With GCP, Spark, Python Sunnyvale, CA (Onsite) Long Term Requirements: GCP Experience • 2+ years of recent GCP experience... ...• Experience with programming languages: Python, Java, Scala, etc. • Experience with scripting languages: Perl, Shell...Senior
InterSources
Sunnyvale, CA1 day ago $100k - $300k
...We are seeking a forward-thinking AI Data Engineer to bridge the gap between our user data... ...# Proficiency in SQL and Python/Java/Scala. # Hands-on experience with Data Warehouses... ..., ClickHouse) and Big Data frameworks (Spark, Flink). # Familiar with message middleware...SeniorFull timeOPPO US Research Center
Palo Alto, CA1 day ago- ...We are actively looking for Data Engineers (Java/spark) for Sunnyvale, CA or Bentonville, AR Job Title: Sr Data Engineer (Java/spark) Location: Sunnyvale, CA or Bentonville, AR (Hybrid) Job Type: Contract Rate: DOE NOTES * W2 candidares...SeniorContract workImmediate start
Redolent
Sunnyvale, CA4 days ago - A technology firm is seeking a skilled Back-End Engineer with a strong focus on data engineering to design and maintain data pipelines and back-end systems... ...successful candidate will work with technologies like Spark, Python, and AWS. Responsibilities include writing SQL...Senior
- ...in design, implementation, and support of solutions big data solution in Hadoop using Hive, Spark, Drill, Impala, HBase. Hands on experience with UNIX... ...and other relational databases. Experience with Scala a plus. Excellent communication and problem-solving...
Samprasoft
Sunnyvale, CA2 days ago - ...We are seeking an exceptional engineer with a background in large scale... ...team that drives Apache Spark’s state-of-the art features and... ...cutting-edge technologies to build data intensive systems, designing... ...proficiency with JVM languages (Scala or Java preferred), Spark,...Senior
Apple
Sunnyvale, CA1 day ago - ...reference. Recent update from client for this data engineer role / Request-ID: 13145-1 _ Bentonville, AR position... ...reference, which has closed with client UST. Looking for Senior Data Engineer with Spark, Scala, GCP experience. Location: This position will...Senior
Texas State Library and Archives Commision
Sunnyvale, CA3 days ago - ...Senior Data Engineer - GCP Location: Mountain View CA (Onsite) What You’ll Do Develop... ...Building data pipelines, leveraging Scala, Pub Sub, Akka, Dataflow on Google Cloud... ...frameworks such as TensorFlow, Apache Spark, and scikit-learn. Bachelor’s or master...Senior
United IT
Mountain View, CA1 day ago - ...a strong passion for using data to drive business decisions,... ...collaborators? We are seeking for a Senior Data Engineer to join our App Store Data... ...programming skills in Scala or Java, proficient in Python... ...data technologies such as: Spark/Flink, Kafka, Airflow, Iceberg, Trino...Senior
Apple
Cupertino, CA1 day ago - ...Sr. Data Engineer Mountain View, CA / Denver, CO / Remote Full-Time Position Summary: We are looking... ...with Python Other languages like Scala, C++, Java, or similar are a plus Experience with spark Expertise with SQL Familiarity with Docker...SeniorFull timeRemote work
1872 Consulting
Mountain View, CA3 days ago - ...patients worldwide. We're a team of engineers, clinicians, and innovators united by one... ...of Position We are seeking a Senior Data Engineer to join the software team within... ...with data engineering frameworks (e.g., Spark, Flink) Experience with managed platforms such...SeniorLocal areaWorldwideFlexible hours
Intuitive
Sunnyvale, CA3 days ago - ...better working world. Technology – Data and Decision Science – Data Engineering – Senior We are seeking a highly skilled... ...in Databricks and experience with Spark for big data processing. Proven experience... ...in languages such as Python, Scala, or SQL. Experience with data...Senior
Ernst & Young Oman
Palo Alto, CA1 day ago - ...Summary... The Catalog Data Science team at Walmart... ..., Deep Learning, and Engineering. We tackle complex... ...compliance at Walmart. As a Senior Data Scientist (Machine... ...frameworks (Ray, Spark) Experience deploying... ...assessments in Python, Spark, Scala, or R, Using open...Senior
Walmart
Sunnyvale, CA9 days ago - ...Senior, Data Engineer Multiple locations Regular/ Permanent Salaried Position Summary... As a Data Engineer on the Global Data Foundation... ...includes working with big data frameworks such as Apache Spark and Scala, orchestration tools like Airflow, storage systems...SeniorPermanent employmentFull timeTemporary workPart time
Walmart
Sunnyvale, CA3 days ago $117k - $234k
...critical business initiatives with Data. You love writing and owning codes... ...act on this data. Partner with engineering, AI/ML, and product teams to ensure... ...testable code in Python, Java, or Scala ; experience with PySpark/Spark for distributed data processing....SeniorFull timeTemporary workPart timeWalmart
Sunnyvale, CA5 days ago- ...implementation, and support of solutions big data solution in Hadoop using Hive, Spark, Drill, Impala, HBase Hands on... ...Experience working with Java, Scala, Python programing languages... ...~ There are three tech leads with Senior engineers and a couple of junior engineers...Work at officeRemote work
Samprasoft
Sunnyvale, CA2 days ago $188.6k - $271.4k
...of patients worldwide. We’re a team of engineers, clinicians, and innovators united by one... ...Function of Position We are seeking a Senior Data Engineer to join the software team... ...with data engineering frameworks (e.g., Spark, Flink) Experience with managed platforms such...SeniorLocal areaWorldwideShift work- ...Senior Full Stack Big Data Engineer Profile: Senior Full Stack Big Data Engineer with focus on Spark, Scala, Kafka & Cassandra. Extensive experience of Java based full stack Big Data implementation on HDP platform (Hortonworks). Hands-on experience in Spark, Scala...SeniorWork experience placement
ClifyX
Sunnyvale, CA1 day ago $120.75k - $251.25k
...looking to write mobile app code, engineer the servers behind our massive... ...help us process trillions of data points a day, what you do here... ...one of: Go/Java/Python/C++/Scala/SQL 5+ years of industry experience... ...Map/Reduce, Pig, Hive, HBase, Spark, Kafka, Oozie, etc.)...SeniorWork at officeFlexible hours- ...Sr. Software Engineer (Spark & Scala) Location: Austin, TX / Sunnyvale, CA Type: Contract Desirable Skills: Customer is looking for resources with Software Engineering experience and not Data Engineering. Strong Software Engineering experience (Application...SeniorContract work
Syntricate Technologies
Sunnyvale, CA1 day ago $177.8k - $240.5k
..., this is the place for you. The ASP Data Engineering Team supports the data needs of the AWS... ...programming language, such as Python, Java, Scala, or NodeJS ~ Experience with AWS... ...data technologies such as: Hadoop, Hive, Spark, EMR Experience operating large data...SeniorLocal areaWorldwideFlexible hoursAmazon
Mountain View, CA5 days ago$143k - $286k
...you'll do... Principal Data Engineer – Agentic Data Platforms... ...guide Staff Engineers and Senior Engineers. Lead design... ...streaming systems (Kafka, Spark Structured Streaming, Flink, Druid, etc.). Experience... ...Fluency in Python, Java, or Scala. Deep experience with...Full timeTemporary workPart timeWalmart
Sunnyvale, CA3 days ago$146.6k - $150k
Tech & AI Data Engineer II - QuantumBlack, AI by McKinsey Job ID: 102... ...hire at the Engineer I/II or Senior Engineer level, depending on... ...~ Proficiency in Python, Scala, or Java for production-grade... ...distributed systems such as (Spark, Dask, Flink) and streaming platforms (Kafka...ApprenticeshipWork at officeEasy workMcKinsey & Company
Palo Alto, CA5 days ago- ...Big Data Engineer Location: Mountain View, CA (3 days onsite) Skill Set: ~ Minimum of 5+ years working with Apache Flink or Spark Structured Streaming (preferably Flink) ~5+ years experience with Java or Scala ~ Strong experience working with microservices...
Kasmo Global
Mountain View, CA1 day ago - ...technology company is seeking an experienced backend engineer to join their Data Layer and Marketing AI platform. You will... ...data-intensive architectures, proficient in Scala, Java, or Python, and familiar with tools like Apache Spark and Kafka. This role involves working across...Senior
- ...develop, and maintain big data pipelines leveraging Hadoop, Spark, and Kafka .... ...Program in Python, Scala, and Java to build robust... ...scientists, analysts, and DevOps engineers to deliver end-to-end data... ...frameworks (Spark Streaming, Flink, Storm). Exposure to...
Purple Drive
Sunnyvale, CA3 days ago - ...Sterling B2B Developer Role: Senior Data Engineer Location: Dallas, TX or Mountain View, CA Duration: Long Term Contract As part... ...Airflow, Composer, Oozie) with deep expertise in SQL, PySpark, or scala. ~5+ years leading data engineering development directly...SeniorLong term contract
Hermitage Infotech
Mountain View, CA1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer - Spark/Flink/Scala. Be the first to apply!
Related searches
- director data engineering Mountain View, CA
- junior big data engineer Mountain View, CA
- senior data engineer Mountain View, CA
- data platform engineer Mountain View, CA
- sr information security engineer Mountain View, CA
- senior data integration developer Mountain View, CA
- data developer Mountain View, CA
- data engineer Mountain View, CA
- data infrastructure engineer Mountain View, CA
- entry level big data engineer Mountain View, CA