Senior Data Engineer (Apache Spark)
Appit LLC
APPIT Software Solutions is hiring a Senior Data Engineer (Apache Spark) in San Francisco, USA . Lead the design of large-scale distributed data processing systems using Apache Spark and cloud platforms at APPIT Software in San Francisco. Responsibilities Architect and optimize large-scale Spark jobs processing terabytes of data daily Design data lake and lakehouse architectures on AWS S3 or Azure Data Lake Storage Mentor junior data engineers on distributed computing best practices and Spark internals Build and maintain real-time and batch data pipelines with robust fault-tolerance Partner with ML engineers to deliver feature stores and training data sets at scale Drive performance tuning including partitioning, caching, and shuffle optimization strategies Requirements 6+ years of data engineering experience with at least 3 years focused on Apache Spark Deep understanding of distributed computing, MapReduce paradigms, and cluster resource management Expert-level Python and/or Scala programming for Spark application development Experience with cloud data services (AWS EMR, Glue, Redshift, or Azure Synapse) Strong knowledge of data lake architectures, Delta Lake, or Apache Iceberg table formats Proven ability to optimize Spark jobs for cost efficiency and processing speed Nice to Have Experience with Databricks Unified Analytics Platform Knowledge of streaming with Spark Structured Streaming or Flink Contributions to open-source data projects About APPIT Software Solutions APPIT Software Solutions is a leading technology company building AI-powered enterprise products. Join 150+ professionals working on cutting-edge solutions in AI/ML, cloud computing, and digital transformation. We offer competitive salaries, flexible work arrangements, and opportunities for career growth across our global offices. #J-18808-Ljbffr
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer (Apache Spark) in San Francisco, CA vacancy
$123.7k - $254.67k
A leading social media platform in San Francisco is seeking a Senior Data Engineer to implement and optimize data infrastructure using AWS. This role requires strong proficiency in Spark and Scala, alongside a Bachelor’s degree in Computer Science or a related field. You...Senior- A leading data collaboration company in San Francisco is seeking a Senior Software Engineer to join their Identity Engineering team. This role involves leading complex projects... ...should have extensive experience with Spark, Hadoop, and cloud platforms such as GCP or AWS...SeniorRemote work
- ...LiveRamp in San Francisco is seeking an experienced data engineer to design, build, and optimize scalable Big Data processing pipelines... ...of experience in software/data engineering, experience with Apache Spark and orchestration tools, and a bachelor’s degree in a...SeniorRemote workFlexible hours
LiveRamp
San Francisco, CA12 hours ago - ...pioneering life sciences organization in the UK is seeking a Data Engineer to work on Veeva Link’s next-gen Data Platform. You will be... ...candidates have expertise in Java or Python and experience with Apache Spark. This role supports flexible working environments to thrive...SuggestedRemote jobWork at officeWork from homeFlexible hours
- ...and logistics operations. Our Data Engineers are the backbone of that... ...powers our AI products. As a Senior Data Engineer on our India engineering... ...orchestration tools such as Apache Airflow or Dagster ~... ...with distributed data systems - Spark, Hive, or HDFS ~ Solid...Senior
Daybreak AI, Inc
San Francisco, CA5 days ago $170k - $220k
...Windfall is seeking a Sr. Data Engineer to join our data team. As a Sr. Data Engineer on our data team, you will be building... ...Cassandra Distributed processing frameworks - Dataflow (Apache Beam) and Apache Spark Orchestration - Airflow Requirements: ~4-8 years...SeniorWindfall
San Francisco, CA1 day ago$130k - $196.5k
...LiveRamp is the data collaboration platform of choice for the world... ...processing pipelines using Spark, Airflow, and related big data... ...design documents. Champion engineering best practices (code reviews,... ...and operating pipelines with Apache Spark (batch and/or streaming)...SeniorWork from homeFlexible hoursNight shiftLiveRamp
San Francisco, CA4 days ago- Requirements 5+ years of experience in data architecture, data modeling, master... ...(Desirable) Hands‑on experience with Spark SQL, AWS (S3, EMR), Apache Pinot, or Hadoop/Big Data ecosystems... ...stores What the job involves As a Data Engineer on the Enterprise team, you'll...SeniorPermanent employment
$242.1k - $293.8k
...daily active users growing at a record pace, we are seeking a senior data engineer to help build the next generation of Ad Data and ML... ...and streaming) and big‑data processing technologies such as Spark, Apache Druid, Flink, Ray, Kubeflow, etc. Excellent cross‑functional...SeniorWork experience placementLocal area- ...resonating with clients all across the country. Job Summary On the Data Engineering team, we transform streams of data into insights that solve... ...‑on experience with leading ETL tools and frameworks (e.g., Apache Spark, Apache Airflow, dbt, Looker, Superset). In‑depth experience...SeniorTemporary workWork at officeFlexible hours
SmithRx
San Francisco, CA3 days ago $155k
...About the Team The Data Platform team sits within Scribd... ...making data work for everyone - engineers, analysts, and business... ...Role We're looking for a Senior AI Data Engineer to lead the... ...SQL, DABs, Genie Spaces), Apache Airflow, Spark, Terraform/Spacelift, and AWS...SeniorFor contractorsLocal areaHome officeFlexible hoursScribd
San Francisco, CA2 days ago$172.5k - $260.1k
...duplicating efforts. Job Category Software Engineering About Salesforce Salesforce is the #1... ...the future of Salesforce. Enterprise Data & AI Solutions Group The Enterprise Data... ...Python, dbt, Airflow, and advanced SQL. Apache Spark, and Snowflake. AI Orchestration: Fluency...SeniorShift work$150k - $170k
...currently looking for an experienced Senior Data Platform Engineer, with significant experience in building... ...solutions using Databricks, Dbt, and Spark Develop and maintain ETL/data... ...pipelines and services Deep experience with Apache Spark and distributed data processing...SeniorFull timeRemote workFlexible hoursEllipsis Health
San Francisco, CA12 hours ago- 6Sense is looking for a Data Engineer in San Francisco to design and build robust big data systems and data transformations. Candidates should... ...strong experience in Big Data technologies like Hadoop and Spark. The position emphasizes the importance of communication...Senior
6sense
San Francisco, CA3 days ago - ...EMEA, and APAC. Role Overview RZR Global is seeking a talented Data Engineer to join our growing engineering team. This role is ideal for... ...data. You will work with technologies such as ClickHouse, Kafka, Spark, HDFS and Fluentd to ensure data is reliable, scalable, and...Senior
RZR Global Inc.
San Francisco, CA12 hours ago $160k - $210k
...from anywhere in the world. About The Role We are seeking a Senior Data Engineer to strengthen Arena Club’s strategic decision‑making, enhance... ...outbound ETL pipelines built on AWS Glue (Python Shell & Spark ETL) Manage Redshift cluster performance across various schemas...SeniorArena Club
San Francisco, CA3 days ago- ...About the job Senior Data Engineer Role: Senior Data Engineer We are seeking a Sr Data Engineer for our Data feeds team to provide... ...working experience of distributed systems Hadoop/MapReduce, Spark, Hive, Kafka and Oozie/Airflow ~ At least 5 years of solid...SeniorWork experience placement
Inizio Partners
San Francisco, CA2 days ago - ...and debugging production‑grade data pipelines at scale in AWS,... ...similar orchestration tools and Spark or comparable distributed processing... ...and collaborate across engineering, product, and data teams Comfort... ...quality What the job involves The Senior Data Engineer will design,...Senior
Komodo Health
San Francisco, CA4 days ago - ...experience building production data pipelines and platforms Deep... ...SQL fluency, including tuning Spark jobs at scale The skills and willingness... ...and PII/PHI handling as core engineering work, not as someone else's... ...What the job involves As a Senior Data Engineer, you will own...SeniorTemporary workWeekend work
NimbleRx
San Francisco, CA12 hours ago $190.8k - $286.8k
...Washington D.C., London and Amsterdam. Making data‑driven decisions is key to Plaid's... ...golden datasets and tooling to teams across engineering, product, and business and help them... ...real‑time pipelines using technologies like Spark, Kafka. Appreciation for the importance of...SeniorLocal areaFlexible hoursPlaid
San Francisco, CA1 day ago- ...driven mobile advertising company in San Francisco is seeking a Data Engineer to join its engineering team. This role involves designing,... ...pipelines, and proficiency in technologies such as Kafka and Spark. This position offers an opportunity to drive performance and...Senior
- ChatGPT Jobs is seeking a Data Engineer to design and maintain production-grade data pipelines for complex supply chain operations. The ideal... ...in Python and SQL, and familiar with orchestration tools like Apache Airflow. The role is located in San Francisco, CA, and allows...SeniorRemote job
- ...leading technology firm in San Francisco is looking for an experienced AWS Data Engineer to design and maintain data pipelines using AWS services. This role requires strong skills in Python and Spark, along with expertise in ETL development and data quality assurance. The...Senior
$160k - $240k
...Senior Data Engineer San Francisco About the Role Together AI is looking for a Senior Data Engineer to help define, build, and operate... ...stream processing pipelines with services like AWS Kinesis, Apache Kafka, or Redpanda ~ Expert-level knowledge of building...SeniorFull timeTogether AI
San Francisco, CA5 days ago$300 per month
...Location Type On-site Department Cloud Engineering Crusoe's mission is to accelerate the abundance... ...This Role: Join Crusoe Energy as a Senior Data Engineer, an early and pivotal hire on... .../Sub, Cloud Storage). Experience with Apache Beam. Background in building...SeniorFull timeTemporary workEpoch Biodesign
San Francisco, CA3 days ago$15 per hour
...contributors. The Wikimedia Foundation’s Data Platform team enables this global... ...research, feature development, and . As a Senior Data Engineer for our Data Platform, you will shape... ...Expertise in tools like Airflow , Kafka , Spark , and Hive . Advanced proficiency...SeniorFor contractorsLocal areaRemote workCloudDevs
San Francisco, CA4 days ago$181.1k - $272.1k
Apple Inc. in San Francisco is hiring a Senior Software Data Engineer for the App Store. In this role, you will be responsible for building and optimizing... ...7 years of experience with big data tools like Hadoop and Spark, strong programming skills in Scala or Java, and a...Senior$180k - $195k
...more information, please visit Role Description As a Senior Data Engineer at Gridware, you'll design and scale the data infrastructure... ...streaming databases ~ Hands-on with modern data stacks - Spark, Kafka, Databricks, Grafana, and other cloud-native tools...SeniorGridware
San Francisco, CA1 day ago- ...San Francisco, Salt Lake City, Portland, Seattle As a Senior Data Engineer, You Will: Design, develop, and operationalize large-scale... ...services such as S3, Glue, Athena, Lambda, RedShift, EMR, Spark, and DynamoDB Analyze, redesign, and re-platform on-premise...SeniorRemote workFlexible hoursShift work
xScion Solutions
San Francisco, CA3 days ago - ...Hartz, and others. The Role: We are looking for a Senior Data Infrastructure Engineer to build and scale the real-time data pipelines that... ...Deep expertise with streaming and batching systems (Kafka, Spark, Flink, or Ray) operating at petabyte scale. ~ Hands-on...Senior
Judgment Labs
San Francisco, CA4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer (Apache Spark). Be the first to apply!
Related searches
- staff data engineer San Francisco, CA
- data engineering intern summer San Francisco, CA
- senior data integration developer San Francisco, CA
- data engineer graduate San Francisco, CA
- data engineer contract San Francisco, CA
- data science developer San Francisco, CA
- senior data center engineer San Francisco, CA
- software data engineer San Francisco, CA
- hadoop big data developer San Francisco, CA
- data developer San Francisco, CA