Spark Migration Architect for Big Data Pipelines
HMG America LLC
A tech data solutions provider is seeking a Spark job migration specialist in San Francisco, California. This role focuses on migrating data pipelines, JAR tasks, and analytics workloads to modern platforms, requiring over 5 years of experience with Apache Spark and cloud services like Azure or AWS. Responsibilities include refactoring code, performance optimization, and regression testing. Ideal candidates have strong skills in HDFS and the Hadoop ecosystem, as well as expertise in SQL and scripting. #J-18808-Ljbffr HMG America LLC
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Spark Migration Architect for Big Data Pipelines in San Francisco, CA vacancy
- ...mastering their craft. As an Architect for the Snowflake CoE you... ...of our clients’ data platform solutions. Lead... ...Amazon ECS) Experience with Big Data Tools/Frameworks: Spark, Hadoop, Presto, Amazon EMR... ...experience developing data pipelines: batch and/or streaming....PipelineBig dataTemporary workLocal area
- ...Solution Delivery Agentic AI Architect-Anthropic Partnership... ...ServiceNow, enterprise data, business processes,... ...services, databases, big‑data technologies, and... ...and feature-engineering pipelines that support reliable model... ...such as Apache Spark and Snowflake for large...PipelineBig data
$151k - $253k
...Microsoft Fabric - L55 - Fabric Architect/Engineer Join our AI &... ...sector solutions in software, data, AI, network, and hybrid... ...Fabric implementation and migration engagements — spanning Lakehouse... ...definitionsStand up ETL/pipeline work using Spark Notebooks, Dataflow Gen2,...PipelineLocal areaVisa sponsorshipFlexible hours- Overview A Spark job migration specialist migrates data pipelines, JAR tasks, and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms. This involves refactoring code (e.g., Hive to PySpark), performance testing, and updating Spark...PipelineBig data
$197.3k - $313.7k
...distributed systems, data-intensive analytics... ...services and ETL pipelines that feed large-scale... ...customers..• Architect database consolidation and migration efforts using PostgreSQL... ...integration with big data query and processing... ...: Apache Spark, Trino/Presto, Parquet...PipelineBig dataLong term contractFull time- ...hands-on access to an incredible vault of data and contribute large-scale recommendation... ...experience with building data processing pipelines, large scale machine learning systems, and big data technologies (e.g., Hadoop/Spark) ~ Practical knowledge of large scale recommender...PipelineBig dataFull timeRelocationRelocation package
Pinterest
San Francisco, CA19 hours ago $206.4k - $379.1k
...looking for a Principal Architect to build and implement... ...in distributed systems, data architecture, and large... ...systems, data pipelines, caching and storage layers... ...deployment — employing Spark, Kafka, Flink, and other... ...organization. The next big idea could be yours. Let...PipelineFull timeTemporary workLocal areaWorldwideFlexible hours$122k - $240.5k
Position Summary Google AI Architect/AI and EngineeringJoin our AI... ...solutions in software, data, AI, network, and hybrid cloud... ...end architectures across data pipelines, feature engineering, model... ...or Jenkins.3+ years executing migration or modernization programs to...PipelineLocal areaVisa sponsorshipFlexible hours- ...digital economy. The role focuses on graph-based ML and large-scale data pipelines to improve deceased monitoring and compliance products,... ..., evaluate new data sources, and support data processing with Spark, PySpark, and AWS, while communicating insights to cross-functional...PipelineBig data
- ...AWS Data Engineer Location: San Francisco and jersey... ...Have Skills : Spark, AWS, data lake, data pipelining python Job... ...Leverage Spark and other big data technologies to process... ..., AWS Certified Solutions Architect, or other relevant certifications...PipelineBig dataRemote work
Apex Informatics
San Francisco, CA1 day ago - ...performance marketers. We leverage massive data and cutting-edge science to automate... ...systems or multi-agent simulations Big data experience with Scala and Spark MLOps experience — model deployment, monitoring, and pipeline orchestration on AWS In-Office...PipelineBig dataFull timeWork at officeRemote workRelocationRelocation package
Pinterest
San Francisco, CA19 hours ago - ...team. This role focuses on building infrastructure for data pipelines and involves solving complex engineering challenges in a... ...distributed systems. Familiarity with AWS, Terraform, and big data frameworks like Spark is a plus. We emphasize a collaborative and fun...PipelineBig data
- ...Cruise Line - The Walt Disney Company is seeking a senior big data engineer to design and maintain Identity and Device data... ...call participation, code reviews, and advancing AWS-based data pipelines using Databricks, Spark, and Airflow. #J-18808-Ljbffr Disney Cruise LinePipelineBig data
- We're looking for a Manager, Data Engineering with deep Big Data expertise to design, develop, and maintain scalable data pipelines and analytics solutions. In this role, you'll partner... ...Big Data technologies such as Hadoop, Spark, Kafka, and FlinkDevelop and optimize data...PipelineBig dataHourly payFreelance
- ...Senior Data EngineerSAN FRANCISCO, CA / ENGINEERING / FULL-TIMEWhat... ..., reliable, scalable data pipelines that process over billions... ..., augmenting, and helping architect our data platform and data warehouses... ....Experience with Apache Spark or other Big Data frameworks; working...PipelineBig dataWork experience placement
1872 Consulting
San Francisco, CA4 days ago - ...Lead Data Engineer - Job DescriptionResponsibilitiesDevelopment... ...teams.Develop and redesign data pipelines using Kafka streams.Implement... ...Spring Boot Java and Databricks Spark streaming.Leadership Duties:... ...processing, data platform, data lake, big data, data warehouse, or...PipelineBig data
My3Tech Inc
San Francisco, CA4 days ago - ...performance marketers. We leverage massive data and cutting-edge science to... ...differences, or incrementality testing ~ Big data experience with Scala and Spark ~ Systems programming experience... ...model deployment, monitoring, and pipeline orchestration on AWS In-...PipelineBig dataFull timeWork at officeRemote workRelocationRelocation package
Pinterest
San Francisco, CA19 hours ago - ...engineers who are excited to build the data foundation for the identity layer... ...datalake, the streaming and batch pipelines that feed it, a multi-provider... ...it. ~ Hands-on experience with big-data or streaming technologies such as Spark, Kafka, or Flink, and interest in...PipelineBig dataFull timeFor contractorsInternshipImmediate start
Persona
San Francisco, CA19 hours ago $140.8k - $176k
...about solving business problems using data and working in a dynamic, creative,... ...stack runs on AWS, Kubernetes, Go, Spark, Python and Apache Airflow. In this... ...~ Nice-to-have: Experience with big data processing / distributed data pipelines and tools such as Apache Airflow and...PipelineBig dataHourly payFull timeWork experience placementWork at officeLocal area3 days per weekLyft
San Francisco, CA19 hours ago- ...Lead Data Engineer With MartechLocation: SFO, CA (Hybrid 2 days a week... ...data processing, and activation pipelines.Architect scalable, event-driven systems... ...tools. Develop high-performance big-data applications using Scala, Databricks, Spark SQL, Spark Streaming, and Python...PipelineBig data2 days per week
Georgia IT Inc
San Francisco, CA4 days ago - ...Senior Data Scientist – AI / Machine LearningLocation: Bay Area, CA (Open/Hybrid... ..., prompt engineering, RAG pipelines, and agentic workflows (where applicable... ...client-facing environmentsKnowledge of big data technologies (Spark, Databricks)Prior experience leading...PipelineBig data
ClifyX
San Francisco, CA4 days ago $138.91k - $285.98k
...hands-on access to an incredible vault of data and contribute large-scale recommendation... ...experience with building data processing pipelines, large scale machine learning systems, and big data technologies (e.g., Hadoop/Spark)Bachelor’s degree in computer science, machine...PipelineBig dataLocal areaRelocation package$180k - $230k
...initiatives, with a strong focus on data‑intensive systems and foundational datasets... ...datasets, ideally in distributed or big‑data environments (e.g., Spark, Databricks, BigQuery, Snowflake,... ...of big data concepts, data pipelines, data cleaning/normalization, and data...PipelineBig dataImmediate startHome officeFlexible hoursSentiLink
San Francisco, CA2 days ago- ...the leading edge of the Unified Data Analytics and AI space. Our... ...an informed point of view on Big Data, Advanced Analytics and AI... ...and use cases.Exceed activity, pipeline, and revenue targets.Track all... ...Intelligence Platform powered by Apache Spark and Delta LakePrioritize...PipelineBig dataLocal areaWorldwide
- ...the leading edge of the Unified Data Analytics and AI space. Our... ...an informed point of view on Big Data, Advanced Analytics and AI... ...and use cases.Exceed activity, pipeline, and revenue targets.Track all... ...Intelligence Platform powered by Apache Spark and Delta LakePrioritize...PipelineBig dataLocal areaWorldwide
$140k - $200k
..., and evolution of our data platform. You will be part... ...you. What You'll Do Architect & Build: Design, build,... ...(AWS, GCP, or Azure). Pipeline Development: Develop and... ...language (e.g., Java, Go). Big Data Expertise: You... ...data technologies such as Spark, Flink, or Dask....PipelineBig dataFull timeCrustdata (YC F24)
San Francisco, CA1 day ago- ...At Etleap, we're redefining how data teams build and manage data pipelines. Our no-code/low-code ETL platform allows... ...to San Francisco or London Big Plus If You Have The Following... ...of current big data frameworks like Spark or Flink, but this is not an absolute...PipelineBig dataWork at officeRelocation
Etleap
San Francisco, CA19 hours ago $179.4k - $263.12k
...About the RoleYou are a Data Engineer, who is... ...robust and extensible big data systems that support... ...technologies (Hadoop, M/R, Hive, Spark, Flink, Metastore,... ...Transformation & Loading) and architecting data systemsExperience... ..., and optimize data pipelines and large-scale data...PipelineBig dataFull time$260k - $350k
...customer-facing software to the data platform that will power... ..., and efficiency. You'll architect data platforms, manage streaming pipelines, optimize cloud... ...Comprehensive knowledge of modern big data5+ years experience... ...as Databricks, Kafka, Spark, etc.Solid foundation in...PipelineBig dataFull timeWork at officeImmediate start$228.6k - $317.23k
P-1110At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next... ...team, responsible for Databricks ETL product line: Jobs, Spark Declarative Pipelines, and Genie Code for Data Engineering. We run one of the...PipelineBig dataLocal areaWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Spark Migration Architect for Big Data Pipelines. Be the first to apply!
Related searches
- big data devops engineer San Francisco, CA
- big data San Francisco, CA
- big data developer San Francisco, CA
- big data cloud engineer San Francisco, CA
- junior big data engineer San Francisco, CA
- big data engineer San Francisco, CA
- hadoop big data developer San Francisco, CA
- entry level big data engineer San Francisco, CA
- pipeline San Francisco, CA
- gas pipeline San Francisco, CA