Spark Migration Architect for Big Data Pipelines
HMG America LLC
A tech data solutions provider is seeking a Spark job migration specialist in San Francisco, California. This role focuses on migrating data pipelines, JAR tasks, and analytics workloads to modern platforms, requiring over 5 years of experience with Apache Spark and cloud services like Azure or AWS. Responsibilities include refactoring code, performance optimization, and regression testing. Ideal candidates have strong skills in HDFS and the Hadoop ecosystem, as well as expertise in SQL and scripting. #J-18808-Ljbffr HMG America LLC
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Spark Migration Architect for Big Data Pipelines in San Francisco, CA vacancy
- ..., and delivering holistic data solutions leveraging Snowflake... ...their craft. As an Architect for the Snowflake CoE you... ...Amazon ECS)* Experience with Big Data Tools/Frameworks: Spark, Hadoop, Presto, Amazon EMR... ...experience developing data pipelines: batch and or streaming.*...PipelineBig dataTemporary workLocal area
- Overview A Spark job migration specialist migrates data pipelines, JAR tasks, and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms. This involves refactoring code (e.g., Hive to PySpark), performance testing, and updating Spark...PipelineBig data
- ...learning (ML) and deep learning models Building scalable data pipelines for preprocessing, feature engineering, and model... ...frameworks like TensorFlow, PyTorch, or JAX ~ Experience with big data tools (Apache Spark, Kafka, Hadoop) and MLOps platforms ~ Familiarity with...PipelineBig dataFull time
Signal Fire Inc
San Francisco, CA17 hours ago - ...hands-on access to an incredible vault of data and contribute large-scale recommendation... ...experience with building data processing pipelines, large scale machine learning systems, and big data technologies (e.g., Hadoop/Spark) ~ Practical knowledge of large scale recommender...PipelineBig dataFull timeRelocationRelocation package
Pinterest
San Francisco, CA17 hours ago - ...models ✔ Experienced in architecting scalable AI systems and... ...scalable training and inference pipelines for AI-powered... ...closely with engineering, data, and product teams to integrate... ...systems ~ Familiarity with big data tools (Apache Spark, Kafka, Hadoop) and MLOps...PipelineBig dataFull time
Signal Fire Inc
San Francisco, CA17 hours ago - ...Design, implement, and manage CI/CD pipelines to facilitate seamless code... ...similar. Configure and maintain data infrastructure appliances. Troubleshoot... ...Kubeflow, or SageMaker). Knowledge of big data technologies (e.g., Hadoop, Spark, or Kafka). Experience with...PipelineBig dataFull time
Maven Robotics
San Francisco, CA17 hours ago - ...designing and implementing machine learning models and data pipelines to enhance our programmatic demand-side platform (DSP).... .... Proficiency in Python and SQL and familiarity with big data tools (e.g., Spark) and ML libraries (e.g., TensorFlow, PyTorch, Scikit-Learn...PipelineBig dataFull time
Rzr Global Inc.
San Francisco, CA17 hours ago $122k - $240.5k
Position Summary Google AI Architect/AI and EngineeringJoin our AI... ...solutions in software, data, AI, network, and hybrid cloud... ...end architectures across data pipelines, feature engineering, model... ...or Jenkins.3+ years executing migration or modernization programs to...PipelineLocal areaVisa sponsorshipFlexible hours$206.4k - $379.1k
...looking for a Principal Architect to build and implement... ...in distributed systems, data architecture, and large... ...systems, data pipelines, caching and storage layers... ...deployment — employing Spark, Kafka, Flink, and other... ...organization. The next big idea could be yours. Let...PipelineFull timeTemporary workLocal areaWorldwideFlexible hours- We're looking for a Manager, Data Engineering with deep Big Data expertise to design, develop, and maintain scalable data pipelines and analytics solutions. In this role, you'll partner... ...Big Data technologies such as Hadoop, Spark, Kafka, and FlinkDevelop and optimize data...PipelineBig dataHourly payFreelance
$140.8k - $176k
...about solving business problems using data and working in a dynamic, creative,... ...stack runs on AWS, Kubernetes, Go, Spark, Python and Apache Airflow. In this... ...~ Nice-to-have: Experience with big data processing / distributed data pipelines and tools such as Apache Airflow and...PipelineBig dataHourly payFull timeWork experience placementWork at officeLocal area3 days per weekLyft
San Francisco, CA17 hours ago- ...team. This role focuses on building infrastructure for data pipelines and involves solving complex engineering challenges in a... ...distributed systems. Familiarity with AWS, Terraform, and big data frameworks like Spark is a plus. We emphasize a collaborative and fun...PipelineBig data
- ...Cruise Line - The Walt Disney Company is seeking a senior big data engineer to design and maintain Identity and Device data... ...call participation, code reviews, and advancing AWS-based data pipelines using Databricks, Spark, and Airflow. #J-18808-Ljbffr Disney Cruise LinePipelineBig data
- ...new accounts, generate fresh pipeline, and exceed quota through new... ...Experienced (3+ years) high-growth Data/AI/Infrastructure leader with... ...the c-suite. Experience in big data, selling against open source... ...creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To...PipelineBig dataWorldwide
- ...Coding or comparable frameworks. Exposure to modern AI/ML pipelines for personalization, segmentation, or content automation. Strong proficiency in big data technologies: Scala, Databricks, Spark SQL, Spark Streaming, Python. Cloud engineering experience...PipelineBig data
NMK GLOBAL, INC
San Francisco, CA1 day ago $141.2k - $278.3k
...Summary Google AI Lead Architect/AI & Engineering:Join our AI... ...sector solutions in software, data, AI, network, and hybrid... ...end architectures across data pipelines, feature engineering, model lifecycle... ...Jenkins.3+ years executing migration or modernization programs to...PipelineLocal areaVisa sponsorshipFlexible hours$138.91k - $285.98k
...hands-on access to an incredible vault of data and contribute large-scale recommendation... ...experience with building data processing pipelines, large scale machine learning systems, and big data technologies (e.g., Hadoop/Spark)Bachelor’s degree in computer science, machine...PipelineBig dataLocal areaRelocation package- ...the leading edge of the Unified Data Analytics and AI space. Our... ...an informed point of view on Big Data, Advanced Analytics and AI... ...and use cases.Exceed activity, pipeline, and revenue targets.Track all... ...Intelligence Platform powered by Apache Spark and Delta LakePrioritize...PipelineBig dataLocal areaWorldwide
$138.91k - $285.98k
...representation learning. We need practical, end-to-end experience building data processing pipelines, large-scale machine learning systems, and working with big data technologies such as Hadoop or Spark. We require a bachelors degree in computer science, machine learning...PipelineBig dataFull timeRemote workRelocation packageFlexible hoursPinterest, Inc.
San Francisco, CA1 day ago- ...MinIO is the data and memory foundation for enterprise AI. Built... ...of their data. The Field Architect role at MinIO is a senior customer... ..., high-performance data pipelines, large object workloads, and... ...such as Databricks, Starburst, Spark, Trino, Iceberg, Delta Lake,...PipelineFlexible hours
Minio, Inc.
San Francisco, CA1 day ago $260k - $350k
...customer-facing software to the data platform that will power... ..., and efficiency. You'll architect data platforms, manage streaming pipelines, optimize cloud... ...Comprehensive knowledge of modern big data5+ years experience... ...as Databricks, Kafka, Spark, etc.Solid foundation in...PipelineBig dataFull timeWork at officeImmediate start$179.4k - $263.12k
...About the RoleYou are a Data Engineer, who is... ...robust and extensible big data systems that support... ...technologies (Hadoop, M/R, Hive, Spark, Flink, Metastore,... ...Transformation & Loading) and architecting data systemsExperience... ..., and optimize data pipelines and large-scale data...PipelineBig dataFull time- ..., and evolution of our data platform. You will be part... ...What You'll Do Architect & Build: Design, build,... ..., GCP, or Azure). Pipeline Development: Develop and... ...language (e.g., Java, Go). Big Data Expertise: You... ...technologies such as Spark, Flink, or Dask....PipelineBig data
Crustdata (YC F24)
San Francisco, CA3 days ago $300k - $400k
...bidding and digital marketing data). You will work closely... ...~ ML System Design: Architect and evolve the end-to-end machine learning pipeline – from data ingestion... ...Strong experience with big data and streaming frameworks (e.g., Apache Spark, Kafka, Hadoop) for processing...PipelineBig dataFull timeZeta Global
San Francisco, CA17 hours ago$228.6k - $317.23k
P-1110At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next... ...team, responsible for Databricks ETL product line: Jobs, Spark Declarative Pipelines, and Genie Code for Data Engineering. We run one of the...PipelineBig dataLocal areaWorldwide- ...Employers 2022 List.Front’s Data team builds the... ...end: designing scalable pipelines and data models, raising... ....What will you be doing?Architect data pipelines that provide... ...knowledge of at least one big data technology (HDFS, EMR, Redshift, Spark, Flink, or Presto)Experience...PipelineBig dataWork at officeImmediate startRemote workWork from homeMonday to Friday
$246.5k - $339k
...we're using the power of tech, data, and machine learning to... ...workflowsProductionize ML workloads using Spark, Delta Lake, MLflow, and... ...multi-tenant usageBuild CI/CD pipelines for ML using Terraform and Git... ...FrameworksPyTorch, MLFlow Big Data & ProcessingSpark, Kafka,...PipelineBig dataWork experience placementWork at officeLocal areaRemote workMonday to FridayFlexible hours3 days per week$141.1k - $190.91k
...a rich and variety of science data. To keep up our innovation, Benchling... ...-facing data warehouse. The Big Data Infrastructure team is... ...data transformations and pipelines for cross-functional datasets,... ...processing technologies (e.g. Kafka, Spark), schema design, and SQL are a...PipelineBig dataFull timeTemporary workWork at officeLocal areaRemote workHome officeFlexible hours3 days per weekBenchling
San Francisco, CA17 hours ago- ...technical leader responsible for architecting, building, and scaling next... ...expertise with modern big data engineering, cloud-native design... ...data processing, and activation pipelines. Architect scalable,... ...applications using Scala, Databricks, Spark SQL, Spark Streaming, and...PipelineBig data
NMK GLOBAL, INC
San Francisco, CA1 day ago - ...generalist engineers to join our team and help us build advanced data pipelines, foundational datasets, as well as the tools and... ...with distributed systems, and previously administered a big data toolset (Ray, Spark, Airflow, etc.). You have hands-on experience...PipelineBig dataWork at officeVisa sponsorship
Spellbrush
San Francisco, CAmore than 2 months ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Spark Migration Architect for Big Data Pipelines. Be the first to apply!
Related searches
- big data devops engineer San Francisco, CA
- big data developer San Francisco, CA
- big data cloud engineer San Francisco, CA
- hadoop big data developer San Francisco, CA
- big data engineer San Francisco, CA
- entry level big data engineer San Francisco, CA
- big data San Francisco, CA
- junior big data engineer San Francisco, CA
- pipeline construction San Francisco, CA
- pipeline surveying San Francisco, CA