Spark Migration Architect for Big Data Pipelines
HMG AMERICA LLC
A tech data solutions provider is seeking a Spark job migration specialist in San Francisco, California. This role focuses on migrating data pipelines, JAR tasks, and analytics workloads to modern platforms, requiring over 5 years of experience with Apache Spark and cloud services like Azure or AWS. Responsibilities include refactoring code, performance optimization, and regression testing. Ideal candidates have strong skills in HDFS and the Hadoop ecosystem, as well as expertise in SQL and scripting. #J-18808-Ljbffr HMG AMERICA LLC
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Spark Migration Architect for Big Data Pipelines in San Francisco, CA vacancy
- ...mastering their craft. As an Architect for the Snowflake CoE you... ...of our clients’ data platform solutions. Lead... ...Amazon ECS) Experience with Big Data Tools/Frameworks: Spark, Hadoop, Presto, Amazon EMR... ...experience developing data pipelines: batch and/or streaming....PipelineBig dataTemporary workLocal area
$123.7k - $254.67k
Pinterest is seeking a Data Engineer to design and implement robust data infrastructure in AWS, using Spark and Scala. You will collaborate with cross-functional teams to evolve our core data pipelines and ensure efficient scaling as we drive massive growth. The ideal...PipelineBig data- Overview A Spark job migration specialist migrates data pipelines, JAR tasks, and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms. This involves refactoring code (e.g., Hive to PySpark), performance testing, and updating Spark...PipelineBig data
- ...Stripe's foundational data access and... ...Stripe's critical big data assets. We serve... ...hundreds of production pipelines, and every... ...executing a multi-year migration to modern, OSS-... ...Responsibilities Architect the unified Iceberg... ...compute engines — Spark, Trino, Flink, and...PipelineBig data
- ...team. This role focuses on building infrastructure for data pipelines and involves solving complex engineering challenges in a... ...distributed systems. Familiarity with AWS, Terraform, and big data frameworks like Spark is a plus. We emphasize a collaborative and fun...PipelineBig data
$185k - $300k
...Staff Data Platform Engineer (Real-time Data Platform... ...: As a high level architect (staff engineer), you will... ...multi-geographical data pipelines at scale. Familiarity... ...and client libraries for big data using tools like... ...DataProc, Kafka, Kubernetes, Spark, DataFlow, Google Cloud...PipelineBig dataairwallex
San Francisco, CA10 hours ago- ...looking for a highly skilled backend and data engineer to join their team in San... ...robust systems and scalable data pipelines, with the aim to optimize financial... ...Kotlin, or Scala. Familiarity with big data technologies such as Apache Spark and Hadoop is essential. The ideal...PipelineBig dataRelocation
$181.1k - $272.1k
...Francisco is hiring a Senior Software Data Engineer for the App Store. In this... ...for building and optimizing data pipelines, collaborating with various teams, and... ...have over 7 years of experience with big data tools like Hadoop and Spark, strong programming skills in Scala...PipelineBig data$156k - $214.5k
...challenge us with interesting new big data and AI use cases. You will... ...to support bi-directional pipeline review and bi-directional engagements... ...multi year, multi phase migration deals. Create templates,... ...creators of Lakehouse, Apache Spark™, Delta Lake and MLflow. To learn...PipelineBig dataPermanent employmentWorldwideRelocationWork visa- ...Role : GCP Data Engineer Location : Concord, CA (Onsite... ...Do (Responsibilities): Architect & Design: Design and... ...architecture for migrated workloads. Develop Reusable... ...includes creating boilerplate pipeline structures, generic... ...Data Warehouse/Big Data Expertise: Teradata...PipelineBig dataContract work
Recutify Inc.
San Francisco, CA22 hours ago - Overview We re seeking a Data Engineer with expertise in building... ...robust, scalable cloud-based data pipelines. Youll play a key role in... ...cloud-native data pipelines using Spark, Kafka, and cloud services... ...reliability, and scalability of big data platforms Requirements BS...PipelineBig dataFull time
$180k - $230k
...initiatives, with a strong focus on data‑intensive systems and foundational datasets... ...datasets, ideally in distributed or big‑data environments (e.g., Spark, Databricks, BigQuery, Snowflake,... ...of big data concepts, data pipelines, data cleaning/normalization, and data...PipelineBig dataImmediate startHome officeFlexible hours$190k - $250k
...We are seeking our first Data Engineer, someone who... ...practices for building data pipelines, and collaborate closely... ...proven track record of architecting end-to-end data... ...stack tools Familiar with big data processing frameworks (e.g., Spark, Hadoop) and data integration...PipelineBig dataHebbia
San Francisco, CA2 days ago- ...build, and maintain scalable cloud data architectures and automated solutions... ...to implement end-to-end data pipelines and workflows. Ensure automation of... ...Python, Bash, Java). Experience with big data technologies such as Hadoop, Spark, or Kafka. Strong knowledge of DevOps...PipelineBig dataContract work
$200k - $350k
...Join a well-funded AI/ML data analytics startup building... ...at scale — no heavy schema migrations or data movement required.... ...Agentic Data Engineering: Architect AI-centric ETL pipelines for data lakehouses, enabling... ..., data modeling, and big data technologies. Proven...PipelineBig dataWork at officeRemote workVisa sponsorship- ...operating the next generation of data infrastructure at Mistral AI... ...and data planes, scaling big data compute and storage... ...full lifecycle ownership: from architecting the migration away from legacy... ...implementing production‑grade pipelines and participating in on‑call...PipelineBig dataRelocation package
$140k - $200k
..., and evolution of our data platform. You will be part... ...you. What You'll Do Architect & Build: Design, build,... ...(AWS, GCP, or Azure). Pipeline Development: Develop and... ...language (e.g., Java, Go). Big Data Expertise: You... ...data technologies such as Spark, Flink, or Dask....PipelineBig dataFull time$218.4k - $365.2k
...future of Salesforce. Own the data and intelligence architecture... ...infrastructure, telemetry pipelines, marketplace measurement framework... ...and destinations. We need an architect to consolidate them into one... ...on shared standards. Migration leadership. Drive the data components...PipelineContract workLive in$220k - $350k
...you, let’s build what’s next. Principal Architect, Data Knowledge Platform Hiring Location: San... ...our data infrastructure (Databricks, Spark, Kafka, etc.), the technology to serve... ...workflows, agent-callable tools, automated pipelines, and transformation primitives. These...PipelineWork at officeLocal areaWorldwide- The mission of the Big Data Operations team is to help teams harness the power of Big Data... ...Hadoop ecosystem like HDFS, Yarn, Hive, Spark, Oozie, Presto and Ranger. MUST have... ...infrastructure automation using Terraform, CI/CD pipelines (Git, Jenkins, etc.), and configuration...PipelineBig dataWork experience placement
- ...customers. This role demands deep technical expertise in Apache Spark and data technologies while fostering customer relationships and... ...goals. The ideal candidate will have extensive experience in big data, a strong background in troubleshooting, and exceptional...Big data
$229.88k - $306.5k
...through the responsible use of data and AI. We believe that... ...are seeking a Principal Domain Architect who will define and drive the... ...latency, high-throughput data pipelines and large-scale data platforms... ...technologies such as Kafka, Kinesis, Spark, or Flink. Strong knowledge...PipelineWork experience placementWork at officeLocal areaWorldwideFlexible hours3 days per week1 day per week$210k - $385k
...Location Type Hybrid Department Data Science Compensation $210K -... ...Build the systems and pipelines that enable Search, Product, and... ...maintainable code Experience with big data systems including... ...ecosystems like Databricks or Spark Comfortable with agentic coding...PipelineBig dataFull timeLocal area- ...generalist engineers to join our team and help us build advanced data pipelines, foundational datasets, as well as the tools and... ...with distributed systems and previously administered a big data toolset (Ray, Spark, Airflow, etc.). You have hands-on experience shipping scalable...PipelineBig dataWork at officeVisa sponsorship
$189.72k - $332.01k
...of how ML works in different areas. Use data driven methods and leverage the unique properties... ...experience with building data processing pipelines, large‑scale machine learning systems, and big data technologies (e.g., Hadoop/Spark). Degree in computer science, machine...PipelineBig dataLocal areaRelocationRelocation package$141.99k - $292.13k
...hands‑on access to an incredible vault of data and contribute large‑scale recommendation... ...experience with building data processing pipelines, large scale machine learning systems, and big data technologies (e.g., Hadoop/Spark) Practical knowledge of large scale recommender...PipelineBig dataLocal areaRelocationRelocation packageFlexible hours$145k - $150k
...Code is seeking a Lead Martech Architect for a senior technical... ...architecture, leading platform migration strategy, and building the engineering... ...— API gateway standards, data contracts, event bus... ...systems, message queuing, data pipeline patterns. Experience designing...Pipeline- ...Description Description: Principal Architect — Veryon India (GCC)... ...time in legacy-to-modern migration work, architecture reviews, and... ...systems at scale. ~ Solid data architecture experience — data... ...Strong understanding of CI/CD pipelines, DevOps practices, and release...PipelineImmediate start
Veryon
San Francisco, CA7 days ago - ...Containers (using Docker/Kubernetes) and Big Data (using Hadoop/Spark). Roles & Responsibilities Design,... ...using Terraform following AWS Well-Architected Design. Deploy and configure... ...-scaling. Design and build a CI/CD Pipeline to deploy applications using Jenkins...PipelineBig dataFull time
$269k - $316k
...Checkr Checkr is building the data platform to power safe and fair... ...Experience working on big data products at scale AI first... ...‑time & batch processing data pipelines serving up billions of data points... ...tooling: AWA, EMR, Databricks, Spark, etc. Familiarity with DevOps...PipelineBig dataLocal areaRemote workFlexible hoursCheckr
San Francisco, CA10 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Spark Migration Architect for Big Data Pipelines. Be the first to apply!
Related searches
- big data developer San Francisco, CA
- hadoop big data developer San Francisco, CA
- junior big data engineer San Francisco, CA
- big data engineer San Francisco, CA
- big data devops engineer San Francisco, CA
- big data cloud engineer San Francisco, CA
- entry level big data engineer San Francisco, CA
- big data San Francisco, CA
- labor union pipeline San Francisco, CA
- pipeline tech San Francisco, CA