Spark job migration specialist
E-Solutions
Spark Job Migration Specialist
Location: SFO, CA - Remote
Duration: Long-Term Contract
A Spark job migration specialist migrates data pipelines, JAR tasks, and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms. This involves refactoring code (e.g., Hive to PySpark), performance testing, and updating Spark 2.x to 3.x.
Key Job Responsibilities
- Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks.
- Convert existing HiveQL scripts and Oozie workflows into optimized Spark SQL or PySpark applications.
- Adapt data pipelines from Azure Synapse to any cloud platform, including updating library dependencies and notebook references.
- Implement Adaptive Query Execution (AQE) in Spark 3 to improve shuffle performance and fix skew joins.
- Perform regression testing to ensure output consistency between old and new systems using validation scripts.
- Use spark.sparkContext.setJobDescription() to label, monitor, and troubleshoot specific Spark tasks in the UI.
Job Description/Profile
- Role: Big Data Migration Engineer (Spark)
- Experience: 5+ years experience with Apache Spark (PySpark/Scala) and Cloud platforms (Azure/AWS).
- Requirements:
- Strong experience with HDFS, Hadoop ecosystem (Hive, Spark, HBase, MapReduce).
- Experience in data migration to cloud / enterprise data platforms.
- Knowledge of: Data ingestion tools (Sqoop, Kafka, NiFi, etc.) Cloud storage (ADLS, S3, Blob Storage) Distributed processing frameworks SQL and performance tuning expertise. Experience in scripting (Python, Shell, Scala).
- Key Migration Focus Areas
- Data Pipelines: Ensuring schema evolution, data correctness, and testing with golden datasets.
- Job Definitions: Reconfiguring job properties, cluster settings, and Spark configurations.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Spark job migration specialist in United States vacancy
$60 - $65 per hour
...Job Title Job Overview: Pay Range: $60hr - $65hr Responsibilities: Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks. Convert HiveQL scripts and Oozie workflows into optimized Spark SQL or PySpark applications. Refactor...Suggested- ...CRM Migration Specialist / CRM Lead (HubSpot to Microsoft Dynamics) - REMOTE Hyderabad, Telangana, India Or refer someone Job Openings CRM Migration Specialist / CRM Lead (HubSpot to Microsoft Dynamics) - REMOTE Hi All, We have urgent requirement for CRM Lead...SuggestedRemote work
$169k - $228.6k
...hobbyists, startups, not-for-profits, or large multi-national organizations. AWS Specialist and Partner Industries (ASPI) is seeking an AWS Specialist Solutions Architects (SSA) for Migrations. AWS SA Specialists are technologists with deep domain-specific expertise, able...SuggestedWorldwideFlexible hours- ...Job Title: Oracle Fusion Migration Specialist Job Description: We are seeking a highly skilled and experienced Oracle Fusion Migration Specialist to join our team. The ideal candidate will have a strong background in Oracle technologies, particularly in migration...SuggestedRemote work
- ...to ensure a fair and empowering workplace for all. Job Description We are seeking a detail-oriented and client-focused Migration Specialist to manage and support client data migration processes. This role involves working closely with clients through remote access...SuggestedRemote work
- ...Role Overview We are seeking a highly skilled and experienced Technical Lead to drive the migration and integration of Microsoft 365 services, Entra ID (formerly Azure AD), and Active Directory environments during Merger & Acquisition (M&A) activities. This role...
- ...Role : Mainframe Migrations specialist (Data Migration Mainframe to RDBMS) Location : Dallas TX In one sentence You will be extracting data from a variety of legacy mainframe databases, mapping the raw data to new database systems in either the...
- ...TIBCO Migration Specialist Mode of work: Remote (but travel required - eexpenses covered) Duration: 5+ months Contract description: Analyze existing TIBCO applications and services to understand their architecture and functionality Develop a comprehensive...Contract workRemote work
- ...Job Description - FileMaker Migration Specialist Introduction The Agency seeks an experienced FileMaker professional to support the modernization of its database infrastructure by migrating from FileMaker Server v13 (on-premise) to FileMaker Cloud . This...Remote work
- 2 openings - URGENT Job Title: Contentful CMS Migration Specialist Remote, USA Client- Hanson and dodge Rate- OPEN 4-6 months Visa- NO H1-B Project Overview We are supporting a large enterprise client based...Remote work
- ...PID: 26-14-09-020-WSU Summary E-INFOSOL LLC is seeking a highly motivated Senior SharePoint Migration Specialist to be a full-time employee in Washington, DC to join our innovative and customer solutions focused team. Job Description We are seeking...Full timeContract workTemporary workLocal area
- ...Maximo Mas Migration AWS COTS Migration Specialist Must be a US Citizen who currently possesses a Top Secret Clearance. Statement of Work: The FBI currently utilizes Maximo as its enterprise supply chain management solution. Branded as the Asset Management System (AMS...Work at office
- ...Overview: Key Responsibilities: Plan, design, and execute Office 365 tenant-to-tenant migration projects (mailboxes, OneDrive, SharePoint, Teams). Manage and administer Microsoft Exchange Online and hybrid Exchange environments. Perform cutover...Work at office
- ...SharePoint Migration and Power Platform Specialist Onsite role in Sacramento, CA (Local) USC/GC 12- Months Contract Job Description: We are seeking a highly skilled and experienced SharePoint Migration and Power Platform Specialist...Contract workLocal areaAfternoon shift
$95k - $125k
...Business Enablement Business Solutions Project Assurance About the Role We are seeking a Data Conversion Specialist who will lead the migration of legacy data into a modern SaaS platform. This role requires a unique blend of technical proficiency and business...For contractorsH1bWork at officeLocal areaRemote work- ...Job Title: Tableau to Databricks Migration Specialist Location: Remote Type: Contract Duration: Long Term Job Summary: We are seeking a highly skilled Tableau to Databricks Migration Specialist to lead and execute the migration...Contract workRemote work
- ...I have an opportunity for " Storage Migration Specialist " _ (REMOTE )" and I am looking for a candidate who can join Immediately if you are interested, reply to me with your updated resume or if you could refer someone I would really appreciate it. Position: Storage...Immediate startRemote work
$40 - $60 per hour
...professional for a contract position requiring FCCS and HFM experience. Responsibilities include system testing during HFM functionality migration, validating data migrations, addressing user queries, and performing functional tests on webform changes in FCCS. Compensation...Hourly payContract work$40 - $60 per hour
...candidate for a contract position requiring expertise in FCCS and HFM. Responsibilities include testing systems, validating data migrations, and acting as a technical support liaison for users. Competitive hourly rates range from $40 to $60, with additional benefits such...Hourly payContract work$12 per hour
...Job title – NetApp to PowerScale Migration Specialist (DAT2024324) Mode C2C / W2 / 1099 Duration 1 Years + Location Remote / Offshore Rate $ 12/- per hour Overview As a NetApp to PowerScale Migration Specialist, you will play a crucial role in facilitating the seamless...Hourly payRemote work$35k
...end of December 2026. This is a Hybrid position reporting to the Columbus, GA office Tues-Thurs. Job Summary: The Migration Specialist within Togetherwork's Higher Education business vertical plays a key role in ensuring our customers experience a smooth transition...Temporary workWork at officeLocal areaRemote workFlexible hours- ...Job Title: Windows 11 Migration Specialist(SCCM & Tanium) Location: Remote Position Overview: We are seeking a highly skilled and experienced windows 11 migration specialist with expertise in SCCM(System Center Configuration Manager) and...Remote work
$29.5 - $47.2 per hour
...supervision of a group of children in a seasonal day camp. The Camp Specialist shall design and deliver a program specialty in their... ...that are age specific and that keep children's interests high. Sparks is a half-day camp which will require Specialists to design their...Hourly paySummer workSeasonal workImmediate start- The County is migrating all Internet sites from an on-premises SharePoint farm to a cloud hosted Adobe Experience Manager platform. We are looking for someone who can help accelerate our migration by helping to: Organize site migrations Communicate with site owners Guide...Local area3 days per week
- ...Intune Migration Specialist client- Hexaware/Carlyle group, location NY day 1 New York - ONSITE Any Visa except OPT / CPT C2C or W2 both are ok Job Description: We are seeking an experienced Intune Migration Specialist to join our IT team. The ideal...
- ...Comsense Enablement & Migration Specialist Reports to: VP, Technology & Strategic Initiatives Job Type: Contractor Job Summary We are seeking an experienced Comsense Specialist to support ERP migration projects for newly acquired companies within the commercial...For contractorsWork at officeLocal area
- ...cutting-edge AI, ML, and cloud solutions. Join our team and help shape the future of technology innovation! Python Streaming Specialist (Kafka & Spark Streaming) Location: Alpharetta, GA (willing to travel to client locations) Employment Type: Full-Time (W2) Role...Full time
- ...Microsoft 365 Migration Specialist Design New York April 23, 2021 $ 250 - $ 300 / month Part Time Urgent Job Overview Date Posted April 23, 2021 Location New York Offered Salary $ 250 - $ 300 / month Expiration date April 20, 2025 Experience 4...Part time
- ...Microfocus to OpenText Migration Specialist ITnova is seeking an experienced professional to support a system migration project for the Port Authority of New York and New Jersey (PANYNJ). The role involves migrating records management data from Microfocus Content Manager...Contract workPart timeRemote workFlexible hours
- ...Start smart: PLM built for the way your business really works. Database Migration Specialist Now Hiring: Database Migration Specialist Job Description A Database Migration Specialist is responsible for planning, executing, and validating the transfer of data between databases...Immediate startFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Spark job migration specialist. Be the first to apply!
Related searches
- vetting specialist United States
- protection specialist United States
- mental health specialist United States
- outreach specialist United States
- fixed income specialist United States
- community outreach specialist United States
- registration specialist United States
- accessibility specialist United States
- intake specialist United States
- bilingual specialist United States

