Spark job migration specialist

E-Solutions

Spark Job Migration Specialist

A Spark job migration specialist migrates data pipelines, JAR tasks, and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms. This involves refactoring code (e.g., Hive to PySpark), performance testing, and updating Spark 2.x to 3.x.

Key Job Responsibilities

Workload Migration: Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks.
Pipeline Re-engineering: Convert existing HiveQL scripts and Oozie workflows into optimized Spark SQL or PySpark applications.
Refactoring: Adapt data pipelines from Azure Synapse to any cloud platform, including updating library dependencies and notebook references.
Performance Optimization: Implement Adaptive Query Execution (AQE) in Spark 3 to improve shuffle performance and fix skew joins.
Testing & Validation: Perform regression testing to ensure output consistency between old and new systems using validation scripts.
Job Customization: Use spark.sparkContext.setJobDescription() to label, monitor, and troubleshoot specific Spark tasks in the UI.

Job Description/Profile

Role: Big Data Migration Engineer (Spark)

Experience: 5+ years experience with Apache Spark (PySpark/Scala) and Cloud platforms (Azure/AWS).

Requirements:

Strong experience with HDFS, Hadoop ecosystem (Hive, Spark, HBase, MapReduce).
Experience in data migration to cloud / enterprise data platforms.
Knowledge of: Data ingestion tools (Sqoop, Kafka, NiFi, etc.

Apply

Vacancy posted 4 days ago

Similar jobs that could be interesting for youBased on the Spark job migration specialist in Encino, CA vacancy

Skillbridge Performance & Optimization Associate Specialist - Recruitment Marketing (for Military members only)
...Research marketing and advertising trends in the recruiting industry. The Recruitment Marketing Performance & Optimization Associate Specialist Skillbridge trainee will support Maximus recruitment marketing teams by improving performance visibility, data accuracy, and...
Suggested
Minimum wage
Full time
Contract work
Temporary work
Traineeship
Work experience placement
Maximus
Northridge, CA
1 day ago
Sr Sourcing Specialist
...Sr Sourcing Specialist Position Summary Position requires a dynamic, self-driven team member with experience working in a manufacturing environment. The Sourcing Specialist will be an integral part of Supply Chain Team and will work closely with Supplier Quality...
Suggested
CIRCOR International
Encino, CA
16 days ago
Occupational Medicine Specialist - Woodland Hills
$136.15 per day
...Occupational Medicine Specialist - Woodland Hills Primary Location: USA-California-Woodland Hills Board Eligible Day Shift Maximum: $136.15 Board Certified Day Shift Maximum: $139.62 Nights / Weekends Maximum: $143.08 Virtual Indicator: Standard Face to...
Suggested
Work at office
Local area
Night shift
Weekend work
Day shift
SCPMG Meetings and Events (KPSymposia)
Woodland Hills, CA
3 days ago
Occupational Medicine Specialist - Woodland Hills / Oxnard
Competitive Compensation and Benefit Package The comprehensive benefits and longevity-based compensation package offered by Southern California Permanente Medical Group (SCPMG) enables physicians to focus on what they do best – provide their patients with exceptional...
Suggested
Full time
Part time
Work at office
Relocation package
Southern California Permanente Medical Group
Woodland Hills, CA
19 hours ago
Sr. Revenue Cycle Specialist: Medicare Collections
...Sr. Revenue Cycle Specialist Client is hiring Sr. Revenue Cycle Specialists who excel in Medicare Collections for our Revenue Operations Department. Successful individuals in this role are highly ambitious, results-driven, and strong in root cause analysis. This position...
Suggested
For contractors
Work at office
Tranzeal
Encino, CA
1 day ago
Mental Health Rehab Specialist
If you are driven by a desire to make a meaningful difference in your community and shape brighter futures, Help Group is the organization for you! We're one of the largest nonprofits of our kind in the U.S., offering diverse programs for individuals and families affected...
Full time
Flexible hours
The Help Group
Van Nuys, CA
19 hours ago
Galleri Mobile Draw Experience Specialist
$77k - $95k
...Galleri Mobile Draw Experience Specialist Our mission is to detect cancer early, when it can be cured. We are working to change the trajectory of cancer mortality and bring stakeholders together to adopt innovative, safe, and effective technologies that can transform...
Full time
Work experience placement
Work at office
Local area
Remote work
Flexible hours
GRAIL, Inc.
Encino, CA
19 hours ago
Veterinary Specialists - Nationwide
...Veterinary Specialists Wanted MedVet seeks veterinary specialists (board-certified or residency-trained) to join our team. The recruited candidates will join the MedVet team treating over 500,000 patients annually. You should be team-oriented and have excellent communication...
Relocation package
MedVet
Encino, CA
2 days ago
TBS Specialist - Relief
$20.19 - $23.08 per hour
...TBS Specialist - Relief L11 - Van Nuys, CA 91401 Overview Salary Range $20.19 - $23.08 Hourly Level Entry Position Type Seasonal Job Shift 1st Shift Education Level 2 Year Degree Travel Percentage None Category Nonprofit - Social Services Description ID#...
Hourly pay
Full time
Part time
Seasonal work
Remote work
Flexible hours
Shift work
Day shift
SAN FERNANDO VALLEY COMMUNITY HEALT
Van Nuys, CA
19 hours ago
Detail Specialist
$16.5 - $18.5 per hour
Compensation Range: $16.50 to $18.50 Hourly Location: 15838 Strathern St, Van Nuys, California 91406 United States of America Are you ready to rev up your career? U-Haul is looking for new team members to clean and detail the newest trucks and trailer fleet in ...
Hourly pay
Start working today
Local area
U-Haul
Van Nuys, CA
19 hours ago
Asset Protection Specialist
$23 per hour
...Job Description Position Purpose: The Asset Protection Specialist is primarily responsible for preventing financial loss caused by theft and fraud and supporting safety and environmental program compliance in their assigned store/multiple stores. They utilize tools...
Local area
Home Depot
Van Nuys, CA
19 hours ago
Sr Growth & Community Engagement Specialist (Must Live In FL)
...Growth & Community Engagement Sr Specialist Molina Healthcare of Florida is hiring several dynamic Growth & Community Engagement Sr Specialists in several areas of Florida. This role will help drive choice membership and retain existing members. This role is crucial...
Live in
Work at office
Local area
Flexible hours
Night shift
Weekend work
Early shift
Molina Healthcare
Encino, CA
2 days ago
Pursuit & Proposal Specialist
$88.8k - $115k
...Pursuit & Proposal Specialist Arcadis is the world's leading company delivering sustainable design, engineering, and consultancy solutions for natural and built assets. We are more than 34,000 people, in over 30 countries, dedicated to improving quality of life. Everyone...
Full time
Part time
Local area
ARCADIS
Encino, CA
13 days ago
Specialist II - Data Science
...Job Title: Specialist II - Data Science Location: Woodland Hills, CA Description: We are looking for 8+ Years of experienced Hands-on experience with Agentic Layer A2A frameworks and MCP Protocol. Expertise in AI/ML engineering, specifically vector embeddings...
Axelon
Woodland Hills, CA
1 day ago
Asset Protection Specialist
Job DescriptionPosition Purpose:The Asset Protection Specialist is primarily responsible for preventing financial loss caused by theft and fraud and supporting safety and environmental program compliance in their assigned store/multiple stores. They
Home Depot
Van Nuys, CA
19 hours ago
Eyewear Specialist
$20 - $24 per hour
...Eyewear Specialist TV Ladera Ranch - Landera Ranch, CA 92694 Overview Salary Range $20.00 - $24.00 Hourly Position Type Full Time- Hourly Education Level High School Travel Percentage Negligible Description Total Vision LLC is a group of highly successful...
Hourly pay
Full time
Work at office
Flexible hours
Total Vision MD LLC
Encino, CA
1 day ago
Delivery Specialist - Hub
$17.87 - $24.5 per hour
Job Title Compensation Pay Range: $17.87 - $24.50 The actual hourly rate will equal or exceed the required minimum wage applicable to the job location. Additional compensation includes annual, quarterly performance, or premiums may be paid in amounts ranging per ...
Hourly pay
Minimum wage
Local area
Immediate start
Flexible hours
O'Reilly Auto Parts
Northridge, CA
3 days ago
Facilities Specialist, Senior
...Job Title Facilities Specialist, Senior Job Description The Workplace Experience team oversees the day-to-day facilities operation & administration and related vendor management. The Facilities Specialist, Senior will report to the Senior Manager. In this role...
Full time
Contract work
Temporary work
Part time
For contractors
Work at office
Local area
Work from home
Home office
2 days per week
Blue Shield Of California
Woodland Hills, CA
22 hours ago
BusinessOperations - Authorization Specialist I - Authorization Specialist I
Prior Authorization Support Position Purpose: Supports the prior authorization request to ensure all authorization requests are addressed properly and in the contractual timeline. Aids utilization management team to document authorization requests and obtain accurate...
Temporary work
Mindlance
Woodland Hills, CA
19 hours ago
Senior Specialist, IT Business Analysis
...Senior Specialist, IT Business Analysis L3Harris Missile Solutions division is a world-recognized aerospace and defense leader providing... ..., evaluation, etc. Work with a team to develop plans to migrate business systems or interface data from Oracle to SAP or vice...
L3Harris Technologies
Canoga Park, CA
3 days ago
Specialist, System Administration TS/SCI
System/Network Administrator System/Network Administrator responsible for maintaining Windows and Linux system infrastructure, network management, system patching and hardening, database management and deployment automation. The person in this position must be able ...
MRINetwork
Encino, CA
1 day ago
Specialist
...shared responsibility and we work together to foster a culture where everyone belongs and is inspired to do their best work. As a Specialist, you build brand loyalty and create owners of Apple products and services. You’re dedicated to selling and delivering a customer...
Local area
Relocation
Night shift
Apple
Northridge, CA
3 days ago
Specialist, Growth & Community Engagement (FLORIDA)
Job Title Responsible for achieving established goals improving Molina's enrollment growth objectives encompassing Medicaid programs. Works collaboratively with key departments across the enterprise to improve overall choice rates and assignment percentages. Knowledge...
Molina Healthcare
Encino, CA
2 days ago
EDI Eligibility Specialist
...Training & development Vision insurance Wellness resources About the Role: Join MedPOINT Management as an EDI Eligibility Specialist and play a crucial role in ensuring seamless electronic data interchange processes. Be part of a dynamic team in Sherman Oaks, CA...
Remote work
MedPOINT Management
Sherman Oaks, CA
19 hours ago
Vanbody Specialist
$19.5 - $21.5 per hour
...Range: $19.50 to $21.50 Hourly Location: 15838 Strathern St, Van Nuys, California 91406 United States of America Van Body Specialist Ready to rev up your career? Are you a skilled technician interested in a rewarding career in a growing industry? Consider...
Hourly pay
Start working today
Local area
U-Haul
Van Nuys, CA
2 days ago
HL7 Integration Specialist
Autonomous Coding HL7 Engineer We are supporting Client in enabling Autonomous Coding through robust HL7 interface engineering. The work involves building, testing, and deploying HL7 interfaces (ADT, ORU, ORM, MDM, DFT, BAR) between client EMRs (Epic, Cerner, Meditech...
Software Technology Inc
Encino, CA
2 days ago
1:1 Elementary Assistive Technology Specialist (AT) - Van Nuys, CA
$41.35 - $61.35 per hour
...Bring your expertise as an Assistive Technology Specialist to an elementary school setting, where your skills will directly improve student engagement and learning. This full-time contract opportunity offers rewarding, hands-on work supporting a single student 1:1, Monday...
Full time
Contract work
Monday to Friday
Soliant Health
Van Nuys, CA
1 day ago
Delivery Specialist
$16.9 - $24.9 per hour
...Delivery Specialist Compensation Pay Range: $16.90 - $24.90 The actual hourly rate will equal or exceed the required minimum wage applicable to the job location. Additional compensation includes annual, quarterly performance, or premiums may be paid in amounts ranging...
Hourly pay
Minimum wage
Local area
Immediate start
Flexible hours
O'Reilly Auto Parts
Woodland Hills, CA
3 days ago
Specialist, Appeals & Grievances (Member Medicaid/Marketplace experience)
$14.76 - $31.97 per hour
Job Title Provides support for claims activities including reviewing and resolving member and provider complaints, and communicating resolution to members or authorized representatives in accordance with the standards and requirements established by the Centers for ...
Hourly pay
Full time
Contract work
Work experience placement
Work at office
Molina Healthcare
Encino, CA
4 days ago
PayPal Dispute Resolution Specialist
...We have an opening for a talented PayPal Dispute Resolution Specialist to perform daily responsibilities with dedication. Ensure compliance with company and safety standards. Provide excellent interactions with customers and colleagues. Perks include competitive pay,...
Flexible hours
TradeJobsWorkforce
Reseda, CA
19 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Spark job migration specialist. Be the first to apply!