Spark Job Migration Specialist

Job Title

Job Overview: Pay Range: $60hr - $65hr

Responsibilities:

• Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks.
• Convert HiveQL scripts and Oozie workflows into optimized Spark SQL or PySpark applications.
• Refactor data pipelines from Azure Synapse to cloud platforms, updating dependencies and notebook references.
• Implement performance optimization techniques including Adaptive Query Execution (AQE) in Spark 3.
• Perform regression testing to validate data consistency between legacy and new systems.
• Develop validation scripts to ensure output accuracy and reliability.
• Customize Spark jobs using spark context configurations for better monitoring and troubleshooting.
• Reconfigure job properties, cluster settings, and Spark configurations for optimized execution.
• Ensure schema evolution, data correctness, and validation using golden datasets.

Required Skills And Qualifications:

• 5+ years of experience with Apache Spark (PySpark or Scala).
• Strong experience with Hadoop ecosystem including HDFS, Hive, HBase, and MapReduce.
• Experience in migrating data pipelines to cloud or enterprise data platforms.
• Proficiency in SQL and performance tuning techniques.
• Hands-on experience with distributed data processing frameworks.
• Experience with scripting languages such as Python, Shell, or Scala.
• Familiarity with data ingestion tools like Sqoop, Kafka, and NiFi.
• Experience working with cloud storage solutions such as ADLS, S3, or Blob Storage.

Preferred Skills And Qualifications:

• Experience with Databricks platform and notebook-based development.
• Familiarity with Azure and AWS cloud environments.
• Strong understanding of data pipeline architecture and optimization strategies.
• Experience handling large-scale data migrations and transformations.
• Knowledge of cluster tuning and Spark configuration best practices.