Data Engineer

Gke Data Engineering Lead

Our client is in need of a GKE Data Engineering Lead for a 2 year engagement to assist in solutioning the architecture as we transform the rebate processing system from legacy platform. The potential candidate will be helping the engineers in rearchitecting, transforming, and designing data pipelines.

This resource will come in and help do a current state assessment, recommend the framework and lead team in building the solution for the migration.

DataProc, Spark, AirFlow, Composer, etc...

Required Experience:

• 3 plus years of experience with Apache Airflow or Composer
• 3 plus years of experience with Data Proc or Spark
• 5 7 years of experience with CI/CD
• Experience completing current state assessments for migration projects
• Experience with producing architecture solutions with Composer and Spark
• Java Engineers and Architects will work with these Data Engineers to understand and help build out that environment
• They will be part and parcel of the work they are doing, helping the rest of the team to build out the environment based on the recommendations they put forth in their assessments
• Needs to be a leader, understands how to assess problems and navigate the team through any issues
• Must understand Java - it is what the environment operates on.
• Must be proactive as they need a leader to help direct the flow of the group's work.
• Healthcare experience desired
• Experience with producing architecture solutions with Composer and Spark
• Experience in large scale batch processing with cloud services.

Desired Skills: Apache Spark, Map Reduce, Spark Streaming, PySpark, Hive, HDFS, Kafka, Redis, Sqoop, Oozie, IBM Data Sage, Springbatch

Required Skills: CI/CD, API, Java