Data Engineer

SR DATA ENGINEER (GCP/SCALA/SPARK/PYTHON)Prep/Feedback on others historically:Lacking experience with Spark processing and Big Data DevelopmentNeeding strong coding experience with Scala/Python for pipeline development (we mostly use scala – so need JVM skills)GCP experience is preferred, but AWS or Azure would be a comparable for us.Responsible for construction and development of "large-scale cloud data processing systems" The Data Engineer must have considerable expertise in data warehousing and the job requires proven coding expertise with Python, Scala, SQL, and Spark languages.Must be able to implement enterprise cloud data architecture designs, and will work closely with the rest of the scrum team and internal business partners to identify, evaluate, design, and implement large scale data solutions, structured and unstructured, public and proprietary data.The Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.Desired Work Experience:Proficient in multiple programming languages, frameworks, domains, and tools.Coding skills in Scala or Python, and Spark ProgrammingExperience with gcp platform development tools Pub/sub, cloud storage, big table, big query, data flow, data proc, and composer desired.Strong Linux/Unix background and hands on knowledge.Knowledge in Hadoop and cloud platforms and surrounding ecosystems.Experience with web services and APIs as in RESTful and SOAPAbility to document designs and conceptsAPI Orchestration and Choreography for consumer appsWell rounded technical expertise in Apache packages and Hybrid cloud architecturesPipeline creation and automation for Data AcquisitionMetadata extraction pipeline design and creation between raw and finally transformed datasetsQuality control metrics data collection on data acquisition pipelinesAble to collaborate with scrum team including scrum master, product owner, data analysts, Quality Assurance, business owners, and data architecture to produce the best possible end productsExperience contributing to and leveraging jira and confluence.Strong experience working with real time streaming applications and batch style large scale distributed computing applications using tools like Spark, Kafka, Flume, pubsub, and airflow.Ability to work with different file formats like Avro, Parquet, and JSON.Managing and scheduling batch jobs.Hands on experience in Analysis, Design, Coding and Testing phases of Software Development Life Cycle (SDLC)Responsibilities:Works on problems of diverse scope and complexity ranging from moderate to substantialAssists senior professionals in determining methods and procedures for new tasksLeads basic or moderately complex projects/activities on semi-regular basisMust possess excellent written and verbal communication skillsAbility to understand and analyze complex data setsExercises independent judgment on basic or moderately complex issues regarding job and related tasksMakes recommendations to management on new processes, tools and techniques, or development of new products and services Makes decisions regarding daily priorities for a work group; provides guidance to and/or assists staff on non-routine or escalated issuesAble to effectively communicate highly technical information to numerous audiences, including management, the user community, and less-experienced staff.? Consistently communicate on status of project deliverablesConsistently provide work effort estimates to management to assist in setting prioritiesDeliver timely work in accordance with estimatesSolve problems as they arise and communicate potential roadblocks to manage expectationsAdhere strictly to all security policiesRequired Skills : Python and/or Scala Coding, Spark, Hadoop, and GCP, AWS, or Azure Cloud (Will be working in Google Cloud Platform GCP)Basic Qualification : Excellent communication skills Confident with coding and development in Python or Scala (OOP Skill) Spark Processing Big Data Hadoop Development Additional Skills : Excellent communication skills Confident with coding and development in Python or Scala (OOP Skill) Spark Processing Big Data Hadoop Development Background Check :YesDrug Screen :Yes