Senior Data Engineer Role

Senior Data Engineer (Cloud)

This role is for a senior data engineer that will join a team responsible for managing a growing cloud-based data ecosystem consisting of a metadata driven data lake and databases that support real time analytics, extracts, and reporting. The right candidate will have a solid background in data engineering and should have a few years of experience on a major cloud platform such as Azure.

Is this person a sole contributor or part of a team?: Team

If so, please describe the team? (Name of team, size of team, etc.): Team Name: Alefgard, Size: Approximately 10+

What are the top 5-10 responsibilities for this position? (Please be detailed as to what the candidate is expected to do or complete on a daily basis):

• Building and maintaining a data processing framework on Azure using Databricks
• Writing code in Apache Spark/Scala
• Working with existing Databricks Delta Lake tables to optimize for CDC performance using techniques
• Working with existing Databricks Notebooks to optimize or address performance concerns
• Create new Databricks Notebooks or stand-alone Apache Spark/Scala code as needed
• Working with existing on premise data management tools as required and willing to learn Ab Initio.

What software tools/skills are needed to perform these daily responsibilities? Databricks Apache Spark Scala programming Azure

What skills/attributes are a must have?

• Data Warehousing / Big Data Best Practices
• Understanding of how best to partition and organize data depending on the technology and use case
• 2 years Experience in regularly dealing with data in hundreds of Terabytes up to 1-2 Petabytes
• 2 years Data engineering experience
• 2 years Cloud platform experience
• 2 years Version Control (Git or equivalent)

What skills/attributes are nice to have?

• Data Integration Tools (Spark/Databricks or equivalent): 2 years
• Scripting (Linux/Unix Shell scripting or equivalent): 2 years
• Ab Initio experience
• Netezza Experience

Where is the work to be performed? (Please list preferred Health Care Company facility, if other please specify i.e. remote work, rural, etc.) Remote work or 1600 McConnor Parkway, Schaumburg, IL

What are the work hours? (ex. 9am-5pm, day/night shifts, rotating shifts, etc)

This is a software engineering position, after hour deployments will be needed on a rotational basis.

What does the interview process look like? How many rounds? Round 1 (Video Webex call with a few of our data engineers)

Discussion on resume with technical questions. Round 2 (Video Webex call with hiring manager(s))