Sr Data Engineer

Sr Big Data Developer/Engineer

100% remote – Apache Airflow (this is a must) Python Pyspark Required: Strong verbal and written communication skills to effectively articulate messages to internal and external teams. Hands on experience with Azure data platform stack (Azure Databricks, Azure Data Factory, Azure Data Lake storage, Azure DevOps, Azure Functions) Experience with dimensional modeling and data warehousing concepts Experience with designing, building, optimizing, troubleshooting end-to-end big data pipelines using structured (relational and files) and semi-structured data Experience with building metadata driven data processing frameworks Strong experience in Spark, SQL, python, Pyspark, shell scripting. Ability to work in an agile team. Ability to take ownership of a request from initial requirements, design, development and production deployment Nice to have: Powershell, scala, java Apache Airflow Azure EventHubs Apace Kafka Streaming data CosmosDB/NoSQL Database

Job Summary: The team member’s number one job responsibility is to deliver the most remarkable patient experience, in every dimension, every time, and understands how to contribute to the health system’s vision of achieving that commitment to patients and families. At Novant Health, people are our business. We treat each other with respect and compassion. We embrace the differences in our strengths while fostering an environment of inclusion, empowerment, inspiration and courage. The team member will use Novant Health’s First Do No Harm (NHFDNH) safety behaviors/error prevention tools and high reliability strategies as appropriate to ensure a safe, remarkable patient experience. Digital Products and Services team members are responsible for securely managing information systems throughout their lifecycle, including knowing what information systems are within their scope of responsibility, understanding what sensitive data is stored, transmitted, or processed on those information systems, enforcing the security principles of least privilege and least functionality, knowing what events may constitute a cybersecurity incident, and understanding their role in security incident response activities. The Data Engineer will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.

Qualifications Education Education Level Required/Preferred Description 2 Year / Associate Degree Required Computer Science or related field 4 Year / Bachelors Degree Preferred Computer Science or related field Experience Years of Experience Required/Preferred Description 5 Required experience in Information Technology 1 Preferred experience with at least one contemporary data transformation tool Refer to the Life Support Training Policy NH-HR-3096. Licensure/Certification Type (ex NP, RN, CMA) Required/Preferred Additional Skills/Requirements (required) • Capable of handling multiple tasks with minimal supervision. • Possess strong organizational, communication, problem solving/analytical skills. • Experienced with standard query language (SQL), knowledge of and experience with Data Warehouse concepts and design. • Must be able to build processes supporting data transformations, data structures, metadata, dependency and workload management. • Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores, and big data pipeline. • Strong verbal and written communication skills. • Ability to read, understand, and develop various data structures. • Able to work independently and take ownership of tasks. Also must be able to work well within a team. • Ability to drive/travel to multiple facilities/locations as needed. • Ability to be on-call and respond to production issues in a timely manner. Additional Skills/Requirements (preferred) • Understanding of Healthcare data is desired. • Working knowledge of and experience with ETL tools such as SSIS, Informatica, or Attunity Compose. • Working knowledge of and experience with data orchestration tools such as Control-M or ADF. • Working knowledge of and experience with one or more of ADLS, ADW, SQL PaaS, Azure Data Bricks, and NoSQL databases Functions Essential Functions • Process: Extracts, tests, loads, and validates data. Analyze the data flow requirements and develops a scalable architecture for staging and loading data. Interprets the transformation rules for all target data objects and develops the software components to support the transformation. Assist with design/implementation of data staging methods. Prepare/implement data verification and testing methods. Create scripts/processes required to extract, transform, clean, and move data and metadata from a source application so they can be loaded into a data warehouse, data mart, or operational data store. Must be able to troubleshoot the existing code and develop new code, stored procedures, views, and functions. • Documentation: Create and maintain ETL, data orchestration, and system documentation. Create source to target data lineage documentation. Maintain the development, test, user acceptance, and production ETL and data orchestration environments. • Compliance: Adheres to the principles of the Novant Health Code of Ethics Compliance Plan in every encounter with customers, vendors, regulating agencies, the public and fellow employees. Adhere to corporate policies, procedures and standards. Follow SDLC standards. • Communication: Effective in giving and receiving the information required to perform the job.