Data Engineer II

Building Location:Peerless BuildingDepartment:1007670 ANALYTICS AND ARCHITECTURE - EH SSJob Description:The Data Engineer II is responsible for designing and optimizing scalable and complex data pipelines and workflows to handle large datasets. They are expected to implement advanced data processing and scheduling techniques using frameworks like Apache spark, ADF and airflow. Ensures data quality, governance, and security through automated testing and monitoring solutions. Collaborate with cross-functional teams to support data-driven decision-making and ensure the scalability and efficiency of data systems.Education Qualifications:Key Responsibilities:Designs and optimizes scalable and robust data pipelines and workflows to handle large datasets to meet functional and non-functional business requirementImplements advanced data processing and scheduling techniques using frameworks like Apache Spark, ADF and AirflowWrite and optimize advanced SQL queries for reporting and applicationsEnsures data quality, governance, and security through automated testing and monitoring solutionsWork closely with data scientists, analysts, and other stakeholders to understand data requirementsProvide technical guidance and mentorship to junior data engineersDevelop infrastructure to support real-time and batch processing workflowsDeliver actionable data products and insights for business-critical toolsEducation Requirements:Bachelor’s degree in computer science, data analytics, statistics, or related fieldRequired Qualifications:2-5 years of experience in data engineering or a related roleProficiency in SQL and experience with database management systemsStrong programming skills in languages such as Python, Java, or ScalaExperience with pipelines using orchestration tools such as Apache Airflow or AWS Glue for seamless data ingestion and deliveryExperience with modern data management technologies (e.g., data lake, lakehouse, data catalogs)Experience supporting both batch and streaming data, enabling near real-time analytics and reportingPreferred Qualifications:Familiarity with applying AI and machine learning techniques to data use casesExperience creating robust data transformation workflows, including metadata management, dependency tracking, and workload orchestrationExperience with DBTExperience with Microsoft FabricExcellent problem-solving skills and attention to detailStrong communication and collaboration skillsLicensure/CertificationQualifications:FTE:1Possible Remote/Hybrid Option:RemoteShift Rotation:Day Rotation (United States of America)Shift Start Time:8:00 AMShift End Time:4:30 PMWeekends:Holidays:NoCall Obligation:NoUnion:Union Posting Deadline:Compensation Range:$89,523.20 - $134,284.80Employee Benefits at Essentia Health: At Essentia Health, we’re committed to supporting your well-being, growth, and work-life balance. Our comprehensive benefits include medical, dental, vision, life, and disability insurance, along with supplemental options to fit your needs. We offer a 401(k) plan with employer contributions to help you plan for the future, and we invest in your professional development through training, tuition reimbursement, and educational programs. To help you thrive both at work and at home, we provide flexible scheduling, generous time off, and wellness resources focused on your physical, mental, and emotional health. Please note that benefit eligibility may vary. For full details, refer to your benefit summary or contact our HR Service Center at View phone number on click.appcast.io.SummaryLocation: Duluth, MNType: Full time