Azure Data Engineer

Azure Data EngineerHybrid onsite in Hopkins MNContract through 5/30/2025Project Summary:This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included.Key Responsibilities:Identify Requirements and Develop SolutionsIdentify requirements for RMS replacement and develop solutions in Azure.Re-engineer solutions to retrofit the rules with AC360 in Azure.Data Pipeline ManagementRegister data pipelines in Azure, onboard datasets, and enable data processing.Build Airflow DAGs and ensure scheduling is integrated.DevOps and Production ManagementApply a DevOps mindset, take ownership of production success, and optimize operations via automation, active alerting, and self-healing.Lead the resolution of production issues to ensure high availability and performance.Code Development and ReviewDevelop high-quality code and define best engineering practices.Perform peer code reviews to ensure successful deliverables with engineering excellence.DocumentationDocument engineering artifacts such as technical design documents, flowcharts, system context diagrams, and code comments.CollaborationCollaborate cross-functionally with product owners, data scientists, business users, project managers, and other engineers to achieve elegant solutions.Work with other engineers on the team to elevate technology and consistently apply best practices.End-to-End Development and TestingExecute true end-to-end development and testing of software products in an agile lifecycle.Partner with key technical and business stakeholders to ensure the quality of the final product through manual and automated tests.Support the new software as it is moved to production.Required Skills/Technologies:Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine.Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience.Airflow, Elastic, Azure: 3 years or higher.Big Data Management: 10 years or higher.Microservices-Based Applications: 5+ years.Azure, Kubernetes, PostGres: Experience required.Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years.CI/CD Processes (Maven, Git, Jenkins): 4+ years.Cloud Technologies: 4+ years.Server-Side Languages (Python, Java, GoLang): 3+ years.Communication Skills: Excellent verbal and written communication skills, 10 years or higher.Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.Preferred Skills/Attributes:Experience with Spark, Scala, Python, Azure stack.Interview Process:Rounds: 1Format: VideoTechnical Level: Real-time coding session