Azure Data Engineer

Azure Data Engineer

Hybrid onsite in Hopkins MN

Contract through 5/30/2025

Project Summary:

This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included.

Key Responsibilities:

• Identify Requirements and Develop Solutions
• Identify requirements for RMS replacement and develop solutions in Azure.
• Re-engineer solutions to retrofit the rules with AC360 in Azure.
• Data Pipeline Management
• Register data pipelines in Azure, onboard datasets, and enable data processing.
• Build Airflow DAGs and ensure scheduling is integrated.
• DevOps and Production Management
• Apply a DevOps mindset, take ownership of production success, and optimize operations via automation, active alerting, and self-healing.
• Lead the resolution of production issues to ensure high availability and performance.
• Code Development and Review
• Develop high-quality code and define best engineering practices.
• Perform peer code reviews to ensure successful deliverables with engineering excellence.
• Documentation
• Document engineering artifacts such as technical design documents, flowcharts, system context diagrams, and code comments.
• Collaboration
• Collaborate cross-functionally with product owners, data scientists, business users, project managers, and other engineers to achieve elegant solutions.
• Work with other engineers on the team to elevate technology and consistently apply best practices.
• End-to-End Development and Testing
• Execute true end-to-end development and testing of software products in an agile lifecycle.
• Partner with key technical and business stakeholders to ensure the quality of the final product through manual and automated tests.
• Support the new software as it is moved to production.

Required Skills/Technologies:

• Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine.
• Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience.
• Airflow, Elastic, Azure: 3 years or higher.
• Big Data Management: 10 years or higher.
• Microservices-Based Applications: 5+ years.
• Azure, Kubernetes, PostGres: Experience required.
• Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years.
• CI/CD Processes (Maven, Git, Jenkins): 4+ years.
• Cloud Technologies: 4+ years.
• Server-Side Languages (Python, Java, GoLang): 3+ years.
• Communication Skills: Excellent verbal and written communication skills, 10 years or higher.
• Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.

Preferred Skills/Attributes:

• Experience with Spark, Scala, Python, Azure stack.

Interview Process:

• Rounds: 1
• Format: Video
• Technical Level: Real-time coding session