Azure Data Engineer

Azure Data Engineer

Hybrid onsite in Hopkins MN


Contract through 5/30/2025

Project Summary:

This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included.

Key Responsibilities:

1. Identify Requirements and Develop Solutions
• Identify requirements for RMS replacement and develop solutions in Azure.
• Re-engineer solutions to retrofit the rules with AC360 in Azure.

2. Data Pipeline Management
• Register data pipelines in Azure, onboard datasets, and enable data processing.
• Build Airflow DAGs and ensure scheduling is integrated.

3. DevOps and Production Management
• Apply a DevOps mindset, take ownership of production success, and optimize operations via automation, active alerting, and self-healing.
• Lead the resolution of production issues to ensure high availability and performance.

4. Code Development and Review
• Develop high-quality code and define best engineering practices.
• Perform peer code reviews to ensure successful deliverables with engineering excellence.

5. Documentation
• Document engineering artifacts such as technical design documents, flowcharts, system context diagrams, and code comments.

6. Collaboration
• Collaborate cross-functionally with product owners, data scientists, business users, project managers, and other engineers to achieve elegant solutions.
• Work with other engineers on the team to elevate technology and consistently apply best practices.

7. End-to-End Development and Testing
• Execute true end-to-end development and testing of software products in an agile lifecycle.
• Partner with key technical and business stakeholders to ensure the quality of the final product through manual and automated tests.
• Support the new software as it is moved to production.

Required Skills/Technologies:
• Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine.
• Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience.
• Airflow, Elastic, Azure: 3 years or higher.
• Big Data Management: 10 years or higher.
• Microservices-Based Applications: 5+ years.
• Azure, Kubernetes, PostGres: Experience required.
• Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years.
• CI/CD Processes (Maven, Git, Jenkins): 4+ years.
• Cloud Technologies: 4+ years.
• Server-Side Languages (Python, Java, GoLang): 3+ years.
• Communication Skills: Excellent verbal and written communication skills, 10 years or higher.
• Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.

Preferred Skills/Attributes:
• Experience with Spark, Scala, Python, Azure stack.

Interview Process:
• Rounds: 1
• Format: Video
• Technical Level: Real-time coding session

Required Skills : • Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine. • Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience. • Airflow, Elastic, Azure: 3 years or higher. • Big Data Management: 10 years or higher. • Microservices-Based Applications: 5+ years. • Azure, Kubernetes, PostGres: Experience required. • Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years. • CI/CD Processes (Maven, Git, Jenkins): 4+ years. • Cloud Technologies: 4+ years. • Server-Side Languages (Python, Java, GoLang): 3+ years. • Communication Skills: Excellent verbal and written communication skills, 10 years or higher. • Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.

Basic Qualification :

Additional Skills :


Background Check : Yes


Drug Screen : Yes


Notes :
Selling points for candidate :
Project Verification Info :MSA: Restricted Client Letter: Will Provide"
Candidate must be your W2 Employee :Yes
Exclusive to Apex :No
Face to face interview required :No
Candidate must be local :Yes
Candidate must be authorized to work without sponsorship ::No
Interview times set :Yes
Type of project :
Master Job Title :
Branch Code :