Data Engineer

Job Description

Job Description

We are looking for a talented and driven Data Engineer with strong expertise in Azure Data Factory and other Microsoft technologies to join our dynamic team. In this role, you will be responsible for designing, developing, and maintaining data pipelines, integration solutions, and data-centric applications. Your in-depth knowledge of Microsoft technologies and proficiency in Azure Data Factory will be key to optimizing data processing, storage, and analysis.

Responsibilities:

1. Design and Develop Data Pipelines: Partner with cross-functional teams to gather data requirements and create robust, scalable, and reliable data pipelines using Azure Data Factory and other Microsoft tools, ensuring high data quality.

2. Data Integration: Consolidate data from diverse sources—including databases, APIs, and third-party platforms—into centralized data lakes or warehouses utilizing Azure Data Factory, Azure SQL Database, Azure Synapse Analytics, and related technologies.

3. Data Transformation: Execute data transformation and cleansing processes to prepare data for analysis, reporting, and business intelligence applications.

4. Data Modeling: Develop and implement data models and schemas that align with business needs and industry standards, promoting efficient data access for end-users and applications.

5. Optimization and Performance Tuning: Regularly monitor and fine-tune data pipelines and processing tasks to enhance performance, reduce latency, and ensure accurate, timely data delivery.

6. Data Security and Compliance: Enforce data security protocols, including encryption and access controls, to protect sensitive information and maintain compliance with data privacy regulations.

7. Troubleshooting and Issue Resolution: Diagnose and resolve data engineering challenges, debugging pipelines and processes to uphold system performance and data accuracy.

8. Collaboration and Documentation: Collaborate with data scientists, analysts, and other stakeholders to address data needs, while maintaining thorough documentation of pipelines, models, and integration workflows.

9. Cloud Infrastructure Management: Oversee Azure cloud resources, monitor usage, and apply best practices for cost-effective infrastructure management.

10. Continuous Learning and Improvement: Keep abreast of the latest Azure Data Factory developments and Microsoft technologies, integrating best practices and innovative solutions to continually enhance data engineering processes.

Requirements:

1. A Bachelor’s degree in Computer Science, Information Technology, or a related discipline.
2. Demonstrated experience as a Data Engineer with a strong emphasis on Microsoft technologies, especially Azure Data Factory.
3. Advanced skills in SQL, data modeling, and working with database platforms such as Azure SQL Database, Azure Synapse Analytics, and Cosmos DB.
4. Hands-on experience designing and developing data pipelines using Azure Data Factory or comparable ETL/ELT tools.
5. Solid understanding of cloud-based data storage and processing solutions within the Microsoft Azure ecosystem.
6. Knowledge of data warehousing principles and data integration methodologies.
7. Proficiency in programming languages like Python, C#, or Java for data manipulation and automation purposes.
8. Familiarity with data security protocols and compliance standards related to data management.
9. Strong analytical and problem-solving abilities with a keen attention to detail.
10. Excellent communication and collaboration skills, capable of working effectively within cross-functional teams.
11. Relevant Microsoft certifications (e.g., Azure Data Engineer Associate) are preferred.
12. Experience with Microsoft Fabric is a plus but not required.