Data Engineer

Job Description

We are seeking a skilled Data Engineer to join our dynamic team at Microsoft and contribute to the data pipeline and infrastructure supporting our Azure-hosted AI Experimentation Platform. This platform is a critical system composed of Azure Web and API App Services, Azure SQL, .NET 9, and Blazor Server, with security managed through Entra ID. It heavily utilizes AI Foundry and AI Search for innovative AI capabilities. You will play a critical role in ensuring the availability, reliability, and scalability of data pipelines and reporting environments to support AI researchers and developers while maintaining high performance and security standards. More specifically, as a Data Engineer, your primary focus will be on ingesting data from multiple sources, processing it using Azure Synapse Analytics (with extensive use of Python notebooks), staging it in Azure Data Lake Storage (ADLS), and indexing it in AI Search for downstream AI experimentation and analysis. You will also be managing our Microsoft Fabric environment(s) to enable various integration, self-serve, and reporting needs.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

• Bachelor’s degree in Computer Science, Data Engineering, Data Science, or a related field (or equivalent experience).

• 3+ years of professional experience in data engineering, with a focus on cloud based data platforms and large-scale data processing.

• Strong proficiency in Python for data processing, including experience with libraries such as Pandas, NumPy, and PySpark, and developing Python notebooks in Azure Synapse Analytics.

• Hands-on experience with Azure data services, including Azure Synapse Analytics, Azure Data Lake Storage (ADLS), and Azure Data Factory for building and managing data pipelines.

• Hands-on experiments with Microsoft Fabric and PowerBI.

• Familiarity with Azure AI Search for indexing and optimizing data to support search and AI use cases.

• Experience ingesting, transforming, and managing data from multiple heterogeneous sources (e.g., databases, APIs, flat files) into a unified data platform. • Knowledge of data modeling, schema design, and data warehousing concepts to support efficient storage and retrieval. Previous experience working at Microsoft