Data Engineer - Databricks

Job Title: Data Engineer – Databricks

Duration : Contract to Hire
Location : Pittsburgh, PA

Work Mode : 5 Days Onsite



Job Description:

Join a high performing data engineering team responsible for building modern, cloud native data platforms on Microsoft Azure . This role offers the opportunity to design and implement scalable, secure, and high performance data solutions using Azure Databricks, with a strong focus on Unity Catalog, data governance, and enterprise analytics within the Investment/Capital Markets domain.


The ideal candidate is passionate about cloud data engineering, enjoys solving complex data challenges, and thrives in a collaborative Agile environment . This position provides the opportunity to work with the latest Azure data technologies while delivering impactful business solutions.



Duties and Responsibilities
. Design, develop, and maintain scalable data pipelines using Azure Databricks.

. Build and optimize ETL/ELT workflows leveraging PySpark, SQL, and Delta Lake.

. Implement and manage Unity Catalog to support enterprise data governance, security, metadata management, and fine-grained access control.

. Develop cloud native data engineering solutions on the Microsoft Azure platform.

. Collaborate with architects, business analysts, and cross functional teams to gather and translate business requirements into technical solutions.

. Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.

. Ensure data quality, reliability, and compliance through validation, monitoring, and best practices.

. Support production deployments, troubleshooting, and performance tuning.

. Participate in code reviews, technical documentation, and continuous improvement initiatives.

. Apply best practices for cloud data engineering, security, and DevOps within Azure environments .



Required qualifications to be successful in this role
. Minimum 5 years of hands-on experience with Azure Databricks.

. Hands on experience with Unity Catalog is mandatory.

. Strong experience building enterprise scale data engineering solutions on Microsoft Azure.

. Expert level experience with Apache Spark (PySpark) and SQL.

. Strong experience with Delta Lake, ETL/ELT development, and data pipeline implementation.

. Experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and related Azure ecosystem services.

. Strong understanding of data governance, metadata management, security, and access control.

. Mandatory experience in the Investment/Capital Markets domain.

. Experience with Git and CI/CD practices.

. Excellent analytical, communication, and problem solving skills.

. Experience working in Agile/Scrum delivery environments.



Preferred Qualifications

. Databricks Certified Data Engineer (Associate or Professional) .

. Microsoft Azure Data Engineer Associate (DP 203) certification .

. Experience with Infrastructure as Code (Terraform/Bicep) .

. Experience implementing DevOps practices for Azure Databricks environments .

. Familiarity with Azure Purview/Microsoft Purview is a plus.

Ref: #404-IT Pittsburgh