Data Engineer - Databricks

Data Engineer - Azure Databricks

Contract-to-Hire

Pittsburgh, PA - Onsite

Job ID J0726-0569

Visa : USC, GC, EAD (No Sponsorship)

Position Overview

• Seeking an experienced Data Engineer to join a high-performing data engineering team responsible for building modern, cloud-native data platforms on Microsoft Azure.

• This role will focus on designing and implementing scalable, secure, and high-performance data solutions using Azure Databricks, Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog. The position requires strong expertise in enterprise data governance, metadata management, security, and access control within the Investment Banking, Securities, or Capital Markets domain.

• The ideal candidate is passionate about cloud data engineering, enjoys solving complex data challenges, and thrives in a collaborative Agile environment.

Required Competencies

Competency Importance Proficiency Level

• Apache Spark

• Required

• Expert

• Microsoft Azure

• Required

• Advanced

• Investment Banking and Securities

• Required

• Advanced

• Structured Query Language - SQL

• Required

• Expert

• Unity Catalog

• Required

• Advanced

• Databricks

• Required

• Expert

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.

• Build and optimize ETL and ELT workflows using PySpark, SQL, Apache Spark, and Delta Lake.

• Implement and manage Unity Catalog for enterprise data governance, metadata management, security, and fine-grained access control.

• Develop cloud-native data engineering solutions on the Microsoft Azure platform.

• Collaborate with architects, business analysts, and cross-functional teams to translate business requirements into technical solutions.

• Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.

• Ensure data quality, reliability, governance, and regulatory compliance through validation, monitoring, and engineering best practices.

• Support production deployments, troubleshooting, root-cause analysis, and performance tuning.

• Participate in code reviews, technical documentation, and continuous improvement initiatives.

• Apply cloud data engineering, security, CI/CD, and DevOps best practices within Azure environments.

Required Qualifications

• 8+ years of overall data engineering or related technology experience preferred.

• Minimum 5 years of hands-on Azure Databricks experience.

• Mandatory hands-on experience implementing and managing Unity Catalog.

• Expert-level experience with Apache Spark, PySpark, and SQL.

• Strong experience developing enterprise-scale data engineering solutions on Microsoft Azure.

• Strong experience with Delta Lake, ETL/ELT development, and data pipeline implementation.

• Experience with Azure Data Factory and Azure Data Lake Storage Gen2.

• Strong knowledge of the Microsoft Azure data ecosystem.

• Strong understanding of data governance, metadata management, data security, and access control.

• Mandatory experience within Investment Banking, Securities, or Capital Markets.

• Experience using Git and implementing CI/CD practices.

• Experience working in Agile/Scrum delivery environments.

• Excellent analytical, communication, and problem-solving skills.

• Must be willing to work onsite in Pittsburgh, PA.

• Must be willing and eligible to convert to full-time employment after six months.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification.

• Microsoft Certified Azure Data Engineer Associate, DP-203.

• Experience with Infrastructure as Code using Terraform or Bicep.

• Experience implementing DevOps practices for Azure Databricks environments.

• Familiarity with Microsoft Purview or Azure Purview.

#M1

#DI-CB2

#L1 - KB1

Ref: #404-IT Pittsburgh

System One, and its subsidiaries including Joulé, ALTA IT Services, CM Access, TPGS, and MOUNTAIN, LTD., are leaders in delivering workforce solutions and integrated services across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible full-time employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.