Azure Databricks Data Engineer

ob Title: Azure Databricks Data Engineer

Location: 3275 Bennett Creek Avenue, Frederick, MD 21704 (100% Onsite)

Duration: 6 Months Contract


Job Summary


We are seeking an experienced Azure Databricks Data Engineer to design, develop, and maintain scalable, high-performance data engineering solutions. The ideal candidate will have strong expertise in Azure Databricks, PySpark, Python, SQL, Delta Lake, and Lakehouse Architecture , with experience building enterprise-grade batch and real-time data pipelines.


Key Responsibilities

• Design, develop, and maintain scalable, high-performance data engineering solutions on Azure Databricks.
• Build and support batch and real-time data pipelines using Apache Spark and Structured Streaming.
• Design and implement Lakehouse and Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Develop reusable ETL/ELT frameworks, standardized data processing components, and enterprise data ingestion patterns.
• Ingest and process large-scale data from enterprise sources including databases, APIs, message queues, and file-based systems.
• Develop complex data transformation logic using PySpark, Python, and SQL.
• Create trusted, governed, and analytics-ready datasets for reporting and downstream applications.
• Implement data quality, validation, reconciliation, audit, and governance controls across data pipelines.
• Manage and optimize Databricks Workflows, Delta Live Tables (DLT), Unity Catalog, and Databricks SQL.
• Collaborate with business, analytics, architecture, and delivery teams in an Agile environment.

Required Technical Skills

Databricks & Data Engineering

• Strong hands-on experience with Azure Databricks, Lakehouse Architecture, and Delta Lake.
• Expertise in PySpark, Python, and advanced SQL.
• Experience with Apache Spark, Structured Streaming, Delta Live Tables (DLT), Databricks Workflows, and Unity Catalog.
• Experience designing and implementing Medallion Architecture and modern Data Lake/Data Warehouse solutions.
• Strong background in data ingestion, ETL/ELT, data transformation, optimization, and deployment practices.
• Experience with event-driven and streaming technologies such as Kafka or Azure Event Hub.

Cloud Platforms

• Microsoft Azure (Preferred)
• AWS or GCP experience is a plus.

Application Integration

• Working knowledge of React and/or Angular for front-end integration and data-centric application development.

Preferred Qualifications

• Experience in Finance, Insurance, Regulatory Reporting, or US GAAP reporting environments.
• Understanding of financial data models, reporting processes, and governance requirements.
• Experience delivering enterprise-scale data platforms and analytics solutions.

Required Experience

• 8+ years of Data Engineering experience.
• 4+ years of hands-on Azure Databricks experience.
• Strong experience with large-scale data processing and cloud-native data solutions.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.