Senior/Lead Data Engineer - Azure & Databricks

Senior/Lead Data Engineer – Azure & Databricks

Location: Alpharetta, GA Type: Full-Time

Visa: Visa-dependent candidates accepted

We are seeking an experienced Senior/Lead Data Engineer with strong hands-on expertise in Azure, Databricks, Spark, Delta Lake, and modern Lakehouse architecture. The ideal candidate has the ability to architect, design, and build complex data pipelines, optimize large-scale workloads, and lead data engineering initiatives in an enterprise environment.

Key Responsibilities

• Architect, design, and implement scalable data platforms and pipelines on Azure + Databricks.
• Build and optimize batch & real-time data ingestion, processing, and transformation workflows.
• Develop ETL/ELT pipelines using Spark (PySpark), ADLS, Delta Lake, and Databricks Jobs/Workflows.
• Design conceptual, logical, and physical data models for analytics and operational workloads.
• Manage Delta Lake features: ACID transactions, schema evolution, Z-Order, OPTIMIZE, Time Travel, DLT.
• Build streaming pipelines using Autoloader, Spark Streaming, and checkpointing.
• Implement CI/CD and deployment automation using Git, ADO Pipelines, Databricks Asset Bundles.
• Ensure strong data governance using Unity Catalog (permissions, lineage, security).
• Collaborate with architects, analysts, and data scientists to translate business needs into solutions.
• Troubleshoot platform issues, optimize performance, and ensure reliability of large-scale workflows.
• Mentor junior engineers and contribute to engineering best practices and reusable frameworks.

Required Skills & Experience

Databricks & Lakehouse

• Strong experience with Azure Databricks runtimes, Spark engine, and newly launched features.
• Hands-on with Delta Lake, ACID properties, schema evolution, Time Travel, OPTIMIZE, Z-Order.
• Strong understanding of Lakehouse architecture and scalable data platform design.

Spark & Programming

• Expert-level PySpark experience handling TB-scale data workloads.
• Strong experience in performance tuning: cluster optimization, file size tuning, parallelism.

Data Engineering

• Strong ETL/ELT pipeline development, orchestration, and scheduling.
• Hands-on with Databricks Workflows, Jobs, Tasks, and CI/CD deployment.
• Strong SQL—complex queries, optimization, table design, schema evolution.

Streaming

• Experience with Autoloader, Spark Streaming, EventHub/Kafka (preferred).

Security & Governance

• Unity Catalog (permissions, lineage, governance, encryption/decryption).

Cloud & Tools

• Azure experience: ADF, Functions, ADLS, Event Hub.
• CI/CD using Azure DevOps & Git.
• IaC: Terraform/ARM (preferred).
• Experience designing full end-to-end architecture.

Soft Skills

• Strong communication, architecture justification, collaboration skills.
• Ability to lead technical discussions and problem-solving.

Nice-to-Have

• Azure Purview (governance)
• Experience on ML or BI workloads on Databricks
• Terraform/ARM
• Azure EventHub/Kafka streaming

Please send me your updated resume ASAP with below details:

Full Legal Name:

Current Location:

Willing to Relocate:-

E-mail:

Cell:

Rate/Salary:

Availability:

Are you still in Project:

Work Status: