Lead Data Engineer

Job Description Lead/Senior Databricks Data Engineer Position: Lead Databricks Data Engineer / Databricks Architect Experience: 10+ Years Location: Remote Job Summary We are looking for an experienced Lead Databricks Data Engineer / Databricks Architect with 10+ years of overall experience in Data Engineering and strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud-based data platforms . The candidate will be responsible for designing and implementing scalable Lakehouse architectures , enterprise data pipelines, data integration solutions, governance frameworks, and high-performance analytics platforms using Databricks.

• Design and develop scalable data engineering solutions using Databricks and Lakehouse architecture .
• Build robust ETL/ELT pipelines using PySpark, Spark SQL, Python, and SQL .
• Design and implement Bronze, Silver, and Gold/Medallion architecture .
• Develop and optimize Delta Lake tables, including MERGE, schema evolution, Change Data Feed, and incremental processing.
• Build batch and real-time/streaming pipelines using Structured Streaming, Auto Loader, and Lakeflow .
• Develop and manage Databricks Jobs/Workflows for pipeline orchestration, scheduling, dependencies, retries, and monitoring.
• Implement enterprise data governance using Unity Catalog , including access control, data lineage, auditing, catalogs, schemas, and external locations. Unity Catalog provides centralized governance, access control, lineage, and auditing across Databricks data and AI assets.
• Perform Spark and Databricks performance tuning , including cluster configuration, partitioning, caching, query optimization, Photon, and workload optimization.
• Design data models supporting Data Warehousing, BI, Analytics, and AI/ML workloads .
• Integrate Databricks with cloud platforms such as AWS, Azure, or Google Cloud Platform .
• Work with cloud services such as AWS S3, Azure ADLS Gen2, Azure Data Factory, AWS Glue, Synapse, Event Hubs/Kafka/Kinesis , as applicable.
• Implement CI/CD pipelines using Git, Azure DevOps/GitHub/Jenkins and Databricks deployment capabilities.
• Work with Terraform/IaC for infrastructure provisioning and automation.
• Troubleshoot production pipeline failures, performance issues, data-quality problems, and Spark/cluster issues.
• Establish data quality, monitoring, logging, and observability practices.
• Provide technical leadership, code reviews, architecture guidance, and mentorship to junior/mid-level engineers.
• Collaborate with Data Architects, Data Scientists, Business Analysts, DevOps teams, and application teams.

• Databricks
• Databricks Lakehouse Platform
• Delta Lake
• Unity Catalog
• Databricks Workflows/Jobs
• Lakeflow / Delta Live Tables
• Auto Loader
• Databricks SQL
• Databricks notebooks
• Databricks Asset Bundles
• Photon
• Cluster/workload optimization
• Big Data
• Apache Spark
• PySpark
• Spark SQL
• Structured Streaming
• Kafka
• Batch and real-time data processing
• Python
• SQL
• Scala
• AWS: S3, Glue, EMR, Lambda, Redshift, IAM, Kinesis
• Azure: ADLS Gen2, ADF, Synapse, Azure DevOps, Event Hubs, Key Vault
• Google Cloud Platform: GCS, BigQuery, Dataflow, Pub/Sub
• ETL/ELT, Data Warehousing, Dimensional Modeling, Data Lake/Lakehouse, Medallion Architecture, CDC, Data Quality, Data Governance, Metadata and Data Lineage
• Git, Azure DevOps, GitHub, Jenkins, Terraform, CI/CD automation, Infrastructure as Code

• MLflow
• Databricks Machine Learning
• Feature Store
• Mosaic AI / GenAI
• dbt
• Apache Airflow
• Power BI / Tableau
• Delta Sharing
• Lakehouse Federation
• Liquid Clustering
• Data security and PII masking

MLflow is particularly useful if the role touches ML/AI, as Databricks supports model tracking, lifecycle management, and deployment workflows alongside governed data.

Qualifications Bachelor's degree in Computer Science, Engineering, Information Technology, or related field. 10+ years of experience in Data Engineering / Big Data / Analytics. 4+ years of hands-on Databricks experience preferred. Strong experience designing enterprise-scale data platforms. Demonstrated experience leading technical projects and mentoring engineers. Strong communication and stakeholder-management skills.

For applications and inquiries, contact:View email address on us.fitly.work