Lead Databricks Data Engineer
EXL Service
Job-ID: 22394Category: Global TechnologyLocation: New York, United StatesWorkplace: HybridPosted: 2026-11-05Job Role: Data engineer-EngineeringExperience (In Years): 9-12Job Description: Lead Databricks Data EngineerJob Title: Lead Databricks Data EngineerExperience: 10-12 YearsEmployment Type: Full-TimeJob SummaryWe are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.Key Responsibilities Data Engineering & ArchitectureDesign and implement enterprise-scale data platforms using Databricks Lakehouse architecture.Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.Define enterprise data models, ingestion frameworks, and transformation strategies.Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.Drive cloud data platform modernization and migration initiatives.Technical LeadershipServe as the technical lead for Databricks and Data Engineering programs.Establish engineering best practices, coding standards, and architecture guidelines.Conduct architecture reviews, code reviews, and solution assessments.Mentor and guide Data Engineers, Technical Leads, and junior team members.Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.Cloud & Platform EngineeringDesign cloud-native data solutions on Azure, AWS, or GCP.Implement secure, scalable, and resilient data architectures.Optimize costs, performance, reliability, and availability of data platforms.Lead CI/CD and Infrastructure-as-Code implementation for data workloads.Define and implement monitoring, observability, and operational excellence practices.Governance & Data QualityEstablish data governance, lineage, cataloging, and security controls.Ensure compliance with organizational and regulatory requirements.Implement data quality frameworks and monitoring solutions.Collaborate with security and compliance teams to enforce data protection standards.Stakeholder & Delivery ManagementPartner with business leaders to understand strategic data requirements.Translate business requirements into scalable technical solutions.Drive delivery planning, risk management, and resource coordination.Present architecture decisions and technical recommendations to leadership teams.Required Technical Skills DatabricksDatabricks Lakehouse PlatformDatabricks WorkflowsDelta LakeUnity CatalogStructured StreamingDatabricks SQLDelta Live Tables (DLT)Databricks Asset BundlesBig Data TechnologiesApache SparkPySparkSpark SQLKafkaReal-Time Streaming ArchitecturesData Lake & Lakehouse ArchitecturesCloud Platforms (Strong Experience in One or More) AzureAzure DatabricksAzure Data Factory (ADF)ADLS Gen2Azure Synapse AnalyticsAzure Key VaultEvent HubAWSS3EMRGlueRedshiftLambdaIAMGCPBigQueryCloud StorageDataflowPub/SubComposerProgrammingPythonSQLPySparkScala (Preferred)Shell ScriptingDevOps & AutomationGitAzure DevOpsGitHub ActionsJenkinsTerraformDockerKubernetesData WarehousingDimensional ModelingStar & Snowflake SchemaData VaultMaster Data ManagementData Governance FrameworksRequired Experience12-15 years of overall IT experience.Minimum 6+ years of hands-on Databricks experience.7+ years of experience in Spark/PySpark-based data engineering.Experience designing enterprise-scale Data Lake and Lakehouse solutions.Strong experience leading large data transformation and modernization programs.Experience handling structured, semi-structured, and unstructured data processing.Experience supporting analytics, AI/ML, and reporting ecosystems.Proven experience managing technical teams and driving architectural decisions.Preferred QualificationsBachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.Databricks Certified Data Engineer Professional.Azure Data Engineer Associate.AWS Data Analytics Specialty.Google Professional Data Engineer Certification.TOGAF or Cloud Architecture certifications are a plus.Nice to HaveMLOps and AI Engineering exposure.Snowflake and Databricks integration experience.Microsoft Fabric exposure.Data Mesh implementation experience.Data Governance tools such as Collibra, Alation, or Informatica.Expertise in Data Quality frameworks such as Great Expectations or Deequ.Key CompetenciesSolution ArchitectureTechnical LeadershipStakeholder ManagementProgram DeliveryData Strategy & GovernanceTeam Mentoring & CoachingProblem Solving & Decision MakingCommunication & Executive Presentation SkillsPerformance OptimizationInnovation & Continuous ImprovementTechnology ClassificationTechnology Tower: Data & AIPractice Area: Data EngineeringPrimary Skill: Databricks Lakehouse EngineeringSecondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.QualificationsGraduate with 4 years BS or Professional Degree. 15 - 18 YearsResponsibilitiesManagement of DB Systems, Performance Tuning, Security, Compliance Standards,
$112k - $200k
...Lead Data Engineer New York, NY (Midtown) | Hybrid, 50% in office | $112,000 - $200,000 base + discretionary bonus I’m working with... ...business teams in a large enterprise Nice to have Databricks and Spark Azure Data Factory and other Azure data services...DatabricksWork at office- ...A global financial institution is seeking a hands-on Lead Data Engineer to design and deliver scalable enterprise data solutions while contributing... ...initiative. The engineer will work with Snowflake and Databricks, build production data pipelines, implement dimensional data...DatabricksFull time
- ...Logicbroker is the Agentic Commerce Orchestration Engine helping enterprise retailers, brands,... ...Job Summary Logicbroker is hiring a Lead Data Engineer to drive delivery across our... ...You’ll lead the build of a modern Databricks platform and shape the Gold-layer models...DatabricksFull timeContract workWork at officeRemote work
$112k - $200k
Join Mizuho as a Lead Data Engineer!We are seeking an experienced LeadData Engineer with 7+ years of experience designing, building, and delivering... ...reliability. Experience with modern data platforms such as Databricks and Spark, as well as data governance and enterprise...DatabricksFull timeWork experience placementWork at officeWorldwide$120k - $180k
...within the newly consolidated Data and Analytics (D&A)... ...infrastructure, data governance, data engineering, data modeling, data analysis... ...intelligence, data science, and AI.The Lead Data Engineer is crucial to D... ...utilizing Big Data and Azure/Databricks for Eligibility Program data...DatabricksFull timeTemporary workWork at officeLocal areaRelocation package3 days per week- Lead Data Migration - Archival - DR Engineer Seize your opportunity to make a personal impact supporting the Case Management Modernization (CMM) Program... ...Professional (CDMP) Snowflake SnowPro Advanced Databricks Certified Data Professional AWS Certified Data Analytics...DatabricksWork at office
- Title: Lead Data Engineer - Databricks Location: New York, NY (3 days/week or US Remote) Ideal Experience Range: 6-12 Years Interview process: 3 rounds, including one onsite round in NY/NJ or at any EXL office across the US. Responsibilities Data Engineering & Platform...DatabricksWork at officeRemote work3 days per week
$170k - $240k
Lead - Data Engineer We are seeking a Lead - Data Engineer to lead the hands-on design, development, and operation of key components of Astrana... ..., performance, security, and automation within an Azure Databricks environment. As a hands-on technical lead, you will...DatabricksRemote work$127k - $236.5k
Lead, Data Engineer L3Harris Enterprise Data and AI team is seeking a Data Engineer with experience in managing enterprise-level data life... ...development methodologies Experience with Python, Pandas, Databricks, JavaScript, Typescript or other scripting languages Experience...DatabricksRemote jobFlexible hours- Lead Machine Learning / Data Science Engineer CapTech is an award-winning consulting firm that collaborates with clients to achieve what's possible through... ...warehousing tools/environments such as Snowflake, Databricks, Azure SQL, Amazon RDS Comfort and proficiency in framing...DatabricksWork at officeRemote workVisa sponsorshipWork visaFlexible hours
- Job Description:DXC Technology is a leading global technology services provider helping... ...Insurance.Position SummaryThe Data Engineering Lead is responsible for the design, development... ...platforms (Azure, AWS, or Google Cloud)Databricks, Snowflake, Synapse, Redshift, BigQuery...DatabricksMinimum wageFull timeWorldwide
- ...A global financial institution is seeking a Lead Data Platform Engineer to drive adoption and enablement for a strategic enterprise cloud data... ...projects. ~ Advanced SQL experience. ~ Experience with Databricks, Snowflake, or comparable modern enterprise data...DatabricksFull timeImmediate start
- Job Title: Senior Data Engineer / Lead Data Engineer Location: Hybrid , NJ Employment Type: Full-Time NEED LOCAL CANDIDATE, NY OR NJ EXP... ...Apache Spark, and SQL . Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake . Work with...DatabricksFull timeLocal area
- ...About the Role This role owns the data infrastructure that investment/research/trading... ...Set the technical standards other engineers will build against as the team grows... ...Strong command of Python, SQL, Spark/Databricks, and modern orchestration tools Demonstrated...DatabricksFull time
- ...Data Protection Engineer In this pivotal role, you will design, develop, and execute a comprehensive... ...our data governance maturity. Leading a dedicated team of data governance... ...observability platforms (e.g., Atlan, Databricks, Anomalo, or equivalent)....Databricks
- ...Title: Data Engineer (Early Career) Duration : FTE/Permanent Salary : 90-95k Location : Onsite in NYC Junior Data Engineer... ...Shaping data models and tuning performance on Snowflake, Databricks, or a similar platform. Automating how data comes in, gets...DatabricksPermanent employmentWork experience placement
$180k - $220k
...COMPENSATION RANGE IN THE $180,000-$220,000 ANNUAL RANGE.*** Data Engineer develops and maintains Microsoft Fabric Lakehouse solutions,... ...enterprise data platforms leveraging OneLake, Data Factory, Databricks, and Azure services. Develop scalable data ingestion,...Databricks- ...client in the investment space is looking for an Analytics Engineer to help design and ship the data, analytics, and AI-enabled tools that investment... ...Exposure to a distributed data platform such as Spark, Databricks, or Snowflake. Working knowledge of at least one asset...DatabricksFull time
$140k - $155k
...innovative technology to deliver industry-leading digital solutions. Synechron’s... ...Consulting, Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of noteworthy... ...users. Python/SQL Server/Snowflake/Databricks application development and system design...DatabricksFull timeTemporary workFlexible hours$137.7k - $229.5k
Position Summary Our Deloitte AI & Engineering team works to transform technology platforms, drive innovation, and help make a significant... ..., and fuel growth through innovation.Work You’ll DoAs a Lead Data Engineer II on the team, you will be responsible for:Support...Local area- Cloud Engineer Location: Remote/US Duration: 6+ Months Job Description: Primary Job Function: The cloud engineering function provides in-... ...incorporating industry best practices. May assist senior engineer or team lead in larger scale more complex cloud designs. Requirements:...Remote work
- ...Senior GenAI Data Scientist New York This is an opportunity to work on enterprise... ...leveraging the latest capabilities within the Databricks ecosystem, including Genie AI and MLflow.... ...interfaces over enterprise data. Lead customers through the full solution lifecycle...DatabricksFull time
- Databricks Data Engineer Data and Analytics | Data Engineering Atlanta, Georgia, United States Austin, Texas, United States Boston, Massachusetts, United States Charlotte, North Carolina, United States Chicago, Illinois, United States Cincinnati, Ohio, United States Cleveland...Databricks
$100k - $180k
...New York, United States Permanent Data Catalog EngineerNew York, NY - Hybrid (3... ...) My client is seeking a Data Catalog Engineer to join a growing Data Governance team focused... ...catalog platforms like Atlan, Collibra, Databricks Unity Catalog, Alation, Informatica or...DatabricksPermanent employmentWork at office3 days per week- ...designing and building scalable and robust data pipelines to enable data-driven... ...teams.Should have experience working on Databricks ,Airflow , Redshift , Snaplogic .Experience... ...very large-scale data warehouse and data engineering projects.Experience developing low latency...Databricks
- ...healthcare company based in the Detroit is looking to hire a Data Engineer. They're looking for someone to work closely with stakeholders... ...engineering solutions. Technical experience with Foundry, Databricks, or Snowflake is preferred, but not required. A background in...DatabricksFull time
- Job Description: Senior ML Data Engineer Feature Engineering ETL Qualifications 7 years in data engineering and at least 4 years focusing... ...preparation for ML Proven experience managing pipelines on Databricks using Apache Spark with a strong understanding of the...DatabricksRemote job
- ...Contract! We are seeking a Senior Software Engineer with a strong background in machine... ...experience in machine learning engineering, data science, statistical modeling, or a... ...data platforms such as AWS, Snowflake, Databricks, or Apache Spark. Bachelor's degree in Computer...DatabricksFull timeContract work
$165k - $210k
Flow Traders is looking for a Core Data Engineer to own the end-to-end lifecycle of our most fundamental data products, including our Security... ...Experience with a modern cloud data stack, such as Spark or Databricks, dbt, a cloud warehouse, and orchestrationExperience with...DatabricksLocal area- Data Engineer Key skills are very strong SQL capability, data pipelines management, Python, and Airflow. Location is New York, mandatory... ...how to develop DAGS. Expert in data warehouses like BigQuery, Databricks Deltalakehouse and how to programmatically ingest, cleanse,...Databricks3 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Databricks Data Engineer. Be the first to apply!



