Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Lead Databricks Data Engineer

EXL Service

Job-ID: 22394Category: Global TechnologyLocation: New York, United StatesWorkplace: HybridPosted: 2026-11-05Job Role: Data engineer-EngineeringExperience (In Years): 9-12Job Description: Lead Databricks Data EngineerJob Title: Lead Databricks Data EngineerExperience: 10-12 YearsEmployment Type: Full-TimeJob SummaryWe are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.Key Responsibilities Data Engineering & ArchitectureDesign and implement enterprise-scale data platforms using Databricks Lakehouse architecture.Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.Define enterprise data models, ingestion frameworks, and transformation strategies.Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.Drive cloud data platform modernization and migration initiatives.Technical LeadershipServe as the technical lead for Databricks and Data Engineering programs.Establish engineering best practices, coding standards, and architecture guidelines.Conduct architecture reviews, code reviews, and solution assessments.Mentor and guide Data Engineers, Technical Leads, and junior team members.Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.Cloud & Platform EngineeringDesign cloud-native data solutions on Azure, AWS, or GCP.Implement secure, scalable, and resilient data architectures.Optimize costs, performance, reliability, and availability of data platforms.Lead CI/CD and Infrastructure-as-Code implementation for data workloads.Define and implement monitoring, observability, and operational excellence practices.Governance & Data QualityEstablish data governance, lineage, cataloging, and security controls.Ensure compliance with organizational and regulatory requirements.Implement data quality frameworks and monitoring solutions.Collaborate with security and compliance teams to enforce data protection standards.Stakeholder & Delivery ManagementPartner with business leaders to understand strategic data requirements.Translate business requirements into scalable technical solutions.Drive delivery planning, risk management, and resource coordination.Present architecture decisions and technical recommendations to leadership teams.Required Technical Skills DatabricksDatabricks Lakehouse PlatformDatabricks WorkflowsDelta LakeUnity CatalogStructured StreamingDatabricks SQLDelta Live Tables (DLT)Databricks Asset BundlesBig Data TechnologiesApache SparkPySparkSpark SQLKafkaReal-Time Streaming ArchitecturesData Lake & Lakehouse ArchitecturesCloud Platforms (Strong Experience in One or More) AzureAzure DatabricksAzure Data Factory (ADF)ADLS Gen2Azure Synapse AnalyticsAzure Key VaultEvent HubAWSS3EMRGlueRedshiftLambdaIAMGCPBigQueryCloud StorageDataflowPub/SubComposerProgrammingPythonSQLPySparkScala (Preferred)Shell ScriptingDevOps & AutomationGitAzure DevOpsGitHub ActionsJenkinsTerraformDockerKubernetesData WarehousingDimensional ModelingStar & Snowflake SchemaData VaultMaster Data ManagementData Governance FrameworksRequired Experience12-15 years of overall IT experience.Minimum 6+ years of hands-on Databricks experience.7+ years of experience in Spark/PySpark-based data engineering.Experience designing enterprise-scale Data Lake and Lakehouse solutions.Strong experience leading large data transformation and modernization programs.Experience handling structured, semi-structured, and unstructured data processing.Experience supporting analytics, AI/ML, and reporting ecosystems.Proven experience managing technical teams and driving architectural decisions.Preferred QualificationsBachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.Databricks Certified Data Engineer Professional.Azure Data Engineer Associate.AWS Data Analytics Specialty.Google Professional Data Engineer Certification.TOGAF or Cloud Architecture certifications are a plus.Nice to HaveMLOps and AI Engineering exposure.Snowflake and Databricks integration experience.Microsoft Fabric exposure.Data Mesh implementation experience.Data Governance tools such as Collibra, Alation, or Informatica.Expertise in Data Quality frameworks such as Great Expectations or Deequ.Key CompetenciesSolution ArchitectureTechnical LeadershipStakeholder ManagementProgram DeliveryData Strategy & GovernanceTeam Mentoring & CoachingProblem Solving & Decision MakingCommunication & Executive Presentation SkillsPerformance OptimizationInnovation & Continuous ImprovementTechnology ClassificationTechnology Tower: Data & AIPractice Area: Data EngineeringPrimary Skill: Databricks Lakehouse EngineeringSecondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.QualificationsGraduate with 4 years BS or Professional Degree. 15 - 18 YearsResponsibilitiesManagement of DB Systems, Performance Tuning, Security, Compliance Standards,

Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Lead Databricks Data Engineer in New York, NY vacancy
  • $112k - $200k

     ...Lead Data Engineer New York, NY (Midtown) | Hybrid, 50% in office | $112,000 - $200,000 base + discretionary bonus I’m working with...  ...business teams in a large enterprise Nice to have Databricks and Spark Azure Data Factory and other Azure data services... 
    Databricks
    Work at office

    Lawrence Harvey

    New York, NY
    16 hours ago
  •  ...A global financial institution is seeking a hands-on Lead Data Engineer to design and deliver scalable enterprise data solutions while contributing...  ...initiative. The engineer will work with Snowflake and Databricks, build production data pipelines, implement dimensional data... 
    Databricks
    Full time

    Madison-Davis, LLC

    New York, NY
    1 day ago
  •  ...Logicbroker is the Agentic Commerce Orchestration Engine helping enterprise retailers, brands,...  ...Job Summary Logicbroker is hiring a Lead Data Engineer to drive delivery across our...  ...You’ll lead the build of a modern Databricks platform and shape the Gold-layer models... 
    Databricks
    Full time
    Contract work
    Work at office
    Remote work

    Logicbroker

    New York, NY
    16 hours ago
  • $112k - $200k

    Join Mizuho as a Lead Data Engineer!We are seeking an experienced LeadData Engineer with 7+ years of experience designing, building, and delivering...  ...reliability. Experience with modern data platforms such as Databricks and Spark, as well as data governance and enterprise... 
    Databricks
    Full time
    Work experience placement
    Work at office
    Worldwide

    Mizuho Financial Group

    New York, NY
    10 hours ago
  • $120k - $180k

     ...within the newly consolidated Data and Analytics (D&A)...  ...infrastructure, data governance, data engineering, data modeling, data analysis...  ...intelligence, data science, and AI.The Lead Data Engineer is crucial to D...  ...utilizing Big Data and Azure/Databricks for Eligibility Program data... 
    Databricks
    Full time
    Temporary work
    Work at office
    Local area
    Relocation package
    3 days per week

    Metropolitan Life Insurance Company

    New York, NY
    2 days ago
  • Lead Data Migration - Archival - DR Engineer Seize your opportunity to make a personal impact supporting the Case Management Modernization (CMM) Program...  ...Professional (CDMP) Snowflake SnowPro Advanced Databricks Certified Data Professional AWS Certified Data Analytics... 
    Databricks
    Work at office

    Gdit

    New York, NY
    2 days ago
  • Title: Lead Data Engineer - Databricks Location: New York, NY (3 days/week or US Remote) Ideal Experience Range: 6-12 Years Interview process: 3 rounds, including one onsite round in NY/NJ or at any EXL office across the US. Responsibilities Data Engineering & Platform... 
    Databricks
    Work at office
    Remote work
    3 days per week

    Wise Skulls

    New York, NY
    2 days ago
  • $170k - $240k

    Lead - Data Engineer We are seeking a Lead - Data Engineer to lead the hands-on design, development, and operation of key components of Astrana...  ..., performance, security, and automation within an Azure Databricks environment. As a hands-on technical lead, you will... 
    Databricks
    Remote work

    KZX - Astrana Health Management, Inc.

    New York, NY
    2 days ago
  • $127k - $236.5k

    Lead, Data Engineer L3Harris Enterprise Data and AI team is seeking a Data Engineer with experience in managing enterprise-level data life...  ...development methodologies Experience with Python, Pandas, Databricks, JavaScript, Typescript or other scripting languages Experience... 
    Databricks
    Remote job
    Flexible hours

    ClearanceJobs

    New York, NY
    2 days ago
  • Lead Machine Learning / Data Science Engineer CapTech is an award-winning consulting firm that collaborates with clients to achieve what's possible through...  ...warehousing tools/environments such as Snowflake, Databricks, Azure SQL, Amazon RDS Comfort and proficiency in framing... 
    Databricks
    Work at office
    Remote work
    Visa sponsorship
    Work visa
    Flexible hours

    CapTech Consulting

    New York, NY
    2 days ago
  • Job Description:DXC Technology is a leading global technology services provider helping...  ...Insurance.Position SummaryThe Data Engineering Lead is responsible for the design, development...  ...platforms (Azure, AWS, or Google Cloud)Databricks, Snowflake, Synapse, Redshift, BigQuery... 
    Databricks
    Minimum wage
    Full time
    Worldwide

    DXC Technology

    New York, NY
    10 hours ago
  •  ...A global financial institution is seeking a Lead Data Platform Engineer to drive adoption and enablement for a strategic enterprise cloud data...  ...projects. ~ Advanced SQL experience. ~ Experience with Databricks, Snowflake, or comparable modern enterprise data... 
    Databricks
    Full time
    Immediate start

    Madison-Davis, LLC

    New York, NY
    3 days ago
  • Job Title: Senior Data Engineer / Lead Data Engineer Location: Hybrid , NJ Employment Type: Full-Time NEED LOCAL CANDIDATE, NY OR NJ EXP...  ...Apache Spark, and SQL . Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake . Work with... 
    Databricks
    Full time
    Local area

    Tekwissen

    New York, NY
    2 days ago
  •  ...About the Role This role owns the data infrastructure that investment/research/trading...  ...Set the technical standards other engineers will build against as the team grows...  ...Strong command of Python, SQL, Spark/Databricks, and modern orchestration tools Demonstrated... 
    Databricks
    Full time

    Mondrian Alpha

    New York, NY
    10 hours ago
  •  ...Data Protection Engineer In this pivotal role, you will design, develop, and execute a comprehensive...  ...our data governance maturity. Leading a dedicated team of data governance...  ...observability platforms (e.g., Atlan, Databricks, Anomalo, or equivalent).... 
    Databricks

    Apollo Solutions

    New York, NY
    8 days ago
  •  ...Title: Data Engineer (Early Career) Duration : FTE/Permanent Salary : 90-95k Location : Onsite in NYC Junior Data Engineer...  ...Shaping data models and tuning performance on Snowflake, Databricks, or a similar platform. Automating how data comes in, gets... 
    Databricks
    Permanent employment
    Work experience placement

    Gotham Technology Group

    New York, NY
    13 days ago
  • $180k - $220k

     ...COMPENSATION RANGE IN THE $180,000-$220,000 ANNUAL RANGE.*** Data Engineer develops and maintains Microsoft Fabric Lakehouse solutions,...  ...enterprise data platforms leveraging OneLake, Data Factory, Databricks, and Azure services. Develop scalable data ingestion,... 
    Databricks

    DBI Staffing

    New York, NY
    14 days ago
  •  ...client in the investment space is looking for an Analytics Engineer to help design and ship the data, analytics, and AI-enabled tools that investment...  ...Exposure to a distributed data platform such as Spark, Databricks, or Snowflake. Working knowledge of at least one asset... 
    Databricks
    Full time

    Cleartech Recruiting

    New York, NY
    3 days ago
  • $140k - $155k

     ...innovative technology to deliver industry-leading digital solutions. Synechron’s...  ...Consulting, Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of noteworthy...  ...users. Python/SQL Server/Snowflake/Databricks application development and system design... 
    Databricks
    Full time
    Temporary work
    Flexible hours

    Synechron

    New York, NY
    10 hours ago
  • $137.7k - $229.5k

    Position Summary Our Deloitte AI & Engineering team works to transform technology platforms, drive innovation, and help make a significant...  ..., and fuel growth through innovation.Work You’ll DoAs a Lead Data Engineer II on the team, you will be responsible for:Support... 
    Local area

    Deloitte

    New York, NY
    1 day ago
  • Cloud Engineer Location: Remote/US Duration: 6+ Months Job Description: Primary Job Function: The cloud engineering function provides in-...  ...incorporating industry best practices. May assist senior engineer or team lead in larger scale more complex cloud designs. Requirements:... 
    Remote work

    Zortech Solutions

    New York, NY
    2 days ago
  •  ...Senior GenAI Data Scientist New York This is an opportunity to work on enterprise...  ...leveraging the latest capabilities within the Databricks ecosystem, including Genie AI and MLflow....  ...interfaces over enterprise data. Lead customers through the full solution lifecycle... 
    Databricks
    Full time

    Harnham

    New York, NY
    2 days ago
  • Databricks Data Engineer Data and Analytics | Data Engineering Atlanta, Georgia, United States Austin, Texas, United States Boston, Massachusetts, United States Charlotte, North Carolina, United States Chicago, Illinois, United States Cincinnati, Ohio, United States Cleveland... 
    Databricks

    Deloitte Digital

    New York, NY
    2 days ago
  • $100k - $180k

     ...New York, United States Permanent Data Catalog EngineerNew York, NY - Hybrid (3...  ...) My client is seeking a Data Catalog Engineer to join a growing Data Governance team focused...  ...catalog platforms like Atlan, Collibra, Databricks Unity Catalog, Alation, Informatica or... 
    Databricks
    Permanent employment
    Work at office
    3 days per week

    Apollo Solutions

    New York, NY
    4 days ago
  •  ...designing and building scalable and robust data pipelines to enable data-driven...  ...teams.Should have experience working on Databricks ,Airflow , Redshift , Snaplogic .Experience...  ...very large-scale data warehouse and data engineering projects.Experience developing low latency... 
    Databricks

    Virtusa

    New York, NY
    1 day ago
  •  ...healthcare company based in the Detroit is looking to hire a Data Engineer. They're looking for someone to work closely with stakeholders...  ...engineering solutions. Technical experience with Foundry, Databricks, or Snowflake is preferred, but not required. A background in... 
    Databricks
    Full time

    Strategic Employment

    New York, NY
    2 days ago
  • Job Description: Senior ML Data Engineer Feature Engineering ETL Qualifications 7 years in data engineering and at least 4 years focusing...  ...preparation for ML Proven experience managing pipelines on Databricks using Apache Spark with a strong understanding of the... 
    Databricks
    Remote job

    TriOptus LLC

    New York, NY
    2 days ago
  •  ...Contract! We are seeking a Senior Software Engineer with a strong background in machine...  ...experience in machine learning engineering, data science, statistical modeling, or a...  ...data platforms such as AWS, Snowflake, Databricks, or Apache Spark. Bachelor's degree in Computer... 
    Databricks
    Full time
    Contract work

    Optomi

    New York, NY
    1 day ago
  • $165k - $210k

    Flow Traders is looking for a Core Data Engineer to own the end-to-end lifecycle of our most fundamental data products, including our Security...  ...​Experience with a modern cloud data stack, such as Spark or Databricks, dbt, a cloud warehouse, and orchestration​Experience with... 
    Databricks
    Local area

    Flow Traders

    New York, NY
    4 days ago
  • Data Engineer Key skills are very strong SQL capability, data pipelines management, Python, and Airflow. Location is New York, mandatory...  ...how to develop DAGS. Expert in data warehouses like BigQuery, Databricks Deltalakehouse and how to programmatically ingest, cleanse,... 
    Databricks
    3 days per week

    Yochana

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Databricks Data Engineer. Be the first to apply!