Lead Databricks Developer

Acunor Inc

Location: New York(Hybrid)

Employment Type: Contract

Experience: 10+ Years Job Summary

We are seeking an experienced Lead Databricks Developer to design, develop, and lead the implementation of enterprise-scale data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in Databricks, Apache Spark, Delta Lake, cloud data platforms, and modern data engineering practices, along with the ability to lead technical teams and drive end-to-end delivery of scalable data solutions.

Key Responsibilities

Lead the design, development, and implementation of enterprise data engineering solutions using Databricks.
Build and optimize scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
Develop and manage Delta Lake architectures for reliable, high-performance data processing.
Design and implement batch and streaming data pipelines for enterprise analytics.
Build and optimize Lakehouse architectures supporting BI, AI, and machine learning workloads.
Collaborate with Solution Architects, Data Architects, Data Scientists, and business stakeholders to define technical solutions.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Implement data quality, governance, lineage, security, and monitoring best practices.
Integrate data from multiple enterprise systems, APIs, databases, and cloud platforms.
Develop reusable frameworks, coding standards, and CI/CD pipelines for data engineering.
Mentor junior developers, perform code reviews, and provide technical leadership.
Support production deployments, troubleshooting, and performance tuning.
Drive Agile delivery while ensuring high-quality software engineering practices.

Required Qualifications

8+ years of experience in Data Engineering with at least 4+ years of hands-on Databricks development.
Strong expertise in Databricks Lakehouse Platform.
Advanced experience with Apache Spark (PySpark and/or Scala).
Strong knowledge of Delta Lake, Spark SQL, and distributed data processing.
Experience building enterprise-scale ETL/ELT pipelines.
Strong proficiency in Python and SQL.
Experience with Azure Databricks, AWS Databricks, or GCP Databricks.
Hands-on experience with cloud storage technologies such as ADLS, S3, or Google Cloud Storage.
Experience integrating structured and unstructured data sources.
Knowledge of data modeling, data warehousing, and dimensional modeling.
Experience with Git, CI/CD pipelines, and DevOps practices.
Strong understanding of data governance, security, and performance optimization.
Experience leading technical teams and mentoring developers.
Strong communication and stakeholder management skills.
Experience working in Agile/Scrum environments.

Preferred Qualifications

Experience with Azure Data Factory, Azure Synapse, AWS Glue, or similar cloud data services.
Knowledge of Kafka, Event Hubs, or other streaming technologies.
Experience supporting AI/ML data platforms and feature engineering.
Databricks Certified Data Engineer certification.
Experience implementing enterprise data modernization and cloud migration initiatives.

Apply

Vacancy posted 3 days ago

Similar jobs that could be interesting for youBased on the Lead Databricks Developer in New York, NY vacancy

Tech Lead - Senior Software Engineer
$200k - $230k
About UsYipitData is the leading market research and analytics firm for the disruptive economy and most recently raised $475M from The... ...Python, TypeScript/React, and cloud infrastructure (AWS, Databricks). You learn new tools fast and don't let unfamiliar tech slow...
Databricks
Work at office
Remote work
Visa sponsorship
Flexible hours
YipitData
New York, NY
5 days ago
Lead Software Engineer
$163.1k - $218.7k
...global organization of engineers, product developers, designers, technologists, data... ...and take action quickly.Job Summary:As a Lead Software Engineer, you will design and build... ...processing tools (e.g., PySpark, Pandas, Databricks, Snowflake).Knowledge of prompt design,...
Databricks
Disney Interactive
New York, NY
2 days ago
Lead Forward Deployed Engineer - Databricks
$189.2k - $372.9k
Position Summary At Deloitte, Lead Forward Deployed Engineers (LFDE) don’t just... ...September 30, 2026 Work you’ll do As a Lead Databricks FDE, you will serve as the senior... ...leading forward-deployed engineering pods that develop and deploy GenAI solutions into production...
Databricks
Local area
Visa sponsorship
Deloitte
New York, NY
2 days ago
Lead Software Engineer
...Lead Software EngineerDisney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more – all working... ...tools (e.g., PySpark, Pandas, Databricks, Snowflake).Knowledge of prompt...
Databricks
Disney
New York, NY
2 days ago
Lead Databricks Forward Deployed Engineer - GPS
$189.2k - $372.9k
...ventures, and fuel growth through innovation. Work you'll do As a Lead Databricks FDE, you will serve as the senior practitioner-leader... ...strategic clients, leading forward-deployed engineering pods that develop and deploy GenAI solutions into production for Deloitte’s...
Databricks
Local area
Deloitte
New York, NY
1 day ago
Lead Software Engineer
...Lead Software Engineer Disney Entertainment and ESPN Product & Technology Technology... ...organization of engineers, product developers, designers, technologists, data scientists... ...tools (e.g., PySpark, Pandas, Databricks, Snowflake). Knowledge of prompt design...
Databricks
Walt Disney Company
New York, NY
3 days ago
Lead Value Engineer - Life Sciences
$157k - $200k
...work for people, companies, and the planet.Role DescriptionAs a Lead Value Engineer specializing in the Life Sciences, you are... ...with top AI and ML technology partners (e.g., Microsoft, OpenAI, Databricks)..With Celonis’ Process Intelligence (PI) platform, we feed operational...
Databricks
Full time
Work experience placement
Work at office
Local area
Immediate start
Remote work
Worldwide
Flexible hours
Shift work
Celonis
New York, NY
1 day ago
Lead AI Engineer
...About the job Lead AI Engineer Role : Lead AI & Analytics Engineer... ...orchestration, data modeling, and platforms such as Databricks, Snowflake, or GCP We are looking for... ..., analysts, and business stakeholders to develop reliable data solutions and ensure high-quality...
Databricks
Remote work
Inizio Partners
New York, NY
3 days ago
Lead Software Engineer, Full Stack
$197.3k - $225.1k
...Lead Software Engineer, Full Stack Do you love building and pioneering in the technology... ...technology projects and guide a team of developers with deep experience in distributed... ...data ETL using Dataflow, AWS Glue or Databricks ~1+ years of experience utilizing Generative...
Databricks
Full time
Part time
Internship
H1b
Local area
Capital One
New York, NY
4 days ago
Senior Lead Software Engineer, Full Stack (Python, ReactJS, AWS)
$229.9k - $262.4k
...Senior Lead Software Engineer, Full Stack (Python, ReactJS, AWS) Do you love building... ...technology projects and a team of developers with deep experience in distributed microservices... .... ~1+ years of experience with Databricks. ~1+ years of experience with Snowflake...
Databricks
Internship
Local area
Jobtailor
New York, NY
4 days ago
Lead Machine Learning Engineer - News
$148.7k - $199.4k
Job Posting Title:Lead Machine Learning Engineer - NewsReq ID:10150521Job Description:Disney... ...and ESPN+, and much more.Innovation: We develop and execute groundbreaking products and... ...with big data technologies such as Databricks, Kinesis, KafkaProven leadership, coaching...
Databricks
Full time
Local area
Worldwide
Day shift
Hulu
New York, NY
4 days ago
Senior Lead Software Engineer, Full Stack (Python, ReactJS, AWS)
$229.9k - $262.4k
...Senior Lead Software Engineer, Full Stack (Python, ReactJS, AWS) Do you love building... ...technology projects and a team of developers with deep experience in distributed microservices... ...~1+ years of experience with Databricks ~1+ years of experience with Snowflake...
Databricks
Full time
Part time
Internship
Local area
Capital One
New York, NY
3 days ago
Lead Machine Learning Engineer (Recommendation Systems)
...T itle: Lead Machine Learning Engineer (Recommendation Systems) Location: Remote (USA) Duration: 1-2 years Kforce... ...production-grade ML systems . Strong hands-on experience with Databricks, MLflow, Python,TensorFlow/PyTorch/Scikit-learn ....
Databricks
Remote work
Conch Technologies Inc
New York, NY
3 days ago
Lead Product Software Engineer
$163.1k - $218.7k
Job Posting Title:Lead Product Software EngineerReq ID:10149877Job Description:Disney... ...global organization of engineers, product developers, designers, technologists, data... ...stacks (e.g., Looker/LookML, Snowflake, Databricks) and partnering with data and analytics...
Databricks
Full time
Work experience placement
Hulu
New York, NY
1 day ago
Lead Software Engineer - Quantitative Portfolio Construction Development
...push the limits of what's possible. As a Lead Software Engineer at JPMorganChase... ...seeking a well-rounded hands-on quantitative developer that is experienced in building systems... ...Spark and modern data platforms (e.g., Databricks).Proficiency in data engineering and platform...
Databricks
JP Morgan Chase
New York, NY
1 day ago
Lead Software Engineer
$119.6k - $222k
...Lead EngineerEpsilon is looking for a Lead Engineer to help drive the design, delivery, technical strategy, and execution of core... ...with modern, big data platforms and processing frameworks such as Databricks, Snowflake, or PySpark.AI: Practical exposure to Generative AI...
Databricks
Temporary work
Freelance
Local area
Flexible hours
Starcom Mediavest Group Germany Gmbh
New York, NY
3 days ago
Lead Applied AI Engineer
...Lead Applied AI Engineer Paramount Skydance Corp. is seeking a Senior Applied AI Engineer... .../ML Solution Development Architect, develop, and deploy end-to-end AI/ML systems... ...platforms (MLflow, Kubeflow, SageMaker, Databricks) and cloud platforms (Azure, AWS, or GCP)...
Databricks
Paramount Global Services
New York, NY
1 day ago
Lead UI Engineer (React), New York
$190k - $230k
...but delivering them into production.   The Lead UI Engineerwill play a critical role on... ...relationships, maintaining a commercial approach. Develop strong relationships with clients as a... ...in a finance context such as Salesforce, Databricks, and others.  Why Lab49? Lab49 is an...
Databricks
Full time
Temporary work
Immediate start
ION Group
New York, NY
3 days ago
Software Engineer Java + Data (PySpark)
...operating model for our clients. Responsibilities Design, develop, and maintain scalable backend services using Java and... ...microservices , and distributed systems Nice-to-have Databricks OR AWS EMR OR Hadoop Search technologies experience , such...
Databricks
Full time
Contract work
Work at office
Lineate
New York, NY
1 day ago
Senior Azure Cloud Engineer
...fine Client: Not Disclosed Senior candidates needed: Local candidates needed: Key skills: 8+ years of Azure, API Development, Databricks (other skills to be determined after conversation with manager) The Enterprise Cloud Team seeks a Senior Azure Cloud Engineer...
Databricks
Local area
Remote work
ShiftCode Analytics
New York, NY
4 days ago
Enterprise Applications Manager
...Proven expertise in cloud architecture and services, particularly AWS and Azure • Strong experience with data platforms, including DataBricks and Snowflake • Proficiency in Microsoft Power Platform tools like PowerApps and Dynamics 365 CE • Familiarity with...
Databricks
Full time
Nigel Frank International
New York, NY
more than 2 months ago
Senior P&C Modeling Actuary
..., test, and maintain predictive models for pricing, reserving, underwriting, and claims, using Python, R, SQL, Machine Learning, Databricks, and Cloud platforms. Ideal candidate is ACAS or FCAS with 3+ years of P&C actuarial experience involving reserving, pricing, or...
Databricks
Pryor Associates Executive Search
New York, NY
3 days ago
Data Engineer
...The ideal candidate should have hands‑on experience with Azure, Databricks, Python, Spark, and SQL, along with a strong understanding of... ...analytics solutions. Roles & Responsibilities Design, develop, and maintain scalable data pipelines using Azure and Databricks...
Databricks
Fabric
New York, NY
1 day ago
Databricks Solutions Architect - Data Platform Lead
...Edgewaterit is seeking a Solutions Architect 4 to support a major national laboratory. This role involves advising on Databricks architecture, providing hands-on design guidance, and mentoring data engineering teams. The ideal candidate will have over 7 years of experience...
Databricks
Edgewater IT LLC Defunct
New York, NY
1 day ago
Data Engineer
...data lakehouse platforms (preferably AWS data lake services or Databricks, alternatively Hadoop), atleast one relational data stores (... ...Dimensional, Data Vault, Kimball, Inmon, etc.), Agile methodology (develop PI plans and roadmaps), TDD (or BDD) and CI/CD tools (Jenkins,...
Databricks
HAN Staffing
New York, NY
4 days ago
Sr. Data Engineer with ML
...environment of on-premises open-source tools and cloud-based platforms. The ideal candidate will have experience working with MLflow, Databricks, and open-source machine learning frameworks, enabling seamless integration, experiment tracking, model management, and...
Databricks
Contract work
Accord Technologies Inc
New York, NY
1 day ago
Data Engineer
...Fitch Ratings is hiring a Lead/Principal Data Engineer based in New York. The role involves leading the design and architecture of... ...cloud-based data solutions with technologies such as Snowflake, Databricks, and PostgreSQL. We seek candidates with over 8 years of data engineering...
Databricks
Fitch Ratings
New York, NY
3 days ago
Director OF Data Management
...A leading consulting firm is seeking a Manager of Talent Acquisition to lead client engagements and drive discovery strategies. The... ...cloud data engineering, particularly with platforms like GCP and Databricks. Responsibilities include managing project discussions,...
Databricks
Remote work
Tredence
New York, NY
3 days ago
Corporate Vice President - Data Architect
...SLAs.Demonstrated experience as a change agent, successfully leading and implementing organizational change initiatives that drive... ...industry-leading data platform technologies such as Data Lake, Databricks, SnowflakeExperience with database technologies for both structured...
Databricks
Local area
3 days per week
New York Life Insurance Company
New York, NY
3 days ago
Senior Data Engineer
$160k - $200k
...understand their most difficult data challenges, and then go head-down and build a solution. You'll bring deep, hands-on expertise with Databricks and the broader modern data stack, and you'll set the technical standard for how we design, build, and deliver data platforms...
Databricks
Work experience placement
Rearc
New York, NY
5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Databricks Developer. Be the first to apply!