Lead Databricks Developer
Acunor Inc
Location: New York(Hybrid)
Employment Type: Contract
Experience: 10+ Years Job Summary
We are seeking an experienced Lead Databricks Developer to design, develop, and lead the implementation of enterprise-scale data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in Databricks, Apache Spark, Delta Lake, cloud data platforms, and modern data engineering practices, along with the ability to lead technical teams and drive end-to-end delivery of scalable data solutions.
Key Responsibilities- Lead the design, development, and implementation of enterprise data engineering solutions using Databricks.
- Build and optimize scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
- Develop and manage Delta Lake architectures for reliable, high-performance data processing.
- Design and implement batch and streaming data pipelines for enterprise analytics.
- Build and optimize Lakehouse architectures supporting BI, AI, and machine learning workloads.
- Collaborate with Solution Architects, Data Architects, Data Scientists, and business stakeholders to define technical solutions.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Implement data quality, governance, lineage, security, and monitoring best practices.
- Integrate data from multiple enterprise systems, APIs, databases, and cloud platforms.
- Develop reusable frameworks, coding standards, and CI/CD pipelines for data engineering.
- Mentor junior developers, perform code reviews, and provide technical leadership.
- Support production deployments, troubleshooting, and performance tuning.
- Drive Agile delivery while ensuring high-quality software engineering practices.
- 8+ years of experience in Data Engineering with at least 4+ years of hands-on Databricks development.
- Strong expertise in Databricks Lakehouse Platform.
- Advanced experience with Apache Spark (PySpark and/or Scala).
- Strong knowledge of Delta Lake, Spark SQL, and distributed data processing.
- Experience building enterprise-scale ETL/ELT pipelines.
- Strong proficiency in Python and SQL.
- Experience with Azure Databricks, AWS Databricks, or GCP Databricks.
- Hands-on experience with cloud storage technologies such as ADLS, S3, or Google Cloud Storage.
- Experience integrating structured and unstructured data sources.
- Knowledge of data modeling, data warehousing, and dimensional modeling.
- Experience with Git, CI/CD pipelines, and DevOps practices.
- Strong understanding of data governance, security, and performance optimization.
- Experience leading technical teams and mentoring developers.
- Strong communication and stakeholder management skills.
- Experience working in Agile/Scrum environments.
- Experience with Azure Data Factory, Azure Synapse, AWS Glue, or similar cloud data services.
- Knowledge of Kafka, Event Hubs, or other streaming technologies.
- Experience supporting AI/ML data platforms and feature engineering.
- Databricks Certified Data Engineer certification.
- Experience implementing enterprise data modernization and cloud migration initiatives.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Lead Databricks Developer in New York, NY vacancy
$200k - $230k
About UsYipitData is the leading market research and analytics firm for the disruptive economy and most recently raised $475M from The... ...Python, TypeScript/React, and cloud infrastructure (AWS, Databricks). You learn new tools fast and don't let unfamiliar tech slow...DatabricksWork at officeRemote workVisa sponsorshipFlexible hours$163.1k - $218.7k
...global organization of engineers, product developers, designers, technologists, data... ...and take action quickly.Job Summary:As a Lead Software Engineer, you will design and build... ...processing tools (e.g., PySpark, Pandas, Databricks, Snowflake).Knowledge of prompt design,...Databricks$189.2k - $372.9k
Position Summary At Deloitte, Lead Forward Deployed Engineers (LFDE) don’t just... ...September 30, 2026 Work you’ll do As a Lead Databricks FDE, you will serve as the senior... ...leading forward-deployed engineering pods that develop and deploy GenAI solutions into production...DatabricksLocal areaVisa sponsorship$189.2k - $372.9k
...ventures, and fuel growth through innovation. Work you'll do As a Lead Databricks FDE, you will serve as the senior practitioner-leader... ...strategic clients, leading forward-deployed engineering pods that develop and deploy GenAI solutions into production for Deloitte’s...DatabricksLocal area- ...Lead Software EngineerDisney Entertainment and ESPN Product & Technology is a global organization of engineers, product developers, designers, technologists, data scientists, and more – all working... ...tools (e.g., PySpark, Pandas, Databricks, Snowflake).Knowledge of prompt...Databricks
Disney
New York, NY3 days ago $157k - $200k
...work for people, companies, and the planet.Role DescriptionAs a Lead Value Engineer specializing in the Life Sciences, you are... ...with top AI and ML technology partners (e.g., Microsoft, OpenAI, Databricks)..With Celonis’ Process Intelligence (PI) platform, we feed operational...DatabricksFull timeWork experience placementWork at officeLocal areaImmediate startRemote workWorldwideFlexible hoursShift work$229.9k - $262.4k
...Senior Lead Software Engineer, Full Stack (Python, ReactJS, AWS) Do you love building... ...technology projects and a team of developers with deep experience in distributed microservices... .... ~1+ years of experience with Databricks. ~1+ years of experience with Snowflake...DatabricksInternshipLocal areaJobtailor
New York, NY1 day ago$148.7k - $199.4k
Job Posting Title:Lead Machine Learning Engineer - NewsReq ID:10150521Job Description:Disney... ...and ESPN+, and much more.Innovation: We develop and execute groundbreaking products and... ...with big data technologies such as Databricks, Kinesis, KafkaProven leadership, coaching...DatabricksFull timeLocal areaWorldwideDay shift$163.1k - $218.7k
Job Posting Title:Lead Product Software EngineerReq ID:10149877Job Description:Disney... ...global organization of engineers, product developers, designers, technologists, data... ...stacks (e.g., Looker/LookML, Snowflake, Databricks) and partnering with data and analytics...DatabricksFull timeWork experience placement- ...T itle: Lead Machine Learning Engineer (Recommendation Systems) Location: Remote (USA) Duration: 1-2 years Kforce... ...production-grade ML systems . Strong hands-on experience with Databricks, MLflow, Python,TensorFlow/PyTorch/Scikit-learn ....DatabricksRemote work
Conch Technologies Inc
New York, NY2 days ago - ...push the limits of what's possible. As a Lead Software Engineer at JPMorganChase... ...seeking a well-rounded hands-on quantitative developer that is experienced in building systems... ...Spark and modern data platforms (e.g., Databricks).Proficiency in data engineering and platform...Databricks
$215.2k - $245.6k
...Lead Software Engineer (Java, Python, AWS)Do you love building and pioneering in the... ...diverse technology projects and a team of developers with deep experience in distributed... ...services including Docker and Kubernetes, Databricks and a variety of AWS tools and servicesBasic...DatabricksFull timePart timeInternshipH1bLocal areaCapital One
New York, NY1 day ago$119.6k - $222k
...Lead Engineer Epsilon is looking for a Lead Engineer to help drive the design, delivery, technical strategy, and execution of core... ...modern, big data platforms and processing frameworks such as Databricks, Snowflake, or PySpark. AI: Practical exposure to Generative...DatabricksTemporary workFreelanceLocal areaFlexible hoursDigitas
New York, NY3 days ago- Capgemini is seeking an ETL Test Lead to drive data validation, ETL testing, and reporting solutions across large data warehouses. The role requires strong Azure Databricks experience and leadership of offshore teams with 5-8 years onsite interaction with clients. The ideal...Databricks
$305k
...Solutions Engineering team and looking for Lead SEs who can operate across the full... ...conversations about data warehouses (Snowflake, Databricks, Redshift, BigQuery), SQL, and analytics infrastructure. You don't need to be a developer, but you need to hold your own with...DatabricksContract workLive inWork at officeWork from homeTextQL, Inc.
New York, NY4 days ago- ...Lead ML/AI Engineer New York, New York, United States Or refer someone Job Openings... ...evaluation of machine learning models. Develop strategies for deploying machine learning... ...with Snowflake/Big Query, Google Dataproc/Databricks or any big data frameworks on Spark ~...Databricks
Inizio Partners
New York, NY1 day ago $229.9k - $262.4k
...Senior Lead Software Engineer, Distributed Systems (Golang + Python on Kubernetes)... ...diverse technology projects and a team of developers with deep experience in distributed microservices... ...of experience with Amazon SageMaker, Databricks, or another comprehensive ML platform...DatabricksFull timePart timeInternshipLocal areaCapital One
New York, NY3 days ago$229.9k - $262.4k
...Overview Senior Lead Software Engineer, Full Stack (Python, ReactJS, AWS) Do you... ...diverse technology projects and a team of developers with deep experience in distributed... ...experience ~1+ years of experience with Databricks ~1+ years of experience with Snowflake...DatabricksFull timePart timeInternshipLocal areaCapital One
New York, NYmore than 2 months ago- Versant in New York, NY is seeking a skilled Manager, Analytics Engineering to lead the evolution of our Databricks environment. This role involves building high-quality datasets and forecasting models that provide insights across audience, content, and revenue. The ideal...Databricks
$190k - $230k
...but delivering them into production. The Lead UI Engineerwill play a critical role on... ...relationships, maintaining a commercial approach. Develop strong relationships with clients as a... ...in a finance context such as Salesforce, Databricks, and others. Why Lab49? Lab49 is an...DatabricksFull timeTemporary workImmediate start- Acuity Inc. (NYSE: AYI) is a market-leading industrial technology company. We use technology... ...systems . The role involves designing, developing, and maintaining scalable enterprise-... ...data-driven features and pipelines using Databricks, Python, SQL, EventHub, Service Bus and...DatabricksContract work
- ...operating model for our clients. Responsibilities Design, develop, and maintain scalable backend services using Java and... ...microservices , and distributed systems Nice-to-have Databricks OR AWS EMR OR Hadoop Search technologies experience , such...DatabricksFull timeContract workWork at office
Lineate
New York, NY1 day ago - ...Proven expertise in cloud architecture and services, particularly AWS and Azure • Strong experience with data platforms, including DataBricks and Snowflake • Proficiency in Microsoft Power Platform tools like PowerApps and Dynamics 365 CE • Familiarity with...DatabricksFull time
Nigel Frank International
New York, NYmore than 2 months ago - ..., test, and maintain predictive models for pricing, reserving, underwriting, and claims, using Python, R, SQL, Machine Learning, Databricks, and Cloud platforms. Ideal candidate is ACAS or FCAS with 3+ years of P&C actuarial experience involving reserving, pricing, or...Databricks
- ...Qualifications & Requirements Experience ~5 to 7 years of data engineering experience, with at least 3+ years of hands-on Databricks experience in production environments. ~ Migration Expertise Proven experience in migrating on-premises data systems to...DatabricksPermanent employmentFull time
Parkar Global Technologies Pvt. Ltd.
New York, NY2 days ago - ...Edgewaterit is seeking a Solutions Architect 4 to support a major national laboratory. This role involves advising on Databricks architecture, providing hands-on design guidance, and mentoring data engineering teams. The ideal candidate will have over 7 years of experience...Databricks
Edgewater IT LLC Defunct
New York, NY3 days ago - ...data lakehouse platforms (preferably AWS data lake services or Databricks, alternatively Hadoop), atleast one relational data stores (... ...Dimensional, Data Vault, Kimball, Inmon, etc.), Agile methodology (develop PI plans and roadmaps), TDD (or BDD) and CI/CD tools (Jenkins,...Databricks
- ...SLAs.Demonstrated experience as a change agent, successfully leading and implementing organizational change initiatives that drive... ...industry-leading data platform technologies such as Data Lake, Databricks, SnowflakeExperience with database technologies for both structured...DatabricksLocal area3 days per week
- ...A leading consulting firm is seeking a Manager of Talent Acquisition to lead client engagements and drive discovery strategies. The... ...cloud data engineering, particularly with platforms like GCP and Databricks. Responsibilities include managing project discussions,...DatabricksRemote work
Tredence
New York, NY5 days ago - +10 years in cloud and data platform engineering or architecture, +5 years with Databricks w/ AWS, Strong AWS and services such as S3, IAM, VPC, and PrivateLink (Front + Backend), Hands on Terraform, Databricks platform security, etcDatabricks
RIT Solutions, Inc.
New York, NY1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Databricks Developer. Be the first to apply!
Related searches
- lead web developer New York, NY
- lead industrial engineer New York, NY
- lead operating engineer New York, NY
- lead algorithm engineer New York, NY
- lead app. developer New York, NY
- lead infrastructure engineer New York, NY
- lead network engineer New York, NY
- lead engineer New York, NY
- lead system engineer New York, NY
- databricks New York, NY