Lead Databricks Engineer

Abode Tech Zone

Role : Lead Databricks Engineer
Location : Iselin, NJ / NY (Hybrid)

zure Data Engineer with Databricks Expertise

Job Summary:
We are seeking highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement and optimize large-scale data pipeline, ensuring scalability, reliability and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.

Key Responsibilities:

Data Pipeline Development:
Build and maintain scalable ETL/ELT pipelines using Databricks.
Leverage PySpark/Spark and SQL to transform and process large datasets.
Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems.
Collaboration & Analysis:
Work Closely with multiple teams to prepare data for dashboard and BI Tools.
Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
Performance & Optimization:
Optimize Databricks workloads for cost efficiency and performance.
Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
Governance & Security:
Implement and manage data security, access controls and governance standards using Unity Catalog.
Ensure compliance with organizational and regulatory data policies.
Deployment:
Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments.
Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.

Technical Skills:

Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime etc.)
Proficiency in Azure Cloud Services.
Solid Understanding of Spark and PySpark for big data processing.
Experience in relational databases.
Knowledge on Databricks Asset Bundles and GitLab.

Preferred Experience:

Familiarity with Databricks Runtimes and advanced configurations.
Knowledge of streaming frameworks like Spark Streaming.
Experience in developing real-time data solutions.

Certifications:

zure Data Engineer Associate or Databricks certified Data Engineer Associate certification. (Optional)

Apply

Vacancy posted 7 hours ago

Similar jobs that could be interesting for youBased on the Lead Databricks Engineer in United States vacancy

Lead AI Engineer
...Description: 4+ years of experience in AI/ML engineering, including 2+ years in MLOps or AI QA roles. 1+ years of experience in a lead role. Python, SQL, PySpark, and ML... ...frameworks such as TensorFlow or PyTorch. Experience with Azure Cloud, Databricks, and/or Snowflake....
Databricks
ShiftCode Analytics
Philadelphia, PA
12 hours ago
Lead BI Engineer
...Overview Follett Higher Education is a leading educational service provider and... ...America. The Lead Business Intelligence Engineer is responsible for designing and implementing... ...Factory, Azure Data Lake Storage, Azure Databricks, SQL Server, SSIS, SSAS, SSRS, and Power...
Databricks
Follett
Westchester, IL
1 day ago
L2/L3 Lead Support Engineer
...Job Title: L2/L3 Lead Support Engineer Location: Jersey City, NJ Duration: Full Time Employment Description: We need a L2/L3 Lead... .... Expert in Java, Python, Spark, AWS ( Redshift, Aurora, SQS, Glue, Airflow, etc), DBX Databricks skills and experience
Databricks
Full time
Lorven Technologies
Jersey City, NJ
1 day ago
Lead Machine Learning Ops Engineer
...Job Description: Role: Lead Machine Learning Ops Engineer Location: Houston, TX Duration: 6+ months - Probably long term Lead... ...on cloud platforms (e.g., AWS, Azure, or GCP ). Databricks, and AWS Preferred. Proficient with GitHub actions...
Databricks
Syntricate Technologies
Houston, TX
4 days ago
Lead I - Software Engineering
$64 - $69 per hour
...Senior Data Platform Security Architect to lead the design, implementation, and... ...architecture and works cross-functionally with engineering, data, compliance, product, and... ...Management. Familiarity with Snowflake, Databricks, Microsoft Fabric Security. Understanding...
Databricks
Remote work
Axelon
Aliso Viejo, CA
12 hours ago
Lead Databricks Engineer with Pyspark & Python Exp - US
...Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days' work from office) Duration: 6+ Months Job Description: Inperson interview is must (Financial background is must)...
Databricks
Work at office
Zortech Solutions
New York, NY
4 days ago
Lead Software Engineer - Remote
$112.7k - $193.2k
...production support Coach and mentor junior engineers Keep skills up to date through... ...years of experience in using tools like Databricks, Snowflake, Python, Pyspark, AKS ~2+ years... ...change and adapt quickly, mentoring and leading other engineers from a technical...
Databricks
Minimum wage
Full time
Work experience placement
Work at office
Local area
Remote work
Flexible hours
UnitedHealth Group
United States
12 hours ago
Lead Software Engineer - Remote
$89.9k - $160.6k
...years of experience in using tools like Databricks, Snowflake, python, pyspark, AKS2+... ...containerization and performance tuning in Data & Engineering PipelinesPreferred QualificationsPython... ...and adapt quickly, mentoring and leading other engineers from a technical...
Databricks
Minimum wage
Full time
Work experience placement
Work at office
Local area
Remote work
Flexible hours
Optum
Eden Prairie, MN
1 day ago
Lead Software Engineers
$159.22k - $222.2k
...IT: ADP Technology Services, Inc. seeks Lead Software Engineers at our Roseland, NJ loc. to engg & delver end-to-end data soltns to meet wide range of bus reqs usng IDMC, Databricks & Denodo. Bach's deg in Comp Sci, Comp Engg, Info Sys, rel'd Eng disc, or reld field +...
Databricks
ADP Technology Services, Inc.
Roseland, NJ
12 hours ago
Lead Databricks Engineer with Pyspark & Python Exp USA
...Lead Databricks Engineer Location: NYC, NY OR Iselin, NJ (Hybrid – 3 days’ work from office) Job Description: Inperson interview is must, Expenses will be paid for in-person if not local (Financial background is must) Major Responsibilities: Work on Finance...
Databricks
Work at office
Local area
Zortech Solutions
Iselin, NJ
4 days ago
Lead Pyspark / SQL Engineer - Threat Detection & DataBricks
...Job Summary: We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content...
Databricks
Remote work
Macpower Digital Assets Edge
United States
2 days ago
Lead AI Engineer - Bengaluru, INDIA
...Job Description Job Description Role: Lead AI Engineer - Bengaluru, INDIA Full Time *Consultants local to INDIA only Primary... ...production-ready AI applications and deploying them on Azure Databricks and Azure cloud infrastructure . You will work end-to-end:...
Databricks
Full time
Local area
Work from home
Flexible hours
Vytwo
Prosper, TX
3 days ago
Lead Engineering Manager, Scalable Compute Infrastructure
$190k - $235k
Databricks is looking for an Engineering Manager to lead a team responsible for critical components of their compute platform. This role will significantly impact product velocity and customer experience while ensuring the scalability of operations. The ideal candidate...
Databricks
I did my part and supported the Regular Toilet
Bellevue, WA
1 day ago
Lead Cloud Engineer
...Lead Cloud Engineer Architects and manage the Azure cloud infrastructure for end user device management like hardware, Operating Systems... ...cloud native solutions, using Azure functions, Azure Databricks, CI/CD pipelines and technical documentation. Lead the integration...
Databricks
Work experience placement
United IT
Houston, TX
12 hours ago
Lead BI Engineer - Westchester, IL
$100k - $120k
Back Lead BI Engineer #26-30066 Westchester, Illinois, United States Apply X Facebook LinkedIn Email Copy Job Category Information Technology... ...such as Azure Data Factory, Azure Data Lake Storage, Azure Databricks, SQL Server, SSIS, SSAS, SSRS, and Power BI. Develop and...
Databricks
Hourly pay
Full time
Part time
Local area
Follett
Westchester, IL
22 days ago
Senior Lead Software Engineer
$130k
...impact. At NAVEX you will join our Product Engineering team that shares a passion for designing... ...and bottom line. As our Senior Lead Software Engineer, you will provide technical... ...analytics platforms, such as Snowflake or Databricks Containerization and orchestration...
Databricks
NAVEX Global
New York, NY
1 day ago
AI/ML Lead Engineer
...Position: ML Engineer with python and AI Location: Chester , PA -onsite... ...Overview We are looking for an AI/ML Lead Engineer with strong Python, Machine Learning... ...manage data pipelines using SQL, Pandas, Databricks, Data Lakes, and NoSQL databases. • Implement...
Databricks
Contract work
VDart
Chester, PA
12 hours ago
Lead I - Software Engineering
...Lead I - Software Engineering Our team builds and maintains data products within the Care & Retail domain, delivering reliable, scalable solutions... ...(Required). Familiarity with Microsoft Fabric, Databricks, and the Azure data stack, including Azure Data Lake Storage...
Databricks
Work experience placement
Flexible hours
Merican
Bellevue, WA
2 days ago
Azure Data Lake & Databricks Lead Engineer
A leading technology consultancy in Glendale, Arizona, is seeking an experienced Software Engineering Lead to spearhead data initiatives focused on Azure Data Lake. The ideal candidate will have extensive experience in data engineering, robust leadership skills, and a strong...
Databricks
Contract work
Talascend, LLC
Glendale, AZ
1 day ago
LEAD ML ENGINEER
...DataFrames, data modeling fundamentals, feature engineering • DevOps: Git, CI/CD, Docker;... ...Responsibilities ML Engineering & Delivery • Lead the design and implementation of... ...DevOps / GitHub Actions integrated with Databricks • Leverage MLflow (Databricks native)...
Databricks
ClifyX
Blue Ash, OH
1 day ago
Lead Databricks Engineer
$106.08k - $176.82k
...Remote Lead Databricks Engineer Sentara Health is looking for a Remote Lead Databricks Engineer. This is a remote position available in the following states: Virginia, North Carolina, Alabama, Delaware, Florida, Georgia, Idaho, Indiana, Kansas, Louisiana, Maine,...
Databricks
Full time
Temporary work
Remote work
Shift work
Sentara Healthcare
United States
3 days ago
Lead Databricks Engineer | Data Pipelines
A data and AI consultancy based in New York is looking for a skilled Data Engineer to lead technical delivery on innovative projects. The ideal candidate will have expertise in Databricks, Apache Spark, Delta Lake, as well as advanced skills in Python and SQL. You will...
Databricks
Kubrick
New York, NY
3 days ago
Lead ML/DS Engineer
...cloud computing, with 5+ years in AI-specific roles Proven expertise in Azure AI, Machine Learning, OpenAI, Cognitive Services, and Databricks Strong experience in hybrid cloud AI architectures (Azure Arc, on-prem AI, edge AI, private AI) Familiarity with business...
Databricks
Innowise Group
Kansas City, MO
2 days ago
Lead AI/ML Engineer
...Lead AI/ML Engineer Our client in Washington, DC is seeking a highly skilled and hands-on Lead AI/ML Engineer to drive the design, development... ...front-end technologies (HTML/CSS/JavaScript) Champion Databricks adoption to enable AutoML, pipeline automation, and...
Databricks
Samprasoft
Washington DC
1 day ago
Lead Software Engineer - Full Stack
...where you can push the limits of what's possible. As a Lead Software Engineer at JPMorgan Chase within the Corporate Technology, you are... ...different data platforms including on premise Hadoop servers, AWS, Databricks and Kubernetes Performing unit test cases and...
Databricks
Work experience placement
JPMorgan Chase Bank, N.A.
Houston, TX
3 days ago
Lead Software Engineer - Markets, Java, AWS, Spring Boot, Terraform
...Lead Software Engineer We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible... ...analytics workflows, and reporting solutions using Python, Databricks, and Spark Develops and maintains web-based applications...
Databricks
Chase
Chicago, IL
1 day ago
Lead AI Software Engineer- AI Enablement
...Lead AI Software Engineer- AI Enablement The Mastercard Cloud Team, in Mastercard's ONE division, owns and drives cloud usage and adoption... ...open source models. Experience with AI/data platforms (Databricks, Iceberg, S3, watsonx, or similar) and GenAI developer tools...
Databricks
Dynamic Yield
O Fallon, MO
12 hours ago
LEAD ML ENGINEER
$120k - $140k
...Job :LEAD ML ENGINEER Location : Blue Ash, OH Onsite) Fulltime Only Salary Range : $120000.00-$140000.00/Year Fulltime Only... ...) using Azure DevOps / GitHub Actions integrated with Databricks Leverage MLflow (Databricks native) for experiment tracking...
Databricks
Full time
Diverse Lynx
Blue Ash, OH
4 days ago
Lead Forward Deployed Engineer, Databricks 2026- US, UK
...Lead Forward Deployed Engineer – Databricks Aimpoint Digital is a market-leading data, AI, and operations research advisory and solution engineering firm. We help organizations build modern data platforms, enterprise AI systems, decision intelligence solutions, optimization...
Databricks
Full time
Remote work
Aimpoint Digital
United States
1 day ago
Lead Software Engineer
$215.2k - $245.6k
...Lead Software Engineer Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast... ...stack centered around Java, Spring, AWS, DynamoDB, and Databricks. Joining this team means taking end-to-end ownership of highly...
Databricks
Full time
Part time
Internship
H1b
Local area
Capital One Financial Corp
New York, NY
4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Databricks Engineer. Be the first to apply!