Lead Data Engineer - Azure Databricks
Conch Technologies Inc
Job Title: Lead Data Engineer Azure Databricks
Location: Remote/Hybrid
Duration :6+ Months
Job Summary
We are looking for an experienced Lead Data Engineer to support enterprise data engineering, analytics, and cloud initiatives. The ideal candidate will have strong expertise in designing, developing, and optimizing large-scale data pipelines, cloud data platforms, and ETL/ELT processes.
The candidate will work closely with business stakeholders, data analysts, data scientists, and engineering teams to deliver reliable, scalable, and secure data solutions that support reporting, analytics, and business decision-making.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and data processing solutions.
- Build and optimize ETL/ELT workflows for large-scale data integration.
- Work with Azure Databricks and other cloud-based data engineering technologies.
- Develop data solutions using SQL, Python, Spark, and related technologies.
- Work with structured and unstructured datasets to support analytics and reporting.
- Design and maintain data warehouses and data models.
- Ensure data quality, security, governance, and compliance.
- Collaborate with business and technical teams to understand data requirements.
- Troubleshoot performance issues and optimize data processing workflows.
- Follow cloud engineering best practices to deliver reliable and scalable solutions.
Required Qualifications
- 10+ years of overall IT experience.
- 5+ years of experience in cloud leadership roles.
- 5+ years of experience in Data Engineering or related data development roles.
- Strong experience with SQL and relational databases.
- Hands-on experience with ETL/ELT development.
- Experience with cloud platforms such as Azure, AWS, or Google Cloud.
- Strong programming skills in Python, Spark, or similar technologies.
- Experience developing and optimizing large-scale data pipelines.
- Knowledge of data warehousing, dimensional modeling, and data integration.
- Understanding of data governance, security, and compliance practices.
- Strong analytical, problem-solving, and communication skills.
Preferred Qualifications
- Experience with Azure Databricks, Azure Data Factory, Azure Synapse, or Snowflake.
- Experience with Hadoop or other big data technologies.
- Healthcare or healthcare payer industry experience.
- Experience supporting Medicare, Medicaid, or healthcare analytics projects.
- Familiarity with Power BI, Tableau, or other reporting tools.
- Exposure to machine learning and advanced analytics environments.
- Experience working in regulated healthcare environments.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer - Azure Databricks in New York, NY vacancy
- ...729598Reference Number: 25-00664Title: Lead Data EngineerLocation: NYC, NYPosted Date: 2... ...Staffing Required Skills:Proficiency in data engineering programming languages (preferably... ...(preferably AWS data lake services or Databricks, alternatively Hadoop), atleast one relational...Databricks
HAN Staffing
New York, NY5 days ago - ...Data Engineers – Global Investment Firm | New York & London Build the data platform behind... ...work. New York: four openings. One Lead Data Engineer and three mid/senior individual... ...on experience with both Snowflake and Databricks ~ Excellent Python (including PySpark...DatabricksFull timeVisa sponsorship
$30 Global Investment Firm
New York, NY3 days ago $134.5k - $265.1k
Position Summary Deloitte is seeking a Lead Data Engineer- Databricks to support the design, build, and delivery of modern data and analytics... ...• Experience with Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)• Experience designing batch...DatabricksLocal areaVisa sponsorshipDeloitte
New York, NY1 day ago$120k - $180k
...within the newly consolidated Data and Analytics (D&A)... ...infrastructure, data governance, data engineering, data modeling, data... ...intelligence, data science, and AI.The Lead Data Engineer is crucial to... ...utilizing Big Data and Azure/Databricks for Eligibility Program data...DatabricksFull timeTemporary workWork at officeLocal areaRelocation package3 days per weekMetropolitan Life Insurance Company
New York, NY2 days ago- ...A global financial institution is seeking a Lead Data Platform Engineer to drive adoption and enablement for a strategic enterprise cloud data... ...projects. ~ Advanced SQL experience. ~ Experience with Databricks, Snowflake, or comparable modern enterprise data...DatabricksFull timeImmediate start
Madison-Davis, LLC
New York, NY3 days ago - ...SummaryWe are seeking a skilled Databricks Engineer with minimum of 5+ years of... ..., and optimize scalable data platforms and analytics solutions... ...cloud technologies preferably Azure, and big data processing to... ...communication skills.Ability to lead technical initiatives and...Databricks
EXL Service
New York, NY4 days ago - ...Title: Azure Solution Architect - Data and AI Location: New York, NY (Hybrid)... ...Partner with Sales, Technical Leads, Delivery Managers, and client... ...Synapse Analytics, Azure Databricks, Spark, Python, GitHub,... ...ways of working, and modern engineering standards. Commercial...DatabricksWork at office
Galent
New York, NY4 days ago - ...Role Overview The Lead Data Engineer will own the design, development, and maintenance of robust data architecture and pipelines leveraging cloud-based technologies, supporting strategic analytics, AI/ML initiatives, and data governance initiatives across the organization...Full time
The Phoenix Group
New York, NY3 days ago - ...The Lead Data Governance & Catalog Engineer Locations: Local: from USA About the job you’re considering: The Lead Data Governance &... ...Integrate metadata across data platforms including Snowflake, Databricks, ERP, CRM, and analytical systems. Support data...DatabricksLocal area
$74 per hour
DescriptionKforce's client in NYC is seeking a highly skilled Data Engineer with strong hands-on experience in Apache Spark/PySpark, AWS EMR, Databricks, and Snowflake. The Data Engineer will be responsible for designing and building scalable, high-volume data pipelines...DatabricksKForce
New York, NY9 hours ago- ...Data Engineers - this is one worth paying attention to. I’m working on a Data Engineering... ...from the ground up. The big focus? Databricks. You’ll take real ownership of the... ...collaboration skills Snowflake SQL Server Azure Python / SQL Data pipelines &...DatabricksFull time
Saragossa
New York, NY2 days ago - Livestock Information Ltd. is seeking a Data Development & Support Analyst to enhance our data platform and reporting services. You will... ...pipelines run efficiently and data is accurately delivered into Azure Data Lake, Data Warehouse, and Power BI environments, while...DatabricksRemote job
- ...We are actively hiring Fulltime AWS, Databricks, GCP, and Snowflake Data Engineers and AI Engineers for hybrid roles (3 days per week in-office) based in New York City, NY, or New Jersey. Candidates must meet the following qualifications: Note: This position requires...DatabricksFull timeInternshipWork at office3 days per week
Capgemini
New York, NY2 days ago - ...and talents will be valued and celebrated. Together we will create a brighter future and make a meaningful difference. As a Lead Data Engineer at JPMorganChase within the Commercial & Investment Bank Operational Resiliency team, you are an integral part of an agile team...
JP Morgan Chase
New York, NY3 days ago $125k - $136k
...possible. Join us and help the world’s leading organizations unlock the value of technology... ...from office). Mandatory Skills: Data Engineering, Database Engineering, Stakeholder... ...Preferred Certifications: SnowPro, Databricks The base compensation range for this...DatabricksPermanent employmentFull timeWork at officeLocal area3 days per week- ...A leading global investment manager is expanding its data engineering function to support investment, research, and operational teams across the firm. The data team... ..., and maintain scalable data pipelines using Databricks and Apache Spark to process and transform large...DatabricksFull time
Mondrian Alpha
New York, NY3 days ago - ...Senior Python Engineer for Data Platform Onsite in New York City I'm hiring a hands-on data engineer for a well-known, mission... ...how you work. Nice to have Snowflake, Fabric, Databricks, Airflow, dbt, Azure Data Factory, or a legacy migration under your belt....DatabricksFull time
Gambit Technologies
New York, NY4 days ago $100k - $150k
...New York, United States Permanent Data Catalog EngineerNew York, NY - Hybrid (3... ...) My client is seeking a Data Catalog Engineer to join a growing Data Governance team focused... ...catalog platforms like Atlan, Collibra, Databricks Unity Catalog, Alation, Informatica or...DatabricksPermanent employmentWork at office3 days per weekApollo Solutions
New York, NY4 days ago$137.7k - $229.5k
Position Summary Our Deloitte AI & Engineering team works to transform technology platforms, drive innovation, and help make a significant... ..., and fuel growth through innovation.Work You’ll DoAs a Lead Data Engineer II on the team, you will be responsible for:Support...Local areaDeloitte
New York, NY9 hours ago- ...Senior Data Engineer The ideal candidate is a hands-on engineer with strong experience across Snowflake, dbt, Apache Airflow, SQL... .../ data warehouses . Exposure to platforms such as Databricks, AWS, Azure, Power BI, Sigma, iLevel, or similar financial/data technologies...DatabricksFull time
The Phoenix Group
New York, NY4 days ago - To support a federal client, the full-time remote Azure Data / Databricks Engineer will design and implement secure data pipelines, modernize legacy Oracle workloads, and collaborate with business teams on data integration and reporting. Key Responsibilities Design and...DatabricksFull timeRemote work
$45 per hour
...Summer 2027 Data Engineer Intern – Alternative Data Location: New York... ...uncovering new insights for leading financial institutions and... ...cloud-based technologies (AWS, Databricks, Snowflake) to process... ...cloud computing platforms (AWS, Azure, or GCP). Strong...DatabricksHourly payPart timeSummer workInternshipSummer internshipWork at officeM Science
New York, NY4 days ago- Lumen Resources is seeking an Azure Databricks Platform Engineer to own the administration, governance, security, and operational health of an enterprise... ..., production jobs, and cost-efficient operations across data, analytics, and AI workloads. The position emphasizes...Databricks
- ...EXL is seeking a Senior Data Engineer to join our Data & Analytics practice... ...engineering practices, and leading teams delivering high-impact... ...working with Snowflake, Databricks, or similar cloud-native data... ...ecosystems including AWS, Azure, or GCP. ~ Strong knowledge...DatabricksFull timeLocal areaRelocation
EXL
New York, NY1 day ago - ...design and implementation requiredAWS data engineering (Glue, Lambda, EventBridge Cradle, S3)... ...preferredFormer Informatica experience, Databricks, SnowflakeNice to Haves:Apache AirflowSomeone... ...and operational reviews with industry-leading engineers.We bring a unique vibe and...DatabricksWork at office
Ampcus
New York, NY8 hours ago - HSO Group B.V. is seeking a Senior Data Architect to lead data modernization initiatives on Azure, including Synapse, Databricks, and Delta Lake. You will design scalable data... ...enterprise data platforms. You will mentor engineers, collaborate with cross-functional teams,...Databricks
$100 - $130 per hour
A financial firm is looking for a Data Engineer - Databricks to join their team. This role is remote. Pay: $100-130/hrUS Citizens or GC Holders OnlyNo c2c - W2 OnlyQualifications:A Bachelor's Degree in Computer Science, Management Information Systems, Computer Engineering...DatabricksRemote workOpen Systems Technologies
New York, NY4 days ago- RDQ427R371At Databricks, we are passionate about enabling data teams to solve the world’s toughest problems — from making... ...their business. Founded by engineers — and customer obsessed — we leap... ...Product/Engineering and Microsoft Azure. You will be the primary technical...DatabricksWorldwide
DataBricks
New York, NY4 days ago $145k - $190k
...Data Engineering Lead Architect – Azure Fractal is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in... ...~ Deep experience with Azure data services, including Databricks, Azure Data Lake Storage, Azure Data Factory, Event Hubs...DatabricksHourly payFull timeLocal areaWorldwideFractal
New York, NY2 days ago- ...build the foundational data infrastructure that... ...simple hypotheses into engineered features, manage secure... ...while collaborating with leading AI and data experts in... ...used data platforms (Databricks, Snowflake, BigQuery,... ...cloud platforms (AWS, Azure, GCP)Exceptional time...DatabricksApprenticeshipWork at officeLocal areaEasy work
McKinsey & Company
New York, NY2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer - Azure Databricks. Be the first to apply!
Related searches
- azure specialist New York, NY
- azure data engineer New York, NY
- cloud engineer azure New York, NY
- microsoft azure architect New York, NY
- azure infrastructure engineer New York, NY
- devops engineer azure New York, NY
- azure developer New York, NY
- azure solution architect New York, NY
- oncology data New York, NY
- data officer New York, NY