Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Lead Data Engineer

Algoworks

Lead Data Engineer

Location: India, Remote

Experience: 10+ Years

About the company: Algoworks is an award-winning artificial intelligence, engineering services and experience transformation firm with offices across the United States, Europe, South America and India. We bring together a global team of engineers, architects, designers, researchers and operators united by rigor, accountability and a commitment to delivering measurable results. For over 20 years, Algoworks has partnered with Fortune 500 organizations across the Americas, Europe and Asia to define, build and run technology that drives meaningful business outcomes. Our work combines human-centered design, engineering excellence and AI-powered capabilities to solve complex challenges with clarity and precision. Innovation, particularly in the responsible application of AI, is embedded in how teams approach problem-solving and continuous improvement. At Algoworks, growth is continuous and closely tied to impact. Teams collaborate across geographies and disciplines, strengthening outcomes through shared insight and collective expertise. The culture values transparency, open dialogue and an environment where every voice is heard and contribution is recognized. Through collaboration, accountability and a focus on results, Algoworks operates at the intersection of technology and people, building not only advanced systems but strong global teams that elevate performance and create lasting impact.

Role Overview

We are seeking a hands-on Senior Data Engineer with strong expertise in Azure Databricks and Azure Data Factory to build, optimize, and maintain scalable enterprise data pipelines. The role will focus on high-performance ETL/ELT development, Delta Lake optimization, data processing across Bronze, Silver, and curated layers, and close collaboration with DWH and reporting teams for downstream consumption. The ideal candidate will act as a senior technical contributor, ensuring reliability, performance, data quality, and maintainability across the data platform.

Key Responsibilities:

1. Pipeline Development

  • Build and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
  • Implement ingestion and transformation logic across Bronze and Silver data layers.
  • Develop batch and incremental data-processing patterns.
  • Design reliable and reusable pipeline components for enterprise workloads.
  • Monitor and troubleshoot pipeline execution and data-processing issues.

2. Curated Layer & Delta Lake Development

  • Implement hydration, merge, and upsert logic using Delta Lake.
  • Build and maintain curated datasets aligned with data quality and business requirements.
  • Handle late-arriving data and incremental updates.
  • Implement reliable data transformation and reconciliation processes.
  • Ensure curated datasets are optimized for downstream consumption.

3. Performance & Storage Optimization

  • Optimize Delta Lake tables for performance and cost efficiency.
  • Select and tune appropriate storage formats such as Parquet and Delta.
  • Apply partitioning, compaction, and file-sizing strategies.
  • Tune Spark jobs for large-scale distributed data processing.
  • Identify and resolve performance bottlenecks across data pipelines and storage layers.

4. Downstream & DWH Collaboration

  • Work closely with DWH and reporting teams to support downstream data consumption.
  • Provide optimized datasets for reporting and analytical workloads.
  • Support data validation and reconciliation with Gold-layer outputs.
  • Collaborate with downstream teams to understand data requirements and optimize delivery.
  • Ensure consistency and reliability of data consumed by reporting and analytics platforms.

5. Engineering Best Practices

  • Implement basic CI/CD practices for data pipelines.
  • Follow coding standards, documentation, and version-control practices.
  • Maintain reusable, scalable, and maintainable pipeline code.
  • Support production troubleshooting and performance tuning.
  • Participate in Agile delivery processes and technical discussions.

6. Data Quality & Production Support

  • Implement data validation and quality checks across ingestion and transformation processes.
  • Investigate data discrepancies and pipeline failures.
  • Perform root-cause analysis and implement corrective actions.
  • Support production deployments and resolve data-processing issues.
  • Maintain reliability and consistency across enterprise data pipelines.
Required Technical Skills and Competencies:
  • Strong hands-on experience in data engineering and enterprise data platforms.
  • Strong experience building data pipelines on Azure.
  • Advanced proficiency in PySpark.
  • Hands-on experience with Azure Databricks.
  • Strong experience with Azure Data Factory.
  • Deep knowledge of Delta Lake tuning and optimization.
  • Strong understanding of storage optimization using Parquet and Delta.
  • Strong SQL skills for data transformation, validation, and reconciliation.
  • Experience working with large datasets and distributed processing.
  • Experience implementing batch and incremental processing patterns.
  • Experience with hydration, merge, and upsert logic.
  • Experience with Git and basic CI/CD pipelines.
  • Familiarity with data quality and validation techniques.
  • Experience working in Agile delivery environments.
Must Have Skills:
  • Azure Databricks.
  • Azure Data Factory.
  • PySpark.
  • Delta Lake.
  • SQL.
  • Data Engineering.
  • ETL/ELT.
  • Data Pipeline Development.
  • Bronze/Silver/Curated Data Layers.
  • Delta Lake Performance Optimization.
  • Spark Performance Tuning.
  • Parquet.
  • Batch and Incremental Processing.
  • Git and Version Control.
  • Strong analytical and problem-solving skills.
Good to Have Skills:
  • Microsoft Fabric.
  • Streaming or near real-time data pipelines.
  • Data governance tools.
  • Metadata management tools.
  • Advanced CI/CD practices.
  • Experience with Gold-layer development.
  • Experience supporting enterprise reporting and DWH platforms.
Desired Attributes:
  • Strong analytical and problem-solving capabilities.
  • Ability to independently design and develop complex data pipelines.
  • Strong focus on performance, scalability, reliability, and data quality.
  • Ability to troubleshoot complex production data issues.
  • Strong attention to detail and coding discipline.
  • Good communication and collaboration skills.
  • Ability to work effectively with DWH, reporting, and cross-functional teams.
  • Proactive approach to performance optimization and continuous improvement.
  • Strong ownership of data engineering deliverables.
Interview Process

2-3 rounds of discussion.

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in United States vacancy
  •  ...Summary : Lead the design and delivery of scalable data engineering solutions on AWS, supporting enterprise data onboarding and domain-oriented data products. The role will provide hands-on technical leadership across AWS-native data services and work with Prudential... 
    Suggested

    Exavalu

    Newark, NJ
    22 days ago
  •  ...Only Candidates with US Work Authorization will be considered. Position Overview: The Lead Data Engineer turns business and domain needs into secure, observable, reusable, and documented pipelines. The role combines hands-on technical leadership, portfolio delivery... 
    Suggested
    Local area
    Shift work

    Bradesco Bank

    Coral Gables, FL
    29 days ago
  •  ...Job Title: Lead Data Engineer Location: Grand Rapid, MI Work Mode: Hybrid Job Type: Contract / Full-Time Certification: Databricks Certification preferred Job Summary We are seeking an experienced Lead Data Engineer to design and implement scalable... 
    Suggested
    Full time
    Contract work

    Celebal Technologies

    Grand Rapids, MI
    4 days ago
  •  ...Providing senior engineering leadership across two strategic teams, the full-time remote Lead Data Engineer will set technical direction, ensure alignment between data and AI strategies, and oversee the delivery of secure, scalable risk products. Key Responsibilities... 
    Suggested
    Full time
    Remote work

    Virtual Vocations Inc

    United States
    1 day ago
  •  ...Straive is a global leader in enterprise-grade Data Analytics and AI solutions, committed to empowering businesses across various...  ...partnership with EQT. Website: LinkedIn Job Tittle- Lead Data Engineer Location - Newark,NJ (Hybrid) Employment Type- Fulltime... 
    Suggested
    Full time
    Worldwide

    Straive

    Newark, NJ
    26 days ago
  •  ...We are seeking a Lead Data Engineer to own the design, architecture, and delivery of scalable data platforms, AI‑powered data products, and enterprise‑grade data services. This role goes beyond hands‑on engineering and includes technical leadership, architectural decision... 
    Full time

    Robert Half

    Chicago, IL
    9 hours ago
  •  ...We’re hiring: Senior/Lead Big Data Engineer We’re looking for a senior- or lead-level Big Data Engineer who can take on complex technical challenges and communicate solutions clearly. This is a hybrid role based in Rockville, MD, or Tysons Corner, VA . Must... 

    Unisys

    Tysons Corner, VA
    a month ago
  •  ...Lead Data Engineer (Azure Databricks) Charlotte, NC/ New York 12 Months Contract Required Skills: Experience using Python, PySpark and its frameworks. Well versed with Azure Databricks Proven experience as a Data Engineer with experience in Azure cloud... 
    Contract work

    Euclid Innovations

    Charlotte, NC
    16 days ago
  •  ...ARE HIRING | Technical Project Manager – Data Migration (Databricks) Location: Grand...  ...experienced Technical Project Manager to lead a large-scale legacy data migration and...  ...will have a strong understanding of Data Engineering, Databricks, and enterprise data migration... 
    Long term contract

    Celebal Technologies

    Grand Rapids, MI
    16 days ago
  • $110k - $120k

     ...creativity and innovative technology to deliver industry-leading digital solutions. Synechron’s progressive technologies and...  ...Intelligence, Consulting, Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of noteworthy financial services and... 
    Temporary work
    Flexible hours

    Synechron

    Charlotte, NC
    13 days ago
  •  ...Position Summary The Lead Data Engineer is responsible for defining, implementing, and maintaining the organization’s data architecture, platforms, data management practices, and reporting capabilities. This role serves as the technical owner of the organization’s data... 

    MTech

    Jupiter, FL
    13 days ago
  •  ...Title : Senior Data Engineer – Engineering Analytics & Data Visualization Location : DC Metro Area, NYC Metro Area, Jersey CIty Target Start Date : ASAP Type : Contract Pay Rate: DOE We are seeking a hands-on Senior... 
    Contract work
    Immediate start

    ConsultNet Technology Services and Solutions

    Rockville, MD
    1 day ago
  • $77.6k - $176k

     ...Job Number: R0254004 Data Engineer, Lead The Opportunity: Make enterprise data and knowledge usable, governed, and dependable for AI applications. You will design data architectures and build the pipelines, knowledge indexes, and retrieval services that ground... 
    Full time
    Contract work
    Part time
    Local area
    Remote work

    Booz Allen Hamilton

    McLean, VA
    20 minutes ago
  •  ...Job Description: Lead Data Engineer (GCP Dataproc & BigQuery) Location: Edison, NJ (Onsite) Employment Type: Full-Time Experience Required: 10+ Years Position Overview We are seeking a highly skilled Lead Data Engineer with deep expertise in Google... 
    Full time

    Tredence Inc.

    Edison, NJ
    4 days ago
  •  ...Job Description: Lead Data Engineer – Agentic AI & Data Intelligence Location: Onsite in Phoenix, AZ, USA Job Summary We are seeking an experienced Lead Data Engineer to lead the strategy, development, evaluation, and delivery of next-generation AI products focused... 

    iShare Inc.

    Phoenix, AZ
    8 days ago
  •  ...Programming & Data Engineering Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-... 
    Contract work

    Altimetrik

    Cary, NC
    4 days ago
  •  ...Grey Parrot Systems is hiring a Lead Data Migration Engineer to own a complex Microsoft Dynamics NAV to Salesforce data migration — end-to-end, from data discovery through production cutover. About Grey Parrot Systems Grey Parrot Systems is a women-owned, BEP-... 
    Contract work
    Immediate start
    Remote work

    Grey Parrot Systems LLC

    United States
    1 day ago
  • $188.25k - $230.08k

     ...Job Description: Lead Data Engineer Collaborate with Innovative 3Mers Around the World Choosing where to start and grow your career has a major impact on your professional and personal life, so it’s equally important you know that the company that you choose to... 
    Full time
    H1b
    Remote work
    Relocation
    Flexible hours

    3M

    Minnesota
    16 days ago
  • $140k - $160k

     ...Lead Data Engineer We are seeking an experienced Lead Data Engineer to join our healthcare technology team with a primary focus on healthcare interoperability. This role will be instrumental in designing, building, and maintaining our Azure-based data infrastructure... 
    Full time
    Contract work
    Work experience placement

    Verisma

    Remote
    11 days ago
  •  ...position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Engineer, Databricks based in the United States. The Lead Data Engineer will drive the delivery of production-grade data engineering... 
    Full time
    Temporary work
    Remote work

    Jobgether

    Remote
    4 days ago
  • $106.61k - $284.28k

     ...a time. POSITION SUMMARY At CVS Health, protecting customer, patient, member, and colleague data is fundamental to our mission. The Lead Data Privacy Engineer will play a critical role in advancing enterprise privacy engineering and data protection capabilities... 
    Hourly pay
    Full time
    Temporary work
    Work experience placement
    Local area

    CVS Health

    Illinois
    18 days ago
  • $110k - $145k

     ...by bringing quality software to the federal space. This position is contingent upon contract award. We’re looking for a Lead Data Engineer to join our VA team, setting the technical approach for moving legacy data warehouse data and pipelines onto an Azure and Databricks... 
    Full time
    Contract work
    For contractors
    Remote work
    Flexible hours

    Oddball

    Remote
    14 days ago
  •  ...Databricks certifications across 325+ customers and been named Databricks' 2026 Partner of the Year (C&SI Emerging). As a Lead Data Engineer, you will lead the build of production Lakehouse pipelines on customer engagements, working directly with customers and often... 
    Full time
    Temporary work
    Remote work

    V4C.ai

    Remote
    7 days ago
  • $180k - $200k

     ...headquartered in New York City with offices around the world. To learn more, go to ROLE OVERVIEW Zeta Global is seeking a Lead Data Engineer to build and scale healthcare-focused applications and data systems that power audience intelligence, activation, measurement... 
    Full time

    Zeta Global

    Remote
    21 days ago
  • $60 per hour

     ...Lead Data Engineer Providence, Rhode Island Onsite $60/hr 2 Roles MUST : ETL (Extract, Transform, Load) using Matillion and DBT ETL tool Required Skills Minimum 12+ years of experience in cloud native data engineering,... 

    United IT Solutions

    Providence, RI
    1 day ago
  •  ...Lead Data Engineer ~ Bachelor's degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience. ~10+ years of experience in software engineering with significant focus on data engineering and distributed systems. ~ Strong... 

    The Judge Group

    Columbus, OH
    1 day ago
  •  ...Functional Job Title: Lead Data Engineer Location (City State, Street Name): Charlotte NC 28273 5 days onsite 4 positions SUBMIT ONLY IF THEY ARE LOOKING FOR BOTH CONTRACT/FULL TIME. DO NOT SUBMIT IF THEY WANT CONTRACT ONLY They must have... 
    Full time
    Contract work

    Anveta

    Charlotte, NC
    1 day ago
  • $125k - $207k

     ...Lead Data Engineer As a Lead Data Engineer, you will design and implement solutions to ingest, store, transform, and distribute big data on Azure Cloud. Our developers use Databricks (Spark, SQL, PySpark), Databricks Workflows, Python, JSON, and SQL to build products... 

    84.51°

    Cincinnati, OH
    5 days ago
  • $220k - $260k

     ...Overview The Lead Data Engineer on the Nebula team plays a significant technical leadership role in shaping and scaling the data foundation that powers analytics, reporting, AI development, and operational decision-making across the organization. This role combines... 
    Full time
    Local area
    Remote work
    Flexible hours

    Lakeview Loan Servicing

    Remote
    9 days ago
  • $147.25k - $190k

     ...As a Lead Data Engineer (Forward Deployed) at JPMorganChase within the Infrastructure Data Platforms team, you will embed with infrastructure product teams to turn how asset and configuration data exists today into trusted, governed data products that power security controls... 
    Contract work

    JPMorgan Chase Bank, N.A.

    Columbus, OH
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!