Lead Data Engineer
Algoworks
Lead Data Engineer
Location: India, Remote
Experience: 10+ Years
About the company: Algoworks is an award-winning artificial intelligence, engineering services and experience transformation firm with offices across the United States, Europe, South America and India. We bring together a global team of engineers, architects, designers, researchers and operators united by rigor, accountability and a commitment to delivering measurable results. For over 20 years, Algoworks has partnered with Fortune 500 organizations across the Americas, Europe and Asia to define, build and run technology that drives meaningful business outcomes. Our work combines human-centered design, engineering excellence and AI-powered capabilities to solve complex challenges with clarity and precision. Innovation, particularly in the responsible application of AI, is embedded in how teams approach problem-solving and continuous improvement. At Algoworks, growth is continuous and closely tied to impact. Teams collaborate across geographies and disciplines, strengthening outcomes through shared insight and collective expertise. The culture values transparency, open dialogue and an environment where every voice is heard and contribution is recognized. Through collaboration, accountability and a focus on results, Algoworks operates at the intersection of technology and people, building not only advanced systems but strong global teams that elevate performance and create lasting impact.
Role Overview
We are seeking a hands-on Senior Data Engineer with strong expertise in Azure Databricks and Azure Data Factory to build, optimize, and maintain scalable enterprise data pipelines. The role will focus on high-performance ETL/ELT development, Delta Lake optimization, data processing across Bronze, Silver, and curated layers, and close collaboration with DWH and reporting teams for downstream consumption. The ideal candidate will act as a senior technical contributor, ensuring reliability, performance, data quality, and maintainability across the data platform.
Key Responsibilities:
1. Pipeline Development
- Build and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
- Implement ingestion and transformation logic across Bronze and Silver data layers.
- Develop batch and incremental data-processing patterns.
- Design reliable and reusable pipeline components for enterprise workloads.
- Monitor and troubleshoot pipeline execution and data-processing issues.
2. Curated Layer & Delta Lake Development
- Implement hydration, merge, and upsert logic using Delta Lake.
- Build and maintain curated datasets aligned with data quality and business requirements.
- Handle late-arriving data and incremental updates.
- Implement reliable data transformation and reconciliation processes.
- Ensure curated datasets are optimized for downstream consumption.
3. Performance & Storage Optimization
- Optimize Delta Lake tables for performance and cost efficiency.
- Select and tune appropriate storage formats such as Parquet and Delta.
- Apply partitioning, compaction, and file-sizing strategies.
- Tune Spark jobs for large-scale distributed data processing.
- Identify and resolve performance bottlenecks across data pipelines and storage layers.
4. Downstream & DWH Collaboration
- Work closely with DWH and reporting teams to support downstream data consumption.
- Provide optimized datasets for reporting and analytical workloads.
- Support data validation and reconciliation with Gold-layer outputs.
- Collaborate with downstream teams to understand data requirements and optimize delivery.
- Ensure consistency and reliability of data consumed by reporting and analytics platforms.
5. Engineering Best Practices
- Implement basic CI/CD practices for data pipelines.
- Follow coding standards, documentation, and version-control practices.
- Maintain reusable, scalable, and maintainable pipeline code.
- Support production troubleshooting and performance tuning.
- Participate in Agile delivery processes and technical discussions.
6. Data Quality & Production Support
- Implement data validation and quality checks across ingestion and transformation processes.
- Investigate data discrepancies and pipeline failures.
- Perform root-cause analysis and implement corrective actions.
- Support production deployments and resolve data-processing issues.
- Maintain reliability and consistency across enterprise data pipelines.
Required Technical Skills and Competencies:
- Strong hands-on experience in data engineering and enterprise data platforms.
- Strong experience building data pipelines on Azure.
- Advanced proficiency in PySpark.
- Hands-on experience with Azure Databricks.
- Strong experience with Azure Data Factory.
- Deep knowledge of Delta Lake tuning and optimization.
- Strong understanding of storage optimization using Parquet and Delta.
- Strong SQL skills for data transformation, validation, and reconciliation.
- Experience working with large datasets and distributed processing.
- Experience implementing batch and incremental processing patterns.
- Experience with hydration, merge, and upsert logic.
- Experience with Git and basic CI/CD pipelines.
- Familiarity with data quality and validation techniques.
- Experience working in Agile delivery environments.
Must Have Skills:
- Azure Databricks.
- Azure Data Factory.
- PySpark.
- Delta Lake.
- SQL.
- Data Engineering.
- ETL/ELT.
- Data Pipeline Development.
- Bronze/Silver/Curated Data Layers.
- Delta Lake Performance Optimization.
- Spark Performance Tuning.
- Parquet.
- Batch and Incremental Processing.
- Git and Version Control.
- Strong analytical and problem-solving skills.
Good to Have Skills:
- Microsoft Fabric.
- Streaming or near real-time data pipelines.
- Data governance tools.
- Metadata management tools.
- Advanced CI/CD practices.
- Experience with Gold-layer development.
- Experience supporting enterprise reporting and DWH platforms.
Desired Attributes:
- Strong analytical and problem-solving capabilities.
- Ability to independently design and develop complex data pipelines.
- Strong focus on performance, scalability, reliability, and data quality.
- Ability to troubleshoot complex production data issues.
- Strong attention to detail and coding discipline.
- Good communication and collaboration skills.
- Ability to work effectively with DWH, reporting, and cross-functional teams.
- Proactive approach to performance optimization and continuous improvement.
- Strong ownership of data engineering deliverables.
Interview Process
2-3 rounds of discussion.
- ...Summary : Lead the design and delivery of scalable data engineering solutions on AWS, supporting enterprise data onboarding and domain-oriented data products. The role will provide hands-on technical leadership across AWS-native data services and work with Prudential...Suggested
- ...Only Candidates with US Work Authorization will be considered. Position Overview: The Lead Data Engineer turns business and domain needs into secure, observable, reusable, and documented pipelines. The role combines hands-on technical leadership, portfolio delivery...SuggestedLocal areaShift work
- ...Job Title: Lead Data Engineer Location: Grand Rapid, MI Work Mode: Hybrid Job Type: Contract / Full-Time Certification: Databricks Certification preferred Job Summary We are seeking an experienced Lead Data Engineer to design and implement scalable...SuggestedFull timeContract work
- ...Providing senior engineering leadership across two strategic teams, the full-time remote Lead Data Engineer will set technical direction, ensure alignment between data and AI strategies, and oversee the delivery of secure, scalable risk products. Key Responsibilities...SuggestedFull timeRemote work
- ...Straive is a global leader in enterprise-grade Data Analytics and AI solutions, committed to empowering businesses across various... ...partnership with EQT. Website: LinkedIn Job Tittle- Lead Data Engineer Location - Newark,NJ (Hybrid) Employment Type- Fulltime...SuggestedFull timeWorldwide
- ...We are seeking a Lead Data Engineer to own the design, architecture, and delivery of scalable data platforms, AI‑powered data products, and enterprise‑grade data services. This role goes beyond hands‑on engineering and includes technical leadership, architectural decision...Full time
- ...We’re hiring: Senior/Lead Big Data Engineer We’re looking for a senior- or lead-level Big Data Engineer who can take on complex technical challenges and communicate solutions clearly. This is a hybrid role based in Rockville, MD, or Tysons Corner, VA . Must...
- ...Lead Data Engineer (Azure Databricks) Charlotte, NC/ New York 12 Months Contract Required Skills: Experience using Python, PySpark and its frameworks. Well versed with Azure Databricks Proven experience as a Data Engineer with experience in Azure cloud...Contract work
- ...ARE HIRING | Technical Project Manager – Data Migration (Databricks) Location: Grand... ...experienced Technical Project Manager to lead a large-scale legacy data migration and... ...will have a strong understanding of Data Engineering, Databricks, and enterprise data migration...Long term contract
$110k - $120k
...creativity and innovative technology to deliver industry-leading digital solutions. Synechron’s progressive technologies and... ...Intelligence, Consulting, Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of noteworthy financial services and...Temporary workFlexible hours- ...Position Summary The Lead Data Engineer is responsible for defining, implementing, and maintaining the organization’s data architecture, platforms, data management practices, and reporting capabilities. This role serves as the technical owner of the organization’s data...
- ...Title : Senior Data Engineer – Engineering Analytics & Data Visualization Location : DC Metro Area, NYC Metro Area, Jersey CIty Target Start Date : ASAP Type : Contract Pay Rate: DOE We are seeking a hands-on Senior...Contract workImmediate start
$77.6k - $176k
...Job Number: R0254004 Data Engineer, Lead The Opportunity: Make enterprise data and knowledge usable, governed, and dependable for AI applications. You will design data architectures and build the pipelines, knowledge indexes, and retrieval services that ground...Full timeContract workPart timeLocal areaRemote work- ...Job Description: Lead Data Engineer (GCP Dataproc & BigQuery) Location: Edison, NJ (Onsite) Employment Type: Full-Time Experience Required: 10+ Years Position Overview We are seeking a highly skilled Lead Data Engineer with deep expertise in Google...Full time
- ...Job Description: Lead Data Engineer – Agentic AI & Data Intelligence Location: Onsite in Phoenix, AZ, USA Job Summary We are seeking an experienced Lead Data Engineer to lead the strategy, development, evaluation, and delivery of next-generation AI products focused...
- ...Programming & Data Engineering Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-...Contract work
- ...Grey Parrot Systems is hiring a Lead Data Migration Engineer to own a complex Microsoft Dynamics NAV to Salesforce data migration — end-to-end, from data discovery through production cutover. About Grey Parrot Systems Grey Parrot Systems is a women-owned, BEP-...Contract workImmediate startRemote work
$188.25k - $230.08k
...Job Description: Lead Data Engineer Collaborate with Innovative 3Mers Around the World Choosing where to start and grow your career has a major impact on your professional and personal life, so it’s equally important you know that the company that you choose to...Full timeH1bRemote workRelocationFlexible hours$140k - $160k
...Lead Data Engineer We are seeking an experienced Lead Data Engineer to join our healthcare technology team with a primary focus on healthcare interoperability. This role will be instrumental in designing, building, and maintaining our Azure-based data infrastructure...Full timeContract workWork experience placement- ...position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Engineer, Databricks based in the United States. The Lead Data Engineer will drive the delivery of production-grade data engineering...Full timeTemporary workRemote work
$106.61k - $284.28k
...a time. POSITION SUMMARY At CVS Health, protecting customer, patient, member, and colleague data is fundamental to our mission. The Lead Data Privacy Engineer will play a critical role in advancing enterprise privacy engineering and data protection capabilities...Hourly payFull timeTemporary workWork experience placementLocal area$110k - $145k
...by bringing quality software to the federal space. This position is contingent upon contract award. We’re looking for a Lead Data Engineer to join our VA team, setting the technical approach for moving legacy data warehouse data and pipelines onto an Azure and Databricks...Full timeContract workFor contractorsRemote workFlexible hours- ...Databricks certifications across 325+ customers and been named Databricks' 2026 Partner of the Year (C&SI Emerging). As a Lead Data Engineer, you will lead the build of production Lakehouse pipelines on customer engagements, working directly with customers and often...Full timeTemporary workRemote work
$180k - $200k
...headquartered in New York City with offices around the world. To learn more, go to ROLE OVERVIEW Zeta Global is seeking a Lead Data Engineer to build and scale healthcare-focused applications and data systems that power audience intelligence, activation, measurement...Full time$60 per hour
...Lead Data Engineer Providence, Rhode Island Onsite $60/hr 2 Roles MUST : ETL (Extract, Transform, Load) using Matillion and DBT ETL tool Required Skills Minimum 12+ years of experience in cloud native data engineering,...- ...Lead Data Engineer ~ Bachelor's degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience. ~10+ years of experience in software engineering with significant focus on data engineering and distributed systems. ~ Strong...
- ...Functional Job Title: Lead Data Engineer Location (City State, Street Name): Charlotte NC 28273 5 days onsite 4 positions SUBMIT ONLY IF THEY ARE LOOKING FOR BOTH CONTRACT/FULL TIME. DO NOT SUBMIT IF THEY WANT CONTRACT ONLY They must have...Full timeContract work
$125k - $207k
...Lead Data Engineer As a Lead Data Engineer, you will design and implement solutions to ingest, store, transform, and distribute big data on Azure Cloud. Our developers use Databricks (Spark, SQL, PySpark), Databricks Workflows, Python, JSON, and SQL to build products...$220k - $260k
...Overview The Lead Data Engineer on the Nebula team plays a significant technical leadership role in shaping and scaling the data foundation that powers analytics, reporting, AI development, and operational decision-making across the organization. This role combines...Full timeLocal areaRemote workFlexible hours$147.25k - $190k
...As a Lead Data Engineer (Forward Deployed) at JPMorganChase within the Infrastructure Data Platforms team, you will embed with infrastructure product teams to turn how asset and configuration data exists today into trusted, governed data products that power security controls...Contract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
- data intern United States
- data collection researcher United States
- data operator United States
- data visualization United States
- data analysis part time United States
- data collection United States
- data modeling United States
- data United States
- data cabling United States
- research data coordinator United States



