Lead Data Engineer
Diverse Lynx
Lead Data Engineer
We are seeking a highly skilled and strategic Lead Data Engineer with strong expertise in PySpark Apache Iceberg, Trino, and modern Data Lakehouse architectures. The ideal candidate will be responsible for designing and driving enterprise-scale data platforms that enable analytics, AI/ML, and business intelligence across global organizations. This is a strategic customer-facing role requiring strong technical leadership, architecture expertise, stakeholder management, and the ability to influence data transformation initiatives.
Key Responsibilities
- Lead the architecture, design, and implementation of large-scale distributed data platforms.
- Build and optimize high-performance data pipelines using PySpark and distributed computing frameworks.
- Design and manage Data Lakehouse solutions using Apache Iceberg for schema evolution, time travel, partition optimization, and data governance.
- Architect and optimize federated query solutions using Trino across multiple data sources.
- Drive enterprise data migration and modernization initiatives from traditional warehouses to Lakehouse architectures.
- Partner with business stakeholders, product owners, architects, and customer teams to translate business requirements into scalable technical solutions.
- Establish best practices for data modelling, performance tuning, security, governance, and observability.
- Mentor a team of data engineers and provide technical leadership across delivery streams.
- Evaluate and incorporate emerging technologies in Data Engineering, Analytics, and AI.
- Support pre-sales discussions, solution proposals, estimations, and customer presentations.
Technical Skills
- 10+ years of experience in Data Engineering and Big Data ecosystems.
- Expert knowledge of PySpark and Spark SQL.
- Strong hands-on experience with Apache Iceberg.
- Strong experience with Trino (Presto) query engine.
- Experience building large-scale batch and near-real-time pipelines.
- Strong SQL skills and query optimization expertise.
- Experience with Data Lake technologies and cloud-based analytics platforms.
- Knowledge of data modelling and distributed storage concepts.
- Experience with orchestration tools such as Airflow or equivalent.
- Experience working with file formats such as Parquet, ORC, and Avro.
- Exposure to CI/CD, Git, DevOps, and Infrastructure as Code practices.
Experience in one or more cloud platforms:
- AWS (EMR, Glue, S3, Athena, Lake Formation)
- Azure (Databricks, Data Factory, ADLS)
- GCP (Dataproc, BigQuery, GCS)
Leadership & Strategic Expectations
- Ability to engage with senior customer stakeholders.
- Drive technical roadmaps and platform modernization strategies.
- Lead architecture reviews and governance forums.
- Identify opportunities for automation, optimization, and AI-driven solutions.
- Strong communication and presentation skills.
- Ability to influence decisions across engineering, product, and business teams.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in Austin, TX vacancy
- About GlobalFoundries:GlobalFoundries is a leading full-service semiconductor foundry providing a... ...more information, visit . Summary of Role: The Data Solutions Group is responsible for integrating manufacturing and engineering data out of a high variety of source systems...SuggestedFull timeLocal area
$159.1k - $265.1k
Position Summary Our Deloitte AI & Engineering team works to transform technology platforms, drive innovation, and help make... ...ventures, and fuel growth through innovation. Work You’ll Do As a Lead AI and Data Science Engineer II on the team, you will be responsible for...SuggestedLocal area- A leading educational institution is seeking a Lead Data Engineer who will design scalable data pipelines and lead a team to improve overall university outcomes. The role requires strong knowledge of cloud platforms, particularly AWS and Databricks, and excellent project...SuggestedRemote jobFlexible hours
$125k - $143.71k
Lead Data Engineer Enterprise Technology - Data to Insights (D2I) All Applicants Weekly Scheduled Hours: 40 FLSA Status: Exempt Earliest Start Date: Immediately Position Duration: Expected to Continue Until Dec 19, 2026 Location: Texas Job Details This is a fixed...SuggestedFull timeFor contractorsFixed term contractWork at officeImmediate startRemote workMonday to FridayFlexible hoursShift workNight shift- Overview We are looking for a skilled and experienced Lead Snowflake Data Engineer to join a top healthcare client in Austin, TX. The Snowflake Engineer will architect, design, and manage data solutions across a modern cloud ecosystem, focusing on integrating Azure Data...SuggestedFull timeWork experience placement
$119.07k - $198.38k
...shape your success while helping clients pursue their financial goals.Job OverviewLPL Financial is seeking an AVP, Data Services - Reference Data Engineering Lead to join our Data & Analytics organization and support the Reference Data Master team. This role is responsible...Full timeWork experience placement- Katalyst CRO in Austin, Texas is looking for a Database Configuration Lead to design and build clinical study databases using EDC/CDM platforms. The ideal candidate will have strong knowledge of clinical trial processes, hands-on experience with Medidata Rave, Oracle Clinical...
- A leading healthcare consulting firm is seeking a skilled Lead Snowflake Data Engineer in Austin, TX. The role involves architecting and managing data solutions, leading a team of data engineers, and ensuring seamless integration of Snowflake within a cloud ecosystem....
$95k - $150k
...you an experienced, passionate pioneer in technology who wants to work in a collaborative environment? As an experienced Senior Data Engineer you will have the ability to share new ideas and collaborate on projects as a consultant without the extensive demands of travel...Permanent employment- ...streamers around the world while gaining meaningful experience across a variety of disciplines.About the teamThe mission of Roku's Data Engineering team is to develop a world-class big data platform that enables internal and external customers to leverage data to grow their...Work at officeLocal areaRemote workWorldwideMonday to ThursdayFlexible hours
$128k - $252.5k
...clients' businesses. We are hiring a Senior Data Scientist to collaborate directly with... ...team of interdisciplinary data scientists, engineers, architects, and consultants, our work... ...sustain professional relationshipsAbility to lead projects or workstreamsAbility to manage...Local area$95.2k - $242k
...that power infrastructure, solutions, and data management our customers need most. Our... ...assumptions and reimagining how work gets done. Engineers define intent, author precise... ...Mechanical EngineerMechanical Engineering leads and delivers the development of innovative...Relocation package$134.5k - $265.1k
...Summary At Deloitte, Forward Deployed Engineers (FDE) don’t just build AI solutions, they... ...into enterprise-scale impact, pairing leading class engineering with pod-based delivery... .../30/2026.Work you’ll doAs a Microsoft AI&Data FDE, you will work side by side with...Local area$91k - $169k
...mission to bring happiness to life through the power of play! As a leading entertainment and content-building company, we are powered by... ...environment where your contributions matter. Joining us as a Data Engineer, you'll play a pivotal role in driving our engineering team's...Full timeWorldwideVisa sponsorshipWork visa- ...partner helping organizations modernize data ecosystems, build real-time analytics capabilities... ...and Gemini Enterprise.You AreA hands-on Engineer with foundational experience in Data... ..., blockers, and insights to project leads and client stakeholders.4. Data Governance...Full timeWork experience placementLive inWork at officeLocal area
$144k - $187k
...special place to work. If you’re driven to align your career with a fantastic mission, look no further!As a Senior Data Engineer at SpyCloud, you will lead the architectural transformation of our data systems, working closely with Product Management, SpyCloud Labs, and...Full timeTemporary workLocal areaRemote workWorldwideVisa sponsorshipFlexible hours$132.1k - $178.8k
...including public sector. The AWS Global Support team interacts with leading companies and believes that world-class support is critical to... ...top of AWS services.The AWS Marketplace team is looking for a Data Engineer to play a key role in building insights from our large and...Flexible hours- ...paced environment and accept the challenge to lead with purpose. If you’re looking to build... ...Analytics platform from scratch on a modern data stack (GCP/BigQuery, CData, dbt). We are looking for a Senior Analytics Engineer to lead the architecture, implementation, and...
$63 - $68 per hour
...| USD 63-68 / Hourly (C2C)Employment Type: Full TimeClient: TechMContact: Meghana GorusuCompany: SRI Tech SolutionsJob Title: Data Engineering & AnalyticsLocation: Cupertino, CA (Hybrid) / Austin, TX / Seattle, WAAbout the Role:We build data systems that power innovation...Hourly payFull time$100.79k - $160.26k
...DLA Piper is a place you can engage in meaningful work and grow your career. Let’s see what we can achieve. Together.SummaryThe Data Engineer, Solutions & Data role designs, builds, and operates data pipelines and data integration processes that translate raw data into...Full timeWork at officeRemote workRelocationVisa sponsorshipRelocation package- ...solutions in client context.•Experience of BI, Reporting and Data Warehousing.Ability to play a Techno Functional role.•... ...with huge volumes of data would be an advantage.•Expected to lead offshore data engineers.Preferred Skill and Experience•Experience in Snowflake & PL...Full timeTemporary workWork experience placementRelocation
$132.1k - $178.8k
Join the OIS/CXI Analytics team to build strategic data infrastructure powering Amazon's Operations Technology ecosystem. Our team provides... ...Fresh, Prime Now, Lockers, Pantry, and Amazon Campus. As a Data Engineer, you will build and maintain scalable data pipelines and ML-...Full timeTemporary workSeasonal workWorldwideFlexible hours$133k - $188.6k
...delivery of manufacturing and enterprise data.Build reusable Bronze, Silver, and Gold data... ...workflows for data preparation, feature engineering, model training, evaluation, deployment,... ...and we embrace the responsibility to lead the change that will make our world better...Full timeH1bLocal areaWork from homeRelocation packageFlexible hours- ...Austin, TXEmployment Type: Full TimeIndustry: Computer SoftwareClient: WiproContact: Meghana GorusuCompany: SRI Tech SolutionsJob Title: Data EngineerLocation: Austin, TXJob Description: Panel is available daily 2-4 PM cSTSubmit with skill...
- ...loyalty and back-office software solutions as well as industry-leading hardware and drive-thru offerings. To learn more, visit... ..., and Instagram.Position Description: We are seeking a Senior Data Engineer to join our team and help us build the next generation of data...Remote workWorldwide
- ...GorusuCompany: SRI Tech SolutionsRole: Applied Data Scientist/ Data Science EngineerLocation:... ...productsCollaborate with software engineers, business stakeholders and product owners... ...technical internal stakeholdersAbility to lead a team of data scientists is a plusExperience...Full timeLocal areaRelocation3 days per week
- Austin, TXDevelopment - Engineering /Full-time /HybridWho We AreArrive Logistics is a leading transportation and technology company in North America with plans to grow... ...environment.Who We WantAs a Senior Data Engineer at Arrive Logistics, you will build and...Full timeWork experience placementCasual workWork at officeLocal areaRelocationRelocation packageDay shift
- ...our ongoing commitment to democratize access to investing and financial planning.We are looking for an energetic and accomplished Data Engineer (individual contributor) with a strong technology and solution delivery acumen who will be responsible for the successful design...Full timeFor contractorsWork at office
- ...solutions in client context.•Knowledge of BI, Reporting and Data Warehousing.•Ability to play a Techno Functional role.•... ...with huge volumes of data would be an advantage.•Expected to lead offshore data engineers.Preferred Skill and Experience•Experience in GEN AI•Experience...Full timeTemporary workWork experience placementRelocation
- ...help our customers get the high-quality gear they need to make the most of their adventures. We are BUILT FOR THE WILD.The Senior Data Engineer will be part of a team that designs, builds, and operates YETI’s enterprise data platform with a strong emphasis on Azure...Full timeH1bLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
