Lead Data Engineer - AWS, Databricks & PySpark

ScolerTec Inc

**FULLY REMOTE**

ScolerTec is seeking Lead Data Engineer – AWS, Databricks & PySpark to design, build, and lead enterprise-scale cloud data engineering solutions.

This is a hands-on technical leadership role responsible for scalable ETL/ELT pipelines, batch and near-real-time data integration, lakehouse/data-platform implementation, cloud data services, CI/CD, data quality, governance, performance optimization, and technical direction of data engineers.

The ideal candidate has strong recent experience with AWS, Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, ETL/ELT, and technical leadership .

Key Responsibilities

Design and implement scalable cloud-based data platforms, including data lake, lakehouse, and data warehouse architectures.
Build enterprise-scale ingestion and ETL/ELT pipelines for batch and near-real-time workloads.
Develop data transformation and integration solutions using Python, SQL, Spark/PySpark, and modern cloud data technologies.
Implement Databricks/Delta Lake-based data engineering solutions and downstream curated/gold data layers.
Integrate pipelines with AWS compute, storage, messaging, and analytics services.
Lead technical design reviews, code reviews, engineering standards, and reusable data-engineering patterns.
Provide day-to-day technical guidance and mentoring to data engineers.
Implement data-quality checks, validation rules, error handling, observability, and pipeline monitoring.
Support streaming and event-driven data processing.
Implement Infrastructure as Code and CI/CD for repeatable data-platform and pipeline deployment.
Optimize pipeline performance, scalability, reliability, and cloud cost.
Support production incident resolution and root-cause analysis.
Partner with architecture, governance, migration, QA, DevSecOps, and security teams.
Maintain technical documentation, runbooks, and audit-ready engineering artifacts.

Qualifications

7+ years of specialized data-engineering experience.
Strong hands-on experience building enterprise-scale data pipelines.
Strong production experience with AWS data and compute services.
Strong hands-on Databricks experience.
Strong Apache Spark/PySpark and Delta Lake experience.
Strong Python and SQL skills.
Strong ETL/ELT and data-transformation experience.
Experience with batch and streaming/near-real-time ingestion.
Experience with data orchestration tools.
Experience with CI/CD and Infrastructure as Code.
Experience with data quality, governance, metadata, and classification.
Proven technical leadership, design review, code review, and mentoring experience.
Federal, public-sector, or regulated-environment experience preferred.

Preferred Certifications

Databricks Certified Data Engineer Professional
AWS Certified Data Engineer
AWS Certified Solutions Architect – Professional
AWS Certified Data Analytics – Specialty
CDMP

Apply

Vacancy posted 2 hours ago

Similar jobs that could be interesting for youBased on the Lead Data Engineer - AWS, Databricks & PySpark in United States vacancy

Lead Software Engineer - Data Engineer/PySpark
$61k - $101k
...such as cloud, AI/ML, or data engineering Experience leading the effective use of approved... ...-native experience with AWS, Azure, or GCP Ability... ...modern data platforms such as Databricks or Snowflake Preferred:... ...-on experience with Spark/PySpark and other big data...
Databricks
Amazon Web Service
Full time
J.P. Morgan
Houston, TX
2 days ago
Lead Software Engineer- Data Engineer/Pyspark/Databricks
...one of the world's leading financial... ...As a Lead Software Engineer at JPMorgan Chase... ...part of an agile data engineering team.... ...native experience (AWS, Azure, or GCP)... ...platforms such as Databricks or Snowflake Deep... ...experience with Spark/PySpark and other big data...
Databricks
Amazon Web Service
JPMorgan Chase & Co.
Houston, TX
3 days ago
Lead Data Engineer
...Lead Data Engineer Location: Remote Duration: Fulltime – Visa independents only or C2H Responsibilities Lead data wrangling activities utilizing Databricks, Python, pyspark, snowflake, kafka and AWS services to process, cleanse, and transform large volumes...
Databricks
Amazon Web Service
Full time
Remote work
E-Solutions
United States
1 day ago
Lead AWS Data Engineer
...Position Title: Lead AWS Data Engineer Location: St. Louis, MO Job Description •... ...scalable, cloud-native data platforms using PySpark, Python, and modern data engineering... ..., Glue, Athena, Data Pipelines, Databricks Unity catalog, DBX S3 Any experience...
Databricks
Amazon Web Service
Tech M USAAvance Consulting
Saint Louis, MO
5 days ago
Senior/Lead Data Engineer
...Data Engineers – Global Investment Firm | New York & London Build the... ...York: four openings. One Lead Data Engineer and three mid... ...experience with both Snowflake and Databricks ~ Excellent Python (including PySpark) and SQL, plus Spark, dbt and AWS ~3+ years of development...
Databricks
Amazon Web Service
Full time
Visa sponsorship
$30 Global Investment Firm
New York, NY
2 hours ago
Lead Data Engineer
...Role: Lead Data Engineer (Priority 1) Location: San Diego( Onsite only ) Job... ...across cloud platforms such as AWS, Azure, or GCP . Key Responsibilities... ...scalable data solutions using Databricks, PySpark, Python, and SQL . Perform core...
Databricks
Amazon Web Service
Full time
VBeyond
San Diego, CA
5 days ago
Lead Data Engineer
...Role: Lead Data Engineer Location: Mountain View, CA / Dallas, TX (Onsite) About the... ...Engineering, with strong hands-on Spark (PySpark/Scala), SQL, and Python. •... ...ETL/ELT pipelines on cloud platforms (AWS EMR/S3, Databricks, or equivalent); workflow orchestration...
Databricks
Amazon Web Service
United IT Solutions
Mountain View, CA
4 days ago
Lead Data Engineer
...Lead Data Engineer Location: 100% Remote (Preferred: Chicago, IL / Chicagoland Area) Client... ...and modernizing an enterprise-scale AWS + Databricks Lakehouse platform. This is a hands-on... ...Programming Python SQL PySpark Spark SQL Data Processing Frameworks...
Databricks
Amazon Web Service
Contract work
Remote work
Argyle Infotech
United States
2 days ago
Lead Data Engineer
...Role description Job Description Lead Data Engineer Who we are: - At CitiusTech... ..., and data warehouse solutions using Databricks, PySpark, Python, SQL, and cloud technologies.... ...manage ETL processing pipelines using AWS/Azure Databricks. Build complex...
Databricks
Amazon Web Service
Remote work
Worldwide
CitiusTech
Dallas, TX
5 days ago
Lead Data Engineer
...Lead Data Engineer We are seeking a Lead Data Engineer to own the complete... ...scalable ETL architecture, AWS data services, secure PHI/... ...ETL and data pipelines using PySpark, Python, advanced SQL, and AWS... ..., plus streaming, Databricks, Snowflake, observability, and...
Databricks
Amazon Web Service
Remote work
Thermo Fisher
United States
5 days ago
Lead Data Engineer
...Commerce Orchestration Engine helping enterprise retailers... ...We are hiring a Lead Data Engineer to drive delivery... ...Work across multiple Databricks instances as part of a... ...experience with Databricks, PySpark, and Spark SQL... ...with sources in Azure, AWS, GCP, and others Why...
Databricks
Amazon Web Service
For contractors
Work at office
Remote work
Logicbroker
New York, NY
4 days ago
Lead Data Engineer
...Senior Data Engineer Location: Juno Beach, FL (Day one onsite) The Senior Data Engineer... ...workflows. Experience working with AWS or comparable cloud environments.... ...Experience with PostgreSQL/Aurora, Redshift, Databricks, Spark/PySpark, or similar data platforms....
Databricks
Amazon Web Service
Yantran LLC
North Palm Beach, FL
3 days ago
Lead Data Engineer
...business process flows, and data flow diagrams.... ...years of experience in data engineering, including design, application... ...including Parquet, Iceberg, PySpark, Lambda, Databricks and Azure data services.... ...seamless integration with AWS or other cloud services....
Databricks
Amazon Web Service
Work at office
Synovus
Columbus, GA
5 days ago
Lead Data Engineer
...have a strong background in data engineering and software development,... .... In this role, you will lead a team of skilled engineers... ...and Spark. • Leverage AWS services (e.g., S3, Glue,... ...Java Rest web services~Digital : Snowflake~Digital : Databricks~Digital : PySpark
Databricks
Amazon Web Service
PROLIM Corporation
Murphy, TX
2 days ago
Lead Data Engineer
...Job Title: Lead Data Engineer Location: Dallas, TX Client: Prodapt/... ...NumPy o SQLAlchemy o PySpark o FastAPI / Flask o Object... ...• Talend • DataStage • AWS Glue • Apache Airflow •... ...PySpark • Hadoop Ecosystem • Databricks • Kafka • Delta Lake •...
Databricks
Amazon Web Service
Contract work
Remote work
Abode Tech Zone
United States
4 days ago
Lead AWS Data Engineer
AWS Data Engineer (Contract) | Beverly Hills, CA (Onsite) Must-Have Experience Candidates should... .../ lakehouse — Amazon Redshift or Databricks. Real production ownership: modeling,... ...frameworks. travel. Apache Spark — PySpark or Spark SQL for large-scale batch/streaming...
Databricks
Amazon Web Service
Contract work
Triunity Software, Inc.
Los Angeles, CA
2 days ago
Lead Data Engineer
...Lead Data Engineer Location: Hybrid 3 times a week in Roseland, NJ (Relocation candidates are welcome) It revolves around DATAbricks, AWS not AZURE Must haves: AWS Databricks Lead experience- this... ...for staff as well Python Pyspark Contact Center Experience is a...
Databricks
Amazon Web Service
Work experience placement
Relocation
RIT Solutions
Roseland, NJ
5 days ago
Lead Data Engineer: Databricks on AWS Platform
...Companies in Los Angeles is seeking a senior hands-on Data Engineering Lead to set the technical direction for the CSGT data platform... ...pipelines, and lead cross-team initiatives using Databricks on AWS, PySpark, and dbt. The role emphasizes AI-first engineering, data...
Databricks
Amazon Web Service
Capital Group Companies
Los Angeles, CA
6 days ago
Lead Data Engineer - Databricks
...NAVA Software is looking for a Lead Data Engineer Details: Lead Data Engineer -Databricks Location: Houston TX Monday... ...in cloud platforms (especially AWS), advanced data modeling, and... ...skills in SQL (T-SQL), Python (PySpark/Pandas), DAX, Power Query (M),...
Databricks
Amazon Web Service
Nava Software Solutions LLC
Houston, TX
2 days ago
Lead Data Engineer
...Position: Senior Data Engineer Experience: 10+ Years Employment Type... ...strong expertise in Python, SQL, PySpark, ETL/ELT, cloud platforms, data... ...data pipelines using tools such as AWS Glue, Informatica, Azure Data Factory, Databricks, or equivalent technologies....
Databricks
Amazon Web Service
Contract work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...are looking for a Senior Data Engineer to join our FinTech customer team and lead the migration of custom... ...platform onto Databricks. This is not a ticket-taking... ...experience - Delta Lake, Spark (PySpark and Spark SQL), job/... ...platform experience (AWS or Azure), including...
Databricks
Amazon Web Service
Openkyber
Atlanta, GA
1 day ago
Lead Data Engineer
...seeking an experienced Senior Data Engineer to design, develop, and optimize... ...performance data ecosystems leveraging Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS, while also contributing to the... ...processing applications using PySpark and Spark SQL. Build and...
Databricks
Amazon Web Service
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...Job Title: Senior Databricks Engineer Location: 100% Remote Interview Process... ...: Job Description Lead/Senior Databricks Data Engineer Position: Lead Databricks... ...Databricks, Apache Spark, PySpark, SQL, Delta Lake, and... ...cloud platforms such as AWS, Azure, or Google Cloud...
Databricks
Amazon Web Service
Second job
Remote work
Openkyber
Atlanta, GA
2 days ago
Lead Data Engineer
...potential. Title and Summary Lead Data Engineer Lead Data Engineer... ...ETL/ELT frameworks utilizing Databricks, Spark, Delta Lake, and cloud... ...based data platforms on Azure, AWS, or GCP. Expertise... ...Databricks Apache Spark / PySpark Hadoop Delta Lake SQL...
Databricks
Amazon Web Service
Full time
Worldwide
Mastercard
Dublin, CA
16 days ago
Lead Data Engineer
...Seeking a data engineer for hybrid contract role in Westmont... ...data platforms on Azure or AWS, including services such as Databricks, Azure Data Factory,... ...with deep expertise in PySpark/Spark for distributed data... ...consumption. Track record of leading workstreams and driving...
Databricks
Amazon Web Service
Hourly pay
Contract work
For contractors
Monday to Friday
Shift work
2 days per week
3 days per week
Openkyber
Atlanta, GA
2 days ago
Big Data Engineer with PySpark
$154k - $194k
...of professional experience in Big Data Engineering ~ Hands-on experience with Apache Spark and PySpark ~ Programming experience with Python... ...technologies is highly preferred: AWS (S3, EMR, Glue, Redshift, Lambda), Azure (ADLS, Databricks, Azure Data Factory, Synapse), GCP...
Databricks
Amazon Web Service
Full time
VITS US
Irving, TX
1 day ago
Lead Data Engineer
...Lead Data Engineer Are you a Data Engineer with over 10 years of experience and expertise working with Databricks on AWS? We are investing in the next evolution of our data, analytics, and artificial intelligence ecosystem. We are looking for a Lead Data Engineer who...
Databricks
Amazon Web Service
Randstad
Chicago, IL
3 days ago
Lead Data Engineer
...team. Job Location Mclean, VA/ Remote Assignment Duration 4 Months Tech Stack Python SQL AWS Spark/Hadoop/Kafka Databricks/Snowflake Redshift Key Technologies Hadoop Kafka Python Spark Databricks Ampcus is an...
Databricks
Amazon Web Service
Remote work
Ampcus
United States
3 days ago
Lead Software Engineer - Data Engineer
$61k - $101k
...certification in software engineering concepts plus 5+... ...production data engineering and/or... ...skills and hands-on PySpark experience. We... ...proven experience leading the effective use... ...require experience with AWS data management... ...on EMR and/or Databricks, including tuning,...
Databricks
Amazon Web Service
Full time
J.P. Morgan
Jersey City, NJ
14 days ago
Lead Data Engineer
...Lead Data Engineer We are seeking a highly experienced Lead Data Engineer to drive the modernization... ...will have deep hands-on expertise in Databricks, Snowflake, Apache Spark, Java, and... ...solutions. Experience working with the AWS cloud ecosystem. Proven experience...
Databricks
Amazon Web Service
Saxon Global
Plano, TX
5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer - AWS, Databricks & PySpark. Be the first to apply!