Lead Data Engineer

Q1 Technologies

Lead Engineer
- Design and build robust, scalable ETLELT pipelines using PySpark to ingest data from diverse sources (databases, logs, APIs, files).
- Transform and curate raw transactional and log data into analysis-ready datasets in the Data Hub and analytical data marts.
- Develop reusable and parameterized Spark jobs for batch and micro-batch processing.
- Optimize performance and scalability of PySpark jobs across large data volumes.
- Ensure data quality, consistency, lineage, and proper documentation across ingestion flows.
- Collaborate with Data Architects, Modelers, and Data Scientists to implement ingestion logic aligned with business needs.
- Work with cloud-based data platforms (e.g., AWS S3, Glue, EMR, Redshift) for data movement and storage.
- Support version control, CICD, and infrastructure-as-code where applicable.
- Participate in Agile ceremonies and contribute to sprint planning, story grooming, and demos.

Apply

Vacancy posted 6 hours ago

Similar jobs that could be interesting for youBased on the Lead Data Engineer in Philadelphia, PA vacancy

Lead Data Engineer / Integration Specialist
$100k
...belongs. Benefits - We care about your well-being. Job Description Data Engineering at the City is a unique opportunity for meaningful, technically challenging, and service-oriented work. The Lead Data Engineer / Integration Specialist will play a key role in designing...
Suggested
Full time
Part time
Work experience placement
City of Philadelphia
Philadelphia, PA
5 hours ago
6424 - Lead Validation Engineer (Data Lake) / CSV Engineer
$87.78k - $136.23k
...skills and learn from our experiences to enhance our collective expertise Validation Engineer (Data Lake) Responsibilities:Qualification Strategy & Validation Leadership Lead qualification activities for an enterprise cloud-based Data Lake platform using modern,...
Suggested
Temporary work
Work experience placement
Verista, Inc.
Conshohocken, PA
17 days ago
Senior Systems Engineer - Radar Signal and Data Processing
...is the place for you. The Work: Lockheed Martin Rotary & Mission Systems is seeking a full-time Senior Systems Engineer. In this role, you will lead the design, analysis, testing, and requirements development and verification of high performance Radar Signal...
Suggested
Full time
Work at office
Remote work
Relocation
Flexible hours
Shift work
3 days per week
Lockheed Martin - US
Moorestown, NJ
14 days ago
Data Engineer
...Data Model and Structure Definition: • Design and develop conceptual, logical, and physical data models for data sets across a variety... ...Skills : Data Analysis Basic Qualification : Additional Skills : Data Engineer Background Check : No Drug Screen : No
Suggested
TriOptus LLC
Philadelphia, PA
6 hours ago
Data Engineer
$139.32k
...18.00 - $139,318.00 / Yearly Hours Per Week 40 Number Of Positions 1 Shift First Shift (Day) Job Description Data Engineer Sansar Tec LLC - Haddon Heights, NJ. Collaborate w/Agile teams to dsgn & dvlp data engg sol. Build distributed, low latency,...
Suggested
Full time
Shift work
Day shift
SANSAR TEC LLC
Haddon Heights, NJ
3 days ago
Data Engineer
$91.7k - $163.7k
...Connecting. Growing together Position in this function may include database architecture, engineering, design, optimization, security, and administration; as well as data modeling, big data development, Extract, Transform, and Load (ETL) development, storage...
Minimum wage
Full time
For contractors
Work experience placement
Work at office
Local area
UnitedHealth Group
Fort Washington, PA
1 day ago
Data Engineer
...Data Engineer Employment Type: Full-Time, Mid-level Department: Business Intelligence CGS is seeking a passionate and driven Data Engineer to support a rapidly growing Data Analytics and Business Intelligence platform focused on providing solutions that empower...
Full time
Remote work
Flexible hours
Contact Government Services LLC
Philadelphia, PA
2 days ago
Data Engineer
...Data Engineer Philadelphia, PA Experience: 5-9 years • Experience in big data, ETL project. • Demonstrable experience with big data tools: Hadoop, Apache Spark, Kafka • Demonstrable experience with data pipeline and workflow management tools: NiFi etc. •...
Keylent Inc
Philadelphia, PA
3 days ago
Junior/Entry Level Data Engineer
$100k
...Cloud World/Oracle Java one (Las Vegas) -2023/2022 and at Gartner Data Analytics Summit (Florida)-2023... ...developers, Data analysts/ Data Scientists, and Machine Learning engineers for full-time positions with clients. Who Should Apply Recent...
Full time
H1b
SynergisticIT
Philadelphia, PA
12 hours ago
Data Engineer
...Senior Data Engineer As a Senior Data Engineer, you will work with the team to build complex data pipelines using a new cutting-edge Data Ingestion Platform as well as several new and exciting products in the Big Data/Advanced Analytics space. You will operate in a...
Software Technology Inc
Blue Bell, PA
3 days ago
Data Engineer
...Philadelphia right now. About the Role: Our data team is organized across three groups: Data Engineering, Data Science, and Strategic Analytics. Data... ...first three months, you'll: Be the technical lead on your first meaningful project, guiding it from design...
Odd job
Full time
Work at office
Local area
Relocation package
Weekend work
Perpay - Career's Page
Philadelphia, PA
3 days ago
Data Engineer
...Extract data from noisy, inconsistent sources Validate and reconcile outputs across multiple inputs Ensure correctness... ...intelligence Core Requirements Strong Python and backend/data engineering experience Experience building production data pipelines...
3B Staffing LLC
Philadelphia, PA
1 day ago
Sr. DATA Engineer
...Data Engineer Location: West Conshohocken PA Duration: 6 + months Job Description: Candidate should have previous experience building a data lake using Oracle database technologies. Candidate should be expert in improving extraction time and DB performance tuning...
ClifyX
Conshohocken, PA
3 days ago
Service Delivery Center, AI & Data, Machine Learning Engineer (MLE) - Senior
$65.5k - $134k
.... Senior Analyst - Financial Services Organization – AI and Data – Service Delivery Center EY is the only professional services... ...(trading, risk, compliance, or banking) Knowledge of prompt engineering and in-context learning optimization Familiarity with model...
Summer holiday
Flexible hours
EY
Philadelphia, PA
12 hours ago
Associate Data Engineer
$70k - $90k
...Associate Data Engineer Our purpose is to help a billion people find the right job! Phenom is an AI-Powered talent experience platform that is redefining the HR tech space. We have grown into a global organization with offices in 6 countries and over 1,600 employees...
Full time
Internship
H1b
Work at office
Work visa
Flexible hours
3 days per week
Phenom
Ambler, PA
3 days ago
Senior Associate, Data Engineer - Databricks
$94.43k - $202.75k
...at The Lighthouse. KPMG is currently seeking a Senior Associate, Data Engineer for our Consulting practice. Responsibilities: Assist with technical design and development activities and lead a small workstream for implementation of large-scale data solutions...
Local area
Visa sponsorship
KPMG
Philadelphia, PA
5 days ago
Information Security Engineer
...JOB TITLE: Information Security Engineer JOB TYPE: Direct-Hire LOCATION: Philadelphia... ...identity, endpoint, network, cloud, and data protection technologies.... ...scripting or similar technical methods. Lead security projects and provide technical support...
Monday to Friday
Shift work
Day shift
RX2 Solutions
Philadelphia, PA
3 days ago
Sr Data Engineer
Senior Data Engineer As a Senior Data Engineer, you will enable data-driven decision-making by collecting, transforming, and visualizing data. You will design, build, maintain, and troubleshoot data processing systems with an emphasis on security, reliability, fault tolerance...
Work experience placement
Tential Solutions
Dresher, PA
1 day ago
Senior Data Engineer: Build Scalable Data Pipelines
First Insurance Company of Hawaii is seeking an experienced data engineer to design, implement and maintain data pipelines and ETL/ELT solutions that support analytics across TMNAS business units. You will collaborate with data analysts and engineers to ensure high-quality...
First Insurance Company Of Hawaii
Bala Cynwyd, PA
1 day ago
Senior Data Engineer: Design Scalable Data Pipelines & Analytics
Tokio Marine North America Services (TMNAS) is seeking an experienced data engineer to design, develop, and maintain data pipelines across TMNAS business units. You will implement ETL/ELT processes and collaborate with analysts, architects, and engineers to ensure high-...
TMNA Services, LLC
Bala Cynwyd, PA
1 day ago
Senior Data Engineer: AI-Driven ETL & Data Warehousing
Tential Solutions is seeking a Senior Data Engineer to facilitate data-driven decision-making in Dresher, United States. This role emphasizes building and maintaining data processing systems with a focus on security, reliability, and efficiency. Key responsibilities include...
Tential Solutions
Dresher, PA
1 day ago
Senior Data Engineer
Overview We are hiring a Data Engineer to join our Data Technologies team. This team partners with middle-and back-office application teams to design, build, and support next-generation data platforms and pipelines that ingest, process, and transform large-scale data into...
Susquehanna International Group
Bala Cynwyd, PA
4 days ago
Service Delivery Center, AI & Data, Machine Learning Engineer (MLE) - Manager
$76.2k - $174.1k
...Financial Services Organization - AI and Data - Service Delivery Center EY is the... ...the financial industry and technology Leading and managing multi-disciplinary teams through... ...of related work experience in AI/ML engineering or MLE/ML Ops Experience working with...
Work experience placement
Summer holiday
Flexible hours
EY
Philadelphia, PA
12 hours ago
Senior Civil Design Engineer (Data Centers)
...difference. Job Description We're looking for a Senior Civil Engineer to join our Advanced Manufacturing group, and you'll have the... ...industrial developments ~ Ability to perform project discipline lead responsibilities on complex, multi-disciplinary projects and...
Work at office
Local area
Jacobs
Philadelphia, PA
3 days ago
Sr Data Scientist
...Data Scientist Our financial services client is seeking a Data Scientist to join their team working hybrid schedule. Develop and deploy Generative AI solutions using large language models (LLMs) and RAG architectures to support business initiatives. Analyze and transform...
Alpine Solutions Group
Philadelphia, PA
2 days ago
Data Scientist
...Data Scientist II - Generative AI The Data Scientist II - Generative AI will apply knowledge and experience to real world problems... ...degree in a quantitative discipline such as Computer Science/Engineering, Statistics, Operations Research. 5+ years of hands-on data science...
Flexible hours
Kaav Inc.
Blue Bell, PA
3 days ago
Data Scientist III
$110.49k - $115.6k
...~ Elsevier, Inc. ~ Data Scientist III ~1600 John F. Kennedy Blvd., Suite 1800, Philadelphia, PA 19103 JOB DESCRIPTION:... ...ingestion into Emmet. Ensure compliance with DPR documentation. Lead the Rising Tide program and Mentor interns. Coordinate with IT developers...
For contractors
Work at office
Local area
Remote work
RELX Group plc
Philadelphia, PA
12 hours ago
Senior Data Scientist
...Data Science-Gen AI Engineer Lead development of time series forecasting models (ARIMA, VAR, state-space models, etc.) for business-critical use cases. Apply econometric techniques such as WLS, panel data models, and causal inference methods to solve real-world...
EXL
Philadelphia, PA
4 days ago
Associate Data Engineer
$1,100 per month
Ascensus is the leading independent technology and service platform that powers savings plans... ...multiple associate‑level Software Engineers to join our Software Engineering teams across... ...Back‑End, Software Quality/Test Automation, Data Engineering, Data Science, DevOps, and...
Full time
Internship
Visa sponsorship
Ascensus
Dresher, PA
12 hours ago
Manager, Data Engineering
...Manager For The Data & Ai Team Valorem Reply is an award-winning digital transformation... ...most recent years as a hands-on manager leading a team on enterprise scale data platform... ...to create working environments for data engineers and scientists, and general knowledge of...
Reply
Philadelphia, PA
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!