Lead Data Engineer
Q1 Technologies
Lead Engineer
- Design and build robust, scalable ETLELT pipelines using PySpark to ingest data from diverse sources (databases, logs, APIs, files).
- Transform and curate raw transactional and log data into analysis-ready datasets in the Data Hub and analytical data marts.
- Develop reusable and parameterized Spark jobs for batch and micro-batch processing.
- Optimize performance and scalability of PySpark jobs across large data volumes.
- Ensure data quality, consistency, lineage, and proper documentation across ingestion flows.
- Collaborate with Data Architects, Modelers, and Data Scientists to implement ingestion logic aligned with business needs.
- Work with cloud-based data platforms (e.g., AWS S3, Glue, EMR, Redshift) for data movement and storage.
- Support version control, CICD, and infrastructure-as-code where applicable.
- Participate in Agile ceremonies and contribute to sprint planning, story grooming, and demos.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in Philadelphia, PA vacancy
- ...Job Description Day-to-Day: • Serve as a senior technical lead within the Data Engineering organization • Oversee data intake processes and determine how new data sources are integrated into the enterprise environment • Design, build, and maintain Microsoft Fabric...SuggestedImmediate start
Insight Global
Fort Washington, PA5 hours ago - ...Role: Lead AWS Data Engineer Location: Philadelphia, PA- Onsite Contract position Role Description We are seeking a Lead Data Engineer to design, build, and optimize large-scale data solutions on AWS. This role combines hands-on technical expertise...SuggestedContract work
Diverse Lynx
Philadelphia, PA4 days ago - ...Lead Data Engineer Lutron is seeking a lead data engineer in Coopersburg, PA or Philadelphia, PA to help build and scale a modern, cloud-native data and AI platform powering analytics, real-time intelligence, and next-generation products. In this role, you will shape...Suggested
Lutron
Philadelphia, PA4 days ago - First Insurance Company of Hawaii is seeking an experienced data engineer to design, implement and maintain data pipelines and ETL/ELT solutions that support analytics across TMNAS business units. You will collaborate with data analysts and engineers to ensure high-quality...Suggested
First Insurance Company of Hawaii
Bala Cynwyd, PA2 days ago $144.77k - $209.11k
Sr. Data Engineer Employment Type: Full-Time, Mid-level Department: Business Intelligence CGS is seeking a passionate and driven Data Engineer to support a rapidly growing Data Analytics and Business Intelligence platform focused on providing solutions that empower our...SuggestedFull timeRemote workFlexible hoursGrabJobs
Philadelphia, PA4 days ago- ...Senior Data Engineer As a Senior Data Engineer, you will enable data-driven decision-making by collecting, transforming, and visualizing data. You will design, build, maintain, and troubleshoot data processing systems with an emphasis on security, reliability, fault tolerance...Work experience placement
Tential Solutions
Dresher, PA2 days ago - ...2010, Asset Based Lending, LLC (“ABL”) is one of the nation’s leading Hard Money Lenders. We provide fast bridge financing to real estate... ...Build The Future Position Overview We are seeking a Senior Data Engineer to own and evolve our enterprise data platform, enabling...
Asset Based Lending LLC
Cherry Hill, NJ3 days ago $139.32k
...18.00 - $139,318.00 / Yearly Hours Per Week 40 Number Of Positions 1 Shift First Shift (Day) Job Description Data Engineer Sansar Tec LLC - Haddon Heights, NJ. Collaborate w/Agile teams to dsgn & dvlp data engg sol. Build distributed, low latency,...Full timeShift workDay shiftSANSAR TEC LLC
Haddon Heights, NJ23 hours ago$140k - $160k
...design, develop, and maintain databases and data integration (ETL) systems to support... ...processes, ETL workflows, and data platforms, leading the creation of technical designs and... ...languages is commonly used in data engineering, such as Python or Java Our Company:...Work experience placementRemote workWork from homeCarrington
Cherry Hill, NJ23 hours ago- ...Sr Data Engineer IntegriChain is the data and application backbone for market access departments of Life Sciences manufacturers. We deliver the data, the applications, and the business process infrastructure for patient access and therapy commercialization. More than...
IntegriChain
Philadelphia, PA5 hours ago - ...We are seeking a talented Data Engineer with strong communication skills, passion for solving business problems with data and has domain knowledge in Finance, Human Resources, and Customer Success. You have empathy, curiosity and desire to improve and constantly learn....Local area
Cornerstone OnDemand
Philadelphia, PA2 days ago - ...We are UMG, the Universal Music Group. We are the world's leading music company. In everything we do, we are committed to artistry, innovation and entrepreneurship. We own and operate a broad array of businesses engaged in recorded music, music publishing, merchandising...Summer workImmediate startFlexible hours
Universal Music Group
Philadelphia, PA23 hours ago $105.1k - $189.6k
...Overview This role is essential for designing and developing data architectures across on-premise, cloud, and hybrid platforms to... ...data needs. It primarily involves collaborating with data engineers to analyze, architect, design, and develop data warehouse and business...Full timeTemporary workPart timeWork experience placementLocal areaFlexible hoursT-Mobile
Philadelphia, PA4 days ago- ...Sr Data Engineer Location: Philadelphia, PA/ Durham, NC Core engineering data skills. Strong in Python ideally using official SDKs to call REST APIs. Experience building ETL/ingestion pipelines: incremental syncs, idempotent upserts, backfills, error handling, retries...
Yantran LLC
Philadelphia, PA3 days ago - ...Data Engineer Philadelphia, PA Experience: 5-9 years • Experience in big data, ETL project. • Demonstrable experience with big data tools: Hadoop, Apache Spark, Kafka • Demonstrable experience with data pipeline and workflow management tools: NiFi etc. •...
Keylent Inc
Philadelphia, PA23 hours ago - ...Extract data from noisy, inconsistent sources Validate and reconcile outputs across multiple inputs Ensure correctness... ...intelligence Core Requirements Strong Python and backend/data engineering experience Experience building production data pipelines...
3B Staffing LLC
Philadelphia, PA3 days ago - ...Data Engineer Location: West Conshohocken PA Duration: 6 + months Job Description: Candidate should have previous experience building a data lake using Oracle database technologies. Candidate should be expert in improving extraction time and DB performance tuning...
ClifyX
Conshohocken, PA5 hours ago - ...Senior Data Engineer As a Senior Data Engineer, you will work with the team to build complex data pipelines using a new cutting-edge Data Ingestion Platform as well as several new and exciting products in the Big Data/Advanced Analytics space. You will operate in a...
Samprasoft
Blue Bell, PA1 day ago - ...Data Engineer Location: Philly or Whitehouse Station, NJ - Hybrid schedule 3 days in the office. Duration: 12 months + The Global Actuarial Systems team is a small, agile group of technology enthusiasts. We are tasked with solving the complex data needs of both US...Full timeWork at office
Georgia IT Inc
Philadelphia, PA23 hours ago - ...Operational Support Data Engineer (Co-op) We are seeking a motivated and curious Data Engineer to join our Enterprise Data & Analytics team. This co-op provides hands‑on experience building and supporting cloud‑based data platforms and pipelines that power enterprise analytics...
Campbell Soup Company
Camden, NJ1 day ago - ...We are seeking an experienced Data Engineer to design, build, and optimize scalable data platforms that support analytics, reporting, and data‑driven decision‑making across the organization. This role will partner closely with Product, Analytics, Data Science, and Engineering...
GCS Recruitment
Camden, NJ3 days ago $95k - $150k
...Junior Data Scientist With Data Engineering Skills Seeking a motivated Junior Data Scientist with Data Engineering skills to join our tech clients analytics and engineering teams. This is an exciting opportunity for recent graduates or early-career professionals who...Work experience placementInternshipSynergisticIT
Philadelphia, PA1 day ago- ...Philadelphia right now. About the Role: Our data team is organized across three groups: Data Engineering, Data Science, and Strategic Analytics. Data... ...first three months, you'll: Be the technical lead on your first meaningful project, guiding it from design...Odd jobFull timeWork at officeLocal areaRelocation packageWeekend work
Perpay - Career's Page
Philadelphia, PA5 hours ago - ...Tokio Marine North America Services seeks a Senior Data Engineer to design and implement complex data pipelines and ETL processes in a mainly on‑prem/cloud data environment. You will collaborate with data analysts, architects and business stakeholders to understand data...
Tokio Marine North America, Inc.
Bala Cynwyd, PA2 days ago $100k - $130k
...possible. To achieve this vision, Aramark will deliver strategic data assets that enable our operators to maximize the hospitality... ...experience for our consumer and clients. We are seeking a Data Engineer who will work collaboratively with our enterprise and line of...Local areaAramark
Philadelphia, PA23 hours ago- ...Senior Data Engineer Description Seeking a talented Senior Data Engineer to build scalable cloud migration solutions for a leading Fortune 500 facilities services company. The Senior Data Engineer will develop and implement a migration solution from SQL Server...
Experis/Manpower Group
Philadelphia, PA23 hours ago $100k
...Cloud World/Oracle Java one (Las Vegas) -2023/2022 and at Gartner Data Analytics Summit (Florida)-2023... ...developers, Data analysts/ Data Scientists, and Machine Learning engineers for full-time positions with clients. Who Should Apply Recent...Full timeH1bSynergisticIT
Philadelphia, PA2 days ago$100k
...Benefits - We care about your well-being. Job Description Data Engineering at the City is a unique opportunity for meaningful, technically challenging, and service-oriented work. The Lead Data Engineer / Integration Specialist will play a key role in designing...Full timePart timeWork experience placement$65.5k - $134k
.... Senior Analyst - Financial Services Organization - AI and Data - Service Delivery Center EY is the only professional services... ...(trading, risk, compliance, or banking) Knowledge of prompt engineering and in-context learning optimization Familiarity with model...Summer holidayFlexible hoursEY
Philadelphia, PA2 days ago- ...Short Description The Analytics Developer will create reports, data visualizations and analytical solutions that deliver actionable... ...Requirements Master’s degree in a STEM field (Science, Technology, Engineering, Mathematics or Business Analytics) with a focus on Business...Full timePart time
Cooper University Health Care
Camden, NJ1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
Related searches
- data auditor Philadelphia, PA
- data coordinator Philadelphia, PA
- data cabling Philadelphia, PA
- data visualization Philadelphia, PA
- data modeling Philadelphia, PA
- data governance Philadelphia, PA
- data loss prevention engineer Philadelphia, PA
- clinical data Philadelphia, PA
- education data Philadelphia, PA
- data officer Philadelphia, PA