Lead Data Engineer
Q1 Technologies
Lead Engineer
- Design and build robust, scalable ETLELT pipelines using PySpark to ingest data from diverse sources (databases, logs, APIs, files).
- Transform and curate raw transactional and log data into analysis-ready datasets in the Data Hub and analytical data marts.
- Develop reusable and parameterized Spark jobs for batch and micro-batch processing.
- Optimize performance and scalability of PySpark jobs across large data volumes.
- Ensure data quality, consistency, lineage, and proper documentation across ingestion flows.
- Collaborate with Data Architects, Modelers, and Data Scientists to implement ingestion logic aligned with business needs.
- Work with cloud-based data platforms (e.g., AWS S3, Glue, EMR, Redshift) for data movement and storage.
- Support version control, CICD, and infrastructure-as-code where applicable.
- Participate in Agile ceremonies and contribute to sprint planning, story grooming, and demos.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer in Philadelphia, PA vacancy
- OverviewLutron is seeking a Lead Data Engineer in Coopersburg, PA or Philadelphia, PA to help build and scale a modern, cloud-native data and AI platform powering analytics, real-time intelligence, and next-generation products. In this role, you will shape the architecture...SuggestedWorldwide
$152.9k - $221.1k
By joining Chubb as Lead Data Engineer for our North America Finance & Actuarial data platform, you'll set the engineering direction for a portfolio of strategic applications — Atlas, BAR, CDW, Posting, and Regulatory Reporting — that directly underpin financial, actuarial...SuggestedPermanent employmentWork experience placementWork at officeLocal area- ...the country.Job DescriptionCapTech Machine Learning Engineers are responsible for designing and implementing data-driven solutions for our clients, with a specific... ...and our clients.Specific responsibilities for the Lead Machine Learning Engineer position include:...SuggestedWork at officeRemote workVisa sponsorshipWork visaFlexible hours
- ...Role: Lead AWS Data Engineer Location: Philadelphia, PA- Onsite Contract position Role Description We are seeking a Lead Data Engineer to design, build, and optimize large-scale data solutions on AWS. This role combines hands-on technical expertise...SuggestedContract work
Diverse Lynx
Philadelphia, PA1 day ago $159.1k - $265.1k
Position Summary Our Deloitte AI & Engineering team works to transform technology platforms, drive innovation, and help make... ...ventures, and fuel growth through innovation. Work You’ll Do As a Lead AI and Data Science Engineer II on the team, you will be responsible for...SuggestedLocal area- ...opportunities, a world-class training facility, and leading market tools, we help our people... ...Leverage a variety of modern big data tools / approaches to solve complex business... ...minimum of five years of technical data engineering experience; U.S. Federal government...Local area
KPMG
Philadelphia, PA22 hours ago - DescriptionWe are looking for a Data Engineer to join a growing team in Conshohocken, Pennsylvania within the financial services industry. In this role, you will build and enhance modern data pipelines and warehouse structures that support reporting, analytics, and business...
- ...by advanced machine learning, large-scale data processing, and cloud technologies. Our... ...and curious Senior Data Science Software Engineer to join our team. In this role, you'll work... ...and technologies.Job Responsibilities:Lead the end-to-end development of machine learning...Full timeTemporary workPart timeWork experience placementLocal areaFlexible hours
$100.79k - $160.26k
...DLA Piper is a place you can engage in meaningful work and grow your career. Let’s see what we can achieve. Together.SummaryThe Data Engineer, Solutions & Data role designs, builds, and operates data pipelines and data integration processes that translate raw data into...Full timeWork at officeRemote workRelocationVisa sponsorshipRelocation package$128k - $252.5k
...clients' businesses. We are hiring a Senior Data Scientist to collaborate directly with... ...team of interdisciplinary data scientists, engineers, architects, and consultants, our work... ...sustain professional relationshipsAbility to lead projects or workstreamsAbility to manage...Local area- Company DescriptionIntegriChain is the data and application backbone for market access departments of Life Sciences manufacturers.... ...platform design patterns across Integrichain.Hands-on Snowflake engineering: Design, build, optimize, and operate Snowflake data models, pipelines...Visa sponsorshipFlexible hours
- ...supported to build a rewarding career with opportunities to grow, innovate and inspire. Make history with us.Operational Support Data Engineer (Co-op)We are seeking a motivated and curious Data Engineer to join our Enterprise Data & Analytics team. This co-op provides...Full time
$127.34k - $160k
POSITION OVERVIEWNBME is looking for a Data Engineer to join a highly skilled and growing team of data engineers and BI developers. This team... ...AI/ML model training, fine-tuning, and feature extraction. Lead evaluation, research, and experimentation efforts with batch and...InternshipWork at officeLocal areaRemote work$105.4k - $207.8k
...Transformation (I&DT) practice. I&DT brings an engineering- and innovation-led mindset to how... ...and scales technology-enabled solutions. Data Studio is Converge for Healthcare's foundational... ...to develop client-facing materials, lead technical discovery conversations, and...Local area- ...partner helping organizations modernize data ecosystems, build real-time analytics capabilities... ...and Gemini Enterprise.You AreA hands-on Engineer with foundational experience in Data... ..., blockers, and insights to project leads and client stakeholders.4. Data Governance...Full timeWork experience placementLive inWork at officeLocal area
- ...WithAt Slalom, we co-create custom software, data, and cloud products with clients who are... ...true change.Our Data Architecture & Engineering team sees data as the foundation for decision... ...and technology consulting company that leads with outcomes to bring more value, in all...Temporary workLocal area
- ...supported to build a rewarding career with opportunities to grow, innovate and inspire. Make history with us.Operational Support Data Engineer - Agentic AI & ML Ops (Co-op)We are seeking a motivated and curious Data Engineer - Agentic AI & ML Ops to join our Enterprise...Full time
$155.6k - $306.8k
...Summary At Deloitte, Forward Deployed Engineers (FDE) don’t just build AI solutions, they... ...into enterprise-scale impact, pairing leading class engineering with pod-based delivery... ...6.Work you’ll doAs a Senior Microsoft AI&Data FDE, you will work side by side with...Local area$224.19k - $351.57k
...seeking a visionary Senior Technologist, Data & Agentic AI Enablement to shape the future... ...by AI agents. Transforming data engineering through Agentic AI, embedding AI across the... ...DescriptionAgentic AI Transformation of Data Engineering:Lead the strategy for integrating Agentic AI...Full timeWork at officeRemote workWorldwide- Overview Susquehanna is one of the world’s leading options market makers, providing liquidity across global financial markets through... ...expertise. We are looking for an experienced Quantitative Data Engineer to join our Prediction Markets team.We specialize in geopolitical...
$100k - $115k
Role: Data EngineerWho You’ll Work WithAt Slalom we co-create custom software, data, AI... ...on injecting intelligence into products, engineering systems that support learning and insight... ...business and technology consulting company that leads with outcomes to bring more value, in all...Temporary workWork at officeLocal areaFlexible hours- OverviewWe are hiring a Data Engineer to join our Data Technologies team. This team partners with middle-and back-office application teams to design, build, and support next-generation data platforms and pipelines that ingest, process, and transform large-scale data into...
- ...every day. You will be supported to build a rewarding career with opportunities to grow, innovate and inspire. Make history with us.Data Engineer / AI Engineer (Co-op)We are seeking a motivated and curious Data Engineer / AI Engineer to join our team. This co-op /...Full timeInternship
$41 per hour
...DescriptionThe Financial Analytics Developer will create reports, data visualizations and analytical solutions that deliver actionable... ...’s degree required, In a STEM field (Science, Technology, Engineering, Mathematics or Business Analytics) with focus on Business Analytics...Full timePart time- ...We have a diverse portfolio of marketed medicines, including leading therapies for sleep disorders and epilepsy, and a growing portfolio... ...supporting complex or leading singular projects related to data engineering requirements and initiatives across Jazz Research and...Full timeWork at officeRemote workWorldwideFlexible hours
- ...UNSTOPPABLE for our employees!Job OverviewWe're looking for a Principal Data Engineer who brings strong technical judgment, a product-oriented... ...top consumer product brands in the United States. This role leads the development and implementation of data pipelines critical...Full timeTemporary workPart timeWork experience placementLocal areaFlexible hours
$52 per hour
...DescriptionThe Advanced Analytics Developer will lead the creation of innovative analytical... ...of success and leverage available data sources to design, develop and deploy analytical... ...in Computer Science, Business Analytics, Engineering or closely related quantitative field or...Full timePart time- ...development opportunities, a world-class training facility, and leading market tools, we help our people continue to grow both... ...career in Advisory.KPMG is currently seeking a Sr. Associate, Data Science Engineer to join our Consulting practice.Responsibilities:Translate advanced...H1bLocal area
- ...Specialist with 7 to 11 years of experience in data analysis proficient in PowerBI and... ...BI tool usageRoles and Responsibilities Lead the endtoend lifecycle of BI reporting products... ...reliable reporting Collaborate with data engineers and business analysts to integrate data...
- ...Data Engineer/Data Scientist Proven experience (6+ years) as a data engineer and data scientist, preferably in a fast-paced technology or analytics-driven environment. Telecom domain must. Big Data / ETL development background needed. Cloud technologies - AWS...
United IT
Philadelphia, PA2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer. Be the first to apply!
Related searches
- data loss prevention analyst Philadelphia, PA
- data labeling Philadelphia, PA
- data collection Philadelphia, PA
- data technician Philadelphia, PA
- data governance Philadelphia, PA
- data visualization Philadelphia, PA
- data coordinator Philadelphia, PA
- test data management Philadelphia, PA
- data processor Philadelphia, PA
- clinical data Philadelphia, PA