Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Data Engineering Lead

AceStack LLC

Job Title: Data Engineering Lead Location: Malvern, PA Contract Job Description: Skills: Java| AWS | Python | PySpark | Event-Driven Pipelines | Data Architecture Summary We are seeking an experienced Tech lead- Data Engineer (15+ years) with a strong background in Java, AWS, Python, PySpark, and event-driven architectures. You will design and build scalable batch and streaming data pipelines, optimize cloud data platforms, and deliver high-quality, reliable datasets that support analytics, reporting, and machine learning workloads. Key Responsibilities Architect, build, and maintain event-driven data pipelines using AWS services such as Kinesis, MSK/Kafka, Lambda, Step Functions, SQS/SNS, and Glue/EMR. Develop ETL/ELT workflows using Python and PySpark, ensuring performance, scalability, and cost efficiency. Implement and optimize Spark-based data transformations, partitioning strategies, and data processing frameworks. Design and manage data lake and warehouse structures using S3, Glue Catalog, Athena, and/or Redshift. Build streaming solutions with checkpointing, stateful transformations, idempotency, and schema evolution. Ensure high standards of data quality, observability, monitoring, and alerting (CloudWatch, Datadog, etc.). Implement data security best practices including IAM, encryption (KMS), networking, and governance. Create reusable frameworks, internal libraries, and CI/CD pipelines for automated deployments. Collaborate with data scientists, analysts, and business teams to deliver well-modeled, reliable datasets. Lead design reviews, mentor junior engineers, and contribute to engineering best practices. Required Qualifications 15+ years of professional experience in Data Engineering. Strong expertise in Python and PySpark for large-scale data processing. Advanced hands-on experience with AWS (S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, DynamoDB, Athena, Redshift). Deep experience building event-driven and streaming data pipelines. Strong SQL experience for analytical and ETL workloads. Hands-on experience with workflow orchestration tools such as Airflow or Step Functions. Experience with CI/CD, Git, and Infrastructure-as-Code (Terraform or CloudFormation). Strong understanding of distributed systems, Spark performance tuning, data modeling, and cloud cost optimization. Knowledge of data security, encryption, networking, and compliance best practices in cloud environments. Soft Skills Strong design and architectural understanding. Excellent communication and stakeholder interaction skills. Ability to work in a globally distributed team. AceStack LLC

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Data Engineering Lead in Malvern, PA vacancy
  • Role Summary We are seeking a highly skilled Senior Data Engineer (or Lead Data Engineer ) to design, build, and optimize scalable data pipelines and ETL solutions across enterprise environments. The ideal candidate will bring deep expertise in Big Data technologies ,... 
    Suggested

    Argyll Infotech, Inc.

    King of Prussia, PA
    1 day ago
  • Role Purpose Serve as the onsite Data Engineering and Architecture lead, driving technology innovation, evaluation, and adoption across the enterprise data landscape. Key Responsibilities Act as the onsite SME for Data Engineering and Data Architecture. Lead Proof of... 
    Suggested

    VBeyond

    Exton, PA
    1 day ago
  • $111.44k - $183.08k

     ...our name, it's our mission and our purpose. The Enterprise Data Management Lead drives the strategy, standards, and execution of enterprise...  ...and architecture principles, this is not a hands-on engineering or ETL/ELT role; the focus is on embedding governance, quality... 
    Suggested
    Work experience placement
    Work at office
    Flexible hours

    Wilmington Savings Fund Society

    Berwyn, PA
    1 day ago
  •  ...designs database systems, and develops tools for real-time and offline analytic processing. Troubleshoots software and processes for data consistency and integrity. Integrates data from a variety of sources for business partners to generate insight and make decisions.... 
    Suggested

    Brains Technology Solutions, Inc.

    Malvern, PA
    16 hours ago
  •  ...Job Description Job Description Lead Data Engineer / Data Architect (Innovation & POC Lead) Location : Exton, PA Rate : DOE Job Description Serve as the onsite Data Engineering and Architecture lead, driving technology innovation, evaluation, and adoption... 
    Suggested

    Georgia IT Inc.

    Exton, PA
    3 days ago
  • Machine Learning Engineer We are seeking an experienced Machine Learning Engineer to join our AI/ML Engineering team. You will be responsible for developing and optimizing complex data pipelines, integrating model pipelines, and building scalable AI/ML solutions, including... 
    Work experience placement

    Samprasoft

    Malvern, PA
    1 day ago
  • Data Lake 2.0 Developer Data Lake 2.0 is a multi-year program aimed at modernizing legacy...  ...that will be in use by Analysts and Data Engineers. The deliverables will help data...  ...Work on features and POCs with industry-leading technologies and tools Work on greenfield... 

    Samprasoft

    Malvern, PA
    1 day ago
  • ROLE SENIOR DATA ENGINEER INDUSTRY Pharma, Development & Manufacturing DURATION CONTRACT, probable extension LOCATION REMOTE (off-shore or near-shore), Malvern, PA 19355 INTERVIEW MS TEAMS conversation the candidate will be required to work in EST starting around 1... 
    Contract work
    Immediate start
    Remote work

    3B Staffing LLC

    Malvern, PA
    1 day ago
  • Responsibilities Pulling data from GitHub and Jira transforming it for input into our data warehouse. Qualifications Familiarity with AWS including lambda, event bridge, s3. Proficient in typescript Datalake / Data warehouse experience is a big plus. Required Skills :... 

    Brains Technology Solutions, Inc.

    Malvern, PA
    1 day ago
  • Senior Data Engineer - 7+ Years of Experience We are seeking a highly experienced Senior Data Engineer with 7+ years of expertise in designing, building, and optimizing robust data solutions. The ideal candidate must possess top-tier skills in Python, AWS services, API... 

    Samprasoft

    Malvern, PA
    1 day ago
  • Data Engineer Duties And Responsibilities Data engineers create and maintain an optimal data infrastructure. They combine raw information to create machine-readable formats. These are the major duties and responsibilities of a data engineer: Putting together large, intricate... 

    3B Staffing LLC

    Malvern, PA
    1 day ago
  • Experience in the below technologies. ETL using EMR & Glue require working experience. SQL/PYSPARK/PYTHON/Jupyter Notebooks/GitHub functions/ CI/CD Knowledge on cloud Services: EMR/GLUE/S3/EC2/Service Catalog/PostgreSQL/Lambda/cloudFormation/SNS/EventBridge Knowledge ...
    Work experience placement

    Brains Technology Solutions, Inc.

    Malvern, PA
    1 day ago
  • Data Engineer Location: Malvern, PA - Hybrid Important Submission Requirement Candidates must take and pass a technical assessment before submission . Only submit candidates who are willing and available to complete the assessment. Position Overview Vanguard is seeking... 

    3B Staffing LLC

    Malvern, PA
    1 day ago
  • Data Engineer As a Data Engineer, you’ll build sophisticated solutions that support a wide range of advanced data driven capabilities. Provides...  ...in specific business areas and company-wide systems. Leads all phases of solution development. Explains technical considerations... 

    Samprasoft

    Malvern, PA
    1 day ago
  • Job Title Responsibilities Ability to run and deploy jobs, view creation and deployment, testing, analyzing, and cleansing large data files Qualifications Expert in SQL AWS S3 and Glue experience Advanced Data Modeling principles Required Skills : Data Warehouse Basic... 

    Kaav Inc.

    Malvern, PA
    1 day ago
  • Job Summary The Databricks AWS Data Engineer will develop and support a new data environment from the ground up, focusing primarily on data pipelines and cloud-based data engineering solutions. The role requires strong development experience with Python, PySpark, and SQL... 

    Compunnel

    Malvern, PA
    1 day ago
  •  ...Troubleshoots software and processes for data consistency and integrity. Integrates complex...  ...business to implement data solutions. Leads all phases of solution development....  ...client. Educates and develops junior data engineers on the team while applying quality control... 

    Kaav Inc.

    Malvern, PA
    1 day ago
  • Job Title: Senior Data Engineer Location: Malvern, PA Onsite Duration:6 Months Experience Required: 10 & Above EXPERIENCE_RANGE_IN_REQUIRED_SKILLS_ Python, AWS- S3, AWS Lambdas, AWS Glue, SQL, DynamoDB , Kafka/ Kinesis , pySpark ROLE_DESCRIPTION - Overview: We are seeking... 
    Local area

    I-Flow

    Malvern, PA
    1 day ago
  • Details given below:- Role:-ETL Data Engineer Malvern, PA Charlotte, NC Phoenix, AZ JD:- • Writes ETL (Extract / Transform / Load) processes, designs database systems and, develops tools for real-time and offline analytic processing. • Deep technical knowledge - including... 

    Trinity Technology Solutions

    Malvern, PA
    1 day ago
  • Job Summary The AWS Data Engineer will design, develop, and implement scalable, high-performance data ingestion and transformation pipelines using AWS services and Apache Spark. The role involves integrating complex datasets from multiple sources, developing advanced SQL... 

    Compunnel

    Malvern, PA
    1 day ago
  • Purple Drive AWS Data Engineer - Python, AWS Glue, PySpark & Attunity We are seeking an experienced AWS Data Engineer with strong expertise in Python, AWS Glue, PySpark, Attunity (Qlik Replicate), and AWS Cloud to design, develop, and maintain scalable enterprise data integration... 

    Purple Drive

    Malvern, PA
    1 day ago
  • Role : Data Engineer with Python Location: Malvern, PA (Onsite) Contract Job Summary We are seeking a Data Engineer with strong Python expertise to design, build, and maintain scalable data pipelines and data infrastructure. The ideal candidate will work with large datasets... 
    Contract work

    AceStack LLC

    Malvern, PA
    3 days ago
  • $130k - $160k

     ...Job Description Job Description Job Title: Data Governance Lead _ Data Governance, Data Quality, Lineage & Observability Location:...  ...with the ability to engage business, risk, compliance, data, engineering and executive stakeholders. Preferred certifications in... 
    Full time

    Diverse Lynx

    Malvern, PA
    24 days ago
  • We are looking for a Snowflake Data Engineer to build modern data solutions that support analytics, AI, and enterprise decision-making. This role will focus on creating reliable pipelines, scalable data models, and secure data products within a Snowflake environment. The... 

    Robert Half

    Villanova, PA
    2 days ago
  •  ...Role at a GlanceLincoln Financial's AI, Data and Analytics (AIDA) organization is building...  ...with AIDA's infrastructure, platform, engineering, MLOps, and AI-native platform teams, with...  ....As a Senior Data Engineer, you wil lead the end-to-end technical direction and delivery... 
    For contractors
    Work experience placement
    3 days per week

    Lincoln Financial Group

    Radnor, PA
    16 hours ago
  •  ...least 4 years of experience in software design, development and release.Well versed with agile development methodologies.Experience in Data Ingestion, Streaming and Big Data technologies.Excellent communication and stakeholder managementAbility to work with distributed... 
    Permanent employment
    Full time
    H1b

    Sonsoft

    King of Prussia, PA
    3 days ago
  •  ...Overview This is a hybrid role - 2 days remote and 3 days in the Malvern, PA office.   CubeSmart is seeking a Manager, Data Engineering to lead the team responsible for building and evolving the enterprise data platform that powers analytics, reporting, AI, and... 
    Work at office
    Remote work

    Cubesmart

    Malvern, PA
    2 days ago
  • A leading technology firm is seeking a Software Developer with at least 4 years of experience in software design, development, and release...  ...requires strong analytical skills and familiarity with Big Data technologies and agile methodologies. Ideal candidates will possess... 
    Permanent employment
    Full time
    H1b

    Sonsoft Inc

    King of Prussia, PA
    16 hours ago
  •  ...opportunity.Requisition #: 76677The Role at a GlanceAbout AIDA Data Strategy and GovernanceLincoln Financial's AI, Data and Analytics...  ...the result. We deliver with AIDA's infrastructure, platform, engineering, MLOps, and AI-native platform teams, with governance built into... 
    For contractors
    Work experience placement
    3 days per week

    Lincoln Financial Group

    Radnor, PA
    16 hours ago
  • Role Overview: This role is for a Data Engineer focused on designing, implementing, and optimizing robust ETL/ELT pipelines and complex data models primarily using PySpark and Snowflake. The engineer will be responsible for ensuring high data quality, integrity, and efficient... 

    Prophecy Technologies

    West Chester, PA
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Data Engineering Lead. Be the first to apply!