Information Technology_USA - USA_Developer

Senior Data Engineer

Location: Malvern, PA

Duration: 6 months

Act as a technical lead for data engineering and cloud data platforms

Design scalable and reliable data architectures

Collaborate with data scientists, analysts, and business stakeholders

Provide guidance on best practices in data engineering and architecture

Lead design discussions and technical decision-making

Mentor junior engineers and review their work

Ensure alignment of data solutions with business goals

Drive innovation in data processing and pipeline design

Promote data governance, security, and compliance standards

Architect and build event-driven data pipelines using AWS services

Develop ETL/ELT workflows using Python and PySpark

Design and optimize batch and streaming data pipelines

Implement Spark-based transformations and performance tuning

Build and manage data lakes and warehouses using S3, Glue, Athena, and Redshift

Develop streaming solutions with checkpointing and stateful processing

Ensure data quality, monitoring, and alerting using tools like CloudWatch and Datadog

Implement security best practices including IAM, encryption, and governance

Create reusable frameworks and internal libraries

Develop and maintain CI/CD pipelines for data workflows

Collaborate with cross-functional teams to deliver high-quality datasets

Optimize cloud infrastructure for performance and cost efficiency

Strong expertise in Python and PySpark

Advanced experience with AWS services (S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, Athena, Redshift)

Experience building event-driven and streaming data pipelines

Strong SQL skills for data processing and analytics

Experience with workflow orchestration tools such as Airflow or Step Functions

Knowledge of distributed systems and data architectures

Experience in Spark performance tuning and optimization

Experience with CI/CD tools and version control (Git)

Knowledge of Infrastructure-as-Code tools like Terraform or CloudFormation

Understanding of data security, encryption, and governance

Data Engineering

AWS Cloud

PySpark

Python

Event-Driven Architecture

Data Pipelines

ETL/ELT

Data Warehousing

Streaming Data Processing

Cloud Optimization

10 years or above