Information Technology_USA - USA_Developer

Senior Data EngineerLocation: Malvern, PADuration: 6 monthsAct as a technical lead for data engineering and cloud data platformsDesign scalable and reliable data architecturesCollaborate with data scientists, analysts, and business stakeholdersProvide guidance on best practices in data engineering and architectureLead design discussions and technical decision-makingMentor junior engineers and review their workEnsure alignment of data solutions with business goalsDrive innovation in data processing and pipeline designPromote data governance, security, and compliance standardsArchitect and build event-driven data pipelines using AWS servicesDevelop ETL/ELT workflows using Python and PySparkDesign and optimize batch and streaming data pipelinesImplement Spark-based transformations and performance tuningBuild and manage data lakes and warehouses using S3, Glue, Athena, and RedshiftDevelop streaming solutions with checkpointing and stateful processingEnsure data quality, monitoring, and alerting using tools like CloudWatch and DatadogImplement security best practices including IAM, encryption, and governanceCreate reusable frameworks and internal librariesDevelop and maintain CI/CD pipelines for data workflowsCollaborate with cross-functional teams to deliver high-quality datasetsOptimize cloud infrastructure for performance and cost efficiencyStrong expertise in Python and PySparkAdvanced experience with AWS services (S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, Athena, Redshift)Experience building event-driven and streaming data pipelinesStrong SQL skills for data processing and analyticsExperience with workflow orchestration tools such as Airflow or Step FunctionsKnowledge of distributed systems and data architecturesExperience in Spark performance tuning and optimizationExperience with CI/CD tools and version control (Git)Knowledge of Infrastructure-as-Code tools like Terraform or CloudFormationUnderstanding of data security, encryption, and governanceData EngineeringAWS CloudPySparkPythonEvent-Driven ArchitectureData PipelinesETL/ELTData WarehousingStreaming Data ProcessingCloud Optimization10 years or above