Data Engineer

Data Engineer

We are seeking a Data Engineer to join a team focused on building and supporting a Python-based, AWS-native data ingestion platform. This platform is responsible for moving and transforming critical financial data from source systems into third-party applications while ensuring compliance, security, and data integrity throughout the data lifecycle. The Data Engineer will be responsible for developing, maintaining, and enhancing data pipelines across multiple processing stages. This role requires collaboration with engineering and platform teams to ensure scalable, secure, and reliable data processing solutions within a cloud-native AWS environment.

Key Responsibilities

• Design, develop, and maintain data pipeline components using Python and AWS-native services.
• Build and support data ingestion, validation, tokenization, reformatting, and publishing workflows.
• Develop and maintain infrastructure using AWS CDK and Infrastructure as Code practices.
• Support and enhance distributed data processing solutions using Glue, Spark, Lambda, ECS, and Flink.
• Create and execute comprehensive unit, integration, and end-to-end testing strategies to ensure platform reliability.
• Participate in system testing and end-to-end validation efforts across multiple environments.
• Monitor, troubleshoot, and remediate production issues across pipelines and supporting infrastructure.
• Ensure compliance with enterprise data protection standards, including sensitive data handling and governance requirements.
• Address security findings, vulnerability remediation, and dependency management across the platform ecosystem.
• Collaborate with cross-functional teams to deliver scalable, maintainable, and high-performing data solutions.
• Contribute to CI/CD processes and software engineering best practices throughout the development lifecycle.

Required Qualifications

• Experience with Python development in enterprise-scale environments.
• Hands-on experience with AWS services, including: Lambda, ECS, Kinesis, S3, DynamoDB, IAM, and CDK.
• Proficiency with Spark and AWS Glue for large-scale data processing.
• Experience with Git, source control management, and CI/CD pipelines.
• Experience with Databricks.
• Experience building and supporting cloud-native data pipelines and data ingestion frameworks.
• Experience developing and maintaining Infrastructure as Code solutions.
• Understanding of data validation, transformation, and data quality practices.
• Knowledge of secure software development practices and vulnerability remediation.
• Experience supporting end-to-end testing, system testing, and production deployments.
• Problem-solving and troubleshooting skills in distributed systems environments.

Preferred Qualifications

• Experience working with financial services or highly regulated data environments.
• Experience supporting sensitive data handling, tokenization, or data governance initiatives.
• Familiarity with Apache Flink and real-time stream processing architectures.
• Experience building event-driven architectures using AWS services.
• Knowledge of containerized application deployments and orchestration platforms.
• Experience with automated testing frameworks and test-driven development practices.
• AWS certifications such as AWS Solutions Architect Associate or AWS Developer Associate.
• Experience working within Agile software development teams.