AWS Data Engineer

Overview:

Job Description - AWS Data Engineer (Python, AWS Glue, PySpark & Attunity)

Job Title

AWS Data Engineer - Python, AWS Glue, PySpark & Attunity

Job Summary

We are seeking an experienced AWS Data Engineer with strong expertise in Python, AWS Glue, PySpark, Attunity (Qlik Replicate), and AWS Cloud to design, develop, and maintain scalable enterprise data integration solutions. The ideal candidate will have hands-on experience building ETL pipelines, developing cloud-native data lakes, implementing real-time and batch data replication, and automating infrastructure using AWS services.

The role requires collaboration with business, analytics, and application teams to deliver secure, scalable, and high-performing data engineering solutions while following DevOps and CI/CD best practices.

Experience Required

• 5-8+ Years of experience in Data Engineering
• 3+ Years of experience with AWS Data Services
• Experience with enterprise ETL/ELT and cloud migration projects

Key Responsibilities

Data Engineering & ETL Development

• Design, develop, and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python .
• Build and optimize data transformation workflows for processing large-scale datasets.
• Develop reusable data ingestion and transformation frameworks.
• Implement batch and real-time data processing pipelines.

Data Integration

• Use Attunity (Qlik Replicate) for real-time and batch data replication.
• Integrate data from on-premises systems to AWS Cloud.
• Support heterogeneous source systems and cloud-based data platforms.

AWS Cloud & Data Lake

• Design and manage enterprise data lakes using Amazon S3 .
• Ensure data quality, security, governance, and lifecycle management.
• Implement serverless data processing using AWS Lambda .
• Integrate Glue, Lambda, S3, and other AWS services for end-to-end workflows.

Infrastructure & Automation

• Configure and manage AWS IAM roles, policies, and permissions.
• Automate infrastructure provisioning using AWS CloudFormation .
• Support Infrastructure-as-Code (IaC) initiatives.
• Participate in CI/CD pipeline implementation and deployment automation.

Performance & Support

• Monitor and optimize ETL jobs, Glue workflows, and cloud resources.
• Troubleshoot production issues and perform root cause analysis.
• Improve pipeline performance, reliability, and scalability.
• Maintain technical documentation and operational runbooks.

Collaboration

• Work with Business Analysts, Data Architects, Analytics teams, and Application Developers.
• Gather business requirements and translate them into technical solutions.
• Participate in Agile ceremonies, code reviews, and design discussions.

Mandatory Technical Skills

AWS Cloud

• Amazon S3
• AWS Glue
• AWS Lambda
• AWS IAM
• AWS CloudFormation

Data Engineering

• Python
• PySpark
• ETL / ELT
• Data Pipelines
• Data Integration
• Data Transformation

Data Replication

• Attunity (Qlik Replicate)
• Real-time Replication
• Batch Processing

Databases

• SQL
• Relational Databases
• Data Warehousing (preferred)

DevOps

• Git
• CI/CD
• Infrastructure as Code (IaC)
• CloudFormation

Monitoring & Operations

• Performance Tuning
• Troubleshooting
• Root Cause Analysis
• Technical Documentation

Preferred Skills

• AWS Step Functions
• Amazon Redshift
• Amazon Athena
• AWS Lake Formation
• Apache Spark
• Airflow
• Terraform
• Docker
• Kubernetes
• Data Quality Frameworks

Soft Skills

• Excellent communication skills
• Stakeholder management
• Strong analytical and problem-solving skills
• Leadership and ownership mindset
• Team collaboration
• Mentoring and coaching
• Planning and execution
• Risk management
• Customer-focused approach