Senior Data Engineer

Cloud Platform: AWS (Compute, Storage, Networking, IAM and related cloud resources) Data Engineering Skills: Python, PySpark, SQL, Prefect, Snowflake, Snowpark, dbt, airbyte, Databricks on AWS DevOps Skills: CI/CD (AWS CodePipeline/CodeBuild/CodeDeploy), Terraform (IaC), Docker, Amazon CloudWatch, Secrets Management, Release & Environment Management Must-Have Skills B.E./B.Tech degree in Computer Science, Engineering, or a related field, with 8-10 years of overall work experience. 5+ years of hands‑on development experience building Cloud Data Platform Data Engineering solutions covering Data Ingestion, Data Quality Validations, Data Processing and Data Integration. Strong hands‑on coding experience in Python, PySpark, and SparkSQL, with solid software engineering practices including testing, version control and code reviews. Hands‑on experience in Airbyte, Snowflake and dbt. Hands‑on experience with Prefect for workflow orchestration, including designing flows and tasks, scheduling, deployments, and parameterized runs. Ability to build reliable, observable Prefect workflows with retry logic, failure handling, and rerun/recovery support for production pipelines. Hands‑on experience with Amazon S3-based data lakes, Databricks on AWS and other AWS-based data ecosystem services. Experience monitoring and troubleshooting orchestrated workflows via the Prefect UI/Cloud, including work pools, deployments and run history. Demonstrated willingness and ability to set up and own DevOps practices for the platforms you build (see DevOps Skills below). Proficiency in analytics use‑case analysis, source system analysis, and data quality assessment. Experience coordinating/collaborating with on‑shore and off‑shore teams for solution delivery. Excellent communication and presentation skills. DevOps Skills Design and build CI/CD pipelines using AWS CodePipeline, CodeBuild, and CodeDeploy (or equivalent tools such as GitHub Actions/Jenkins) to automate build, test, and release cycles. Provision and manage AWS cloud resources using Infrastructure as Code (Terraform), including version‑controlled, reusable modules. Containerize applications and workflows with Docker; deploy and manage containers using Amazon ECS/EKS. Manage promotion of code and configuration across Development, Test, and Production environments with clear release and rollback strategies (blue‑green/canary deployments). Implement secure secrets and configuration management using AWS Secrets Manager or Parameter Store, with least‑privilege IAM policies. Set up monitoring, logging, and alerting using Amazon CloudWatch (metrics, dashboards, alarms) to maintain operational visibility into pipelines and workflows. Define and implement retry, recovery, and rerun strategies for failed jobs/workflows to ensure production reliability. Write automation scripts (Python/Bash) for deployment, operational tasks, and routine platform maintenance. Collaborate with data engineers, application developers, and cloud engineers to support end‑to‑end delivery, and troubleshoot production issues when required. Nice-to-Have Skills Experience with Snowpark, and additional orchestration frameworks such as Amazon MWAA (Managed Airflow) or AWS Step Functions. Exposure to Docker and containerized deployments; familiarity with Amazon ECS/EKS (Kubernetes) is a plus. Experience with event‑driven architectures (Amazon EventBridge, SQS, SNS, Lambda) and REST API integration across distributed services. Exposure to Data Management principles including Data Governance, Data Cataloging, and Master Data Management. Exposure to cloud‑native MDM tooling. Exposure to BI analytics using Amazon QuickSight. #J-18808-Ljbffr