Sr. AWS Data Engineer

Sr. AWS Data Engineer

Location: Fort Mill, SC

Work mode: Onsite

Key Responsibilities

• Design, build, and support scalable data pipelines using AWS Glue , Lambda , and EventBridge for event-driven and batch processing .
• Build and maintain AWS Glue ETL jobs using PySpark/Python for ingestion, transformation, and curation across data lake layers .
• Develop and manage Glue Workflows and AWS orchestration to coordinate workflows, triggers, and dependencies across AWS services .
• Implement near real-time data processing patterns using AWS Lambda and EventBridge .
• Write, optimize, and maintain complex Postgres SQL for validation, transformation, reporting, and performance tuning.
• Manage metadata and schema definitions in AWS Glue Data Catalog to support governance and discoverability.
• Monitor and troubleshoot pipeline performance using CloudWatch , logs, alerts, and AWS monitoring tools.
• Contribute to system reliability , scalability , and performance optimization of the data platform.

Required Qualifications

• 10+ years of data engineering experience with production pipelines , ETL/ELT , and large-scale data migrations .
• Strong hands-on Python and PySpark/Spark development experience, including AWS Glue ETL jobs for ingestion, transformation, and curation across data lake layers .
• Production experience with AWS data services such as Glue , S3 , IAM , Lambda , EventBridge , CloudWatch , logs/alerts , Secrets Manager , and Glue Data Catalog .
• Experience with batch and event-driven data architectures , including near real-time processing patterns and pipeline design.
• Glue Workflow and AWS Orchestrator experience, including workflow orchestration, triggers, dependencies, and cross-service orchestration across AWS services .
• Advanced SQL skills with Postgres/PostgreSQL and/or Amazon Aurora , including complex transformations, performance tuning , validation, reporting, loading, and schema management .
• Ability to manage metadata and schema definitions using AWS Glue Data Catalog to support governance and discoverability.
• Strong understanding of migration reliability practices, including delta loads , reconciliation , error handling , restart/checkpointing , rollback planning , and idempotency .
• Knowledge of data security , PII handling , encryption , tokenization , least-privilege access , secrets management , and metadata governance .
• Hands-on CI/CD and Git experience using GitHub Actions , Octopus Deploy , or similar tools for deployment automation.
• Infrastructure as Code experience with Terraform or similar tools.
• Ability to monitor , troubleshoot , support, and remediate production data pipeline issues independently using AWS monitoring tools.
• Ability to contribute to system reliability , scalability , and performance optimization of the data platform.
• Experience using AI-assisted development tools such as GitHub Copilot , Microsoft Copilot , or Cursor AI to improve coding productivity, documentation, testing, and troubleshooting.
• Strong collaboration skills with BI , analytics , Snowflake , and downstream data consumers.

Preferred Skills

• Snowflake experience, including data consumption, warehouse integration, and collaboration with downstream analytics or BI teams.
• Experience with Athena Federated Query , Glue crawlers , Lake Formation , SNS , CloudTrail , Parquet , Iceberg , or Delta .
• OAuth 2.0 , secure API integration , tokenization , or regulated-data experience.
• Financial services background; ServiceNow or formal change-management experience preferred.
• Data modeling , partitioning , schema evolution , test-driven development , or static-analysis experience.
• AWS certification is a plus.