Lead Data Platform Engineer - Astronomer / Airflow

Lead Data Platform Engineer – Astronomer / Airflow

CTG is seeking to fill a Lead Data Platform Engineer – Astronomer / Airflow position for our client. Location: Atlanta, GA Duration: 12 months Duties: Lead L2/L3 support for enterprise Astro (Astronomer-managed Apache Airflow) environments. Own Incident, Problem, and Change Management processes for the Data Orchestration Platform. Perform advanced root cause analysis (RCA) for pipeline failures, scheduler issues, worker instability, and infrastructure bottlenecks. Improve DAG reliability, SLA adherence, and reduce MTTR across critical data workflows. Establish and maintain operational standards, including support playbooks, runbooks, and escalation procedures. Design and maintain dynamic DAG frameworks (Factory pattern) to enable scalable pipeline onboarding. Enable automated onboarding of new data domains, data sources, and transformation workloads. Support and optimize complex ETL/ELT workflows across Data Lakes (S3-based architectures) and Data Warehouses (e.g., Snowflake, Amazon Redshift). Collaborate with data engineering, platform, and DevOps teams to ensure stable, scalable orchestration services. Provide technical guidance and support on open-source platform implementation, customization, and integration.

Skills: Deep expertise in Astronomer-managed Apache Airflow (Astro) environments. Strong Python development skills for DAGs, operators, plugins, and automation frameworks. Experience with ETL/ELT pipeline design and distributed data processing patterns. Proficiency with AWS cloud services including EKS, S3, RDS, IAM, CloudWatch, and VPC. Kubernetes operational experience (Airflow on EKS or equivalent). Infrastructure-as-Code experience (Terraform or CloudFormation preferred). Strong knowledge of CI/CD pipelines, Git-based workflows, and deployment automation. Familiarity with data platforms such as Snowflake and Amazon Redshift. Strong analytical and troubleshooting skills for distributed systems. Experience: Proven experience supporting and operating enterprise-scale Airflow/Astro environments. Experience leading production incident response, RCA, and operational reliability improvements. Hands-on experience building scalable DAG frameworks and reusable pipeline templates. Background in cloud-native data engineering and platform operations. Experience optimizing performance, scaling workloads, and improving orchestration efficiency. Prior experience in enterprise data platform support or SRE/DataOps roles strongly preferred. Education: Bachelor's degree in Computer Science, Information Systems, Engineering, or related field, or equivalent experience. Excellent verbal and written English communication skills and the ability to interact professionally with a diverse group are required.