Data Engineer -Iceberg

Data Engineer

Client: Elevance Health



Location: Atlanta, GA



Rate: $60 65/hr on C2C



Experience: 8-12+ years

Job Description

We are seeking an experienced Data Engineer with strong hands-on experience in SQL, Python, SSIS, Airflow, Apache Spark, Apache Iceberg, and Databricks . The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions in a cloud-based environment.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, SSIS, Airflow, and Spark.
• Develop complex SQL queries, stored procedures, data transformations, and performance optimization solutions.
• Build and maintain data pipelines and workflows using Apache Airflow , including scheduling, dependencies, monitoring, retries, and failure handling.
• Develop and optimize Apache Spark/PySpark applications for large-scale data processing.
• Work with Databricks to develop, deploy, and optimize data engineering workloads.
• Work with Apache Iceberg tables and modern data lake/lakehouse architectures.
• Migrate and modernize legacy SSIS ETL workflows into cloud-based and Spark/Databricks-based data pipelines.
• Implement data ingestion, transformation, validation, and integration processes from multiple sources.
• Optimize Spark jobs, SQL queries, data partitioning, and storage strategies for performance and scalability.
• Implement data quality checks, error handling, logging, monitoring, and pipeline recovery mechanisms.
• Collaborate with data architects, developers, analysts, and business teams to understand requirements and deliver reliable data solutions.
• Follow best practices for data security, governance, performance, and operational reliability .

Required Skills

• Strong hands-on experience with SQL
• Strong programming experience with Python
• Hands-on experience with Apache Spark / PySpark
• Experience with Databricks
• Strong experience with Apache Airflow
• Hands-on experience with SSIS
• Experience with Apache Iceberg
• Strong understanding of ETL/ELT and data pipeline development
• Experience working with large-scale datasets and distributed data processing
• Strong troubleshooting and performance-tuning skills
• Good understanding of modern data lake/lakehouse architectures