Data Tech Lead

Data Tech Lead with AWS

Experience: +12 Years

Skills

Data engineer, Python, AWS, Apache Spark, Lambda

Location: Fort Mill SC

Are you the right applicant for this opportunity Find out by reading through the role overview below.

We at Coforge are hiring Data Tech Lead with the following skill sets.

Job Description

• Lead the design, development, and support of scalable enterprise data platforms using AWS cloud-native services and modern data engineering technologies.
• Guide Data Lake and Lakehouse implementation using Amazon S3, AWS Glue, Lake Formation, Athena, Postgres, Apache Iceberg, Delta Lake, and Parquet.
• Develop and review batch, incremental, and Change Data Capture pipelines for data from databases, APIs, files, enterprise applications, and streaming platforms.
• Build and optimize ETL and ELT workflows using AWS Glue, Apache Spark, PySpark, Python, SQL, Lambda, and distributed processing frameworks.
• Create reusable frameworks for ingestion, transformation, validation, reconciliation, exception handling, error recovery, and operational monitoring.
• Implement real-time and near-real-time processing using Kafka, Amazon MSK, Amazon Kinesis, Lambda, SNS, SQS, and EventBridge.
• Lead workflow orchestration using Apache Airflow, Amazon MWAA, AWS Step Functions, and event-based scheduling.
• Implement metadata, schema, catalog, lineage, and discovery capabilities using AWS Glue Data Catalog and Lake Formation.
• Improve performance and cost efficiency through partitioning, compaction, retention, lifecycle management, storage optimization, Spark tuning, and query optimization.
• Enforce data quality controls covering completeness, accuracy, consistency, integrity, uniqueness, and source-to-target reconciliation. xhyhwjd
• Implement secure and governed access using IAM, KMS, S3 policies, Lake Formation permissions, encryption, and fine-grained controls.