Data Tech Lead

Job Description

Data Tech Lead with AWS

Experience: +12 Years

Skills : Data engineer, Python, AWS, Apache Spark, Lambda

Location: Fort Mill SC

We at Coforge are hiring Data Tech Lead with the following skill sets.

Job Description

\n
• Lead the design, development, and support of scalable enterprise data platforms using AWS cloud-native services and modern data engineering technologies.
\n
• Guide Data Lake and Lakehouse implementation using Amazon S3, AWS Glue, Lake Formation, Athena, Postgres, Apache Iceberg, Delta Lake, and Parquet.
\n
• Develop and review batch, incremental, and Change Data Capture pipelines for data from databases, APIs, files, enterprise applications, and streaming platforms.
\n
• Build and optimize ETL and ELT workflows using AWS Glue, Apache Spark, PySpark, Python, SQL, Lambda, and distributed processing frameworks.
\n
• Create reusable frameworks for ingestion, transformation, validation, reconciliation, exception handling, error recovery, and operational monitoring.
\n
• Implement real-time and near-real-time processing using Kafka, Amazon MSK, Amazon Kinesis, Lambda, SNS, SQS, and EventBridge.
\n
• Lead workflow orchestration using Apache Airflow, Amazon MWAA, AWS Step Functions, and event-based scheduling.
\n
• Implement metadata, schema, catalog, lineage, and discovery capabilities using AWS Glue Data Catalog and Lake Formation.
\n
• Improve performance and cost efficiency through partitioning, compaction, retention, lifecycle management, storage optimization, Spark tuning, and query optimization.
\n
• Enforce data quality controls covering completeness, accuracy, consistency, integrity, uniqueness, and source-to-target reconciliation.
\n
• Implement secure and governed access using IAM, KMS, S3 policies, Lake Formation permissions, encryption, and fine-grained controls.
\n