Senior Data Engineer

Data Engineer

We are seeking an experienced Data Engineer to design, build, and enhance data ingestion pipelines and metadata frameworks that support data lineage, modernization, and analytics enablement. The ideal candidate has hands-on experience with AWS Glue, Python/PySpark, ETL pipelines, and metadata management tools, with the ability to work in a dynamic, cloud-centric data environment.

Key Responsibilities

• Design, develop, and maintain data ingestion and ETL pipelines for metadata integration across systems like SQL Server, Informatica, AWS Glue, and Azure Data Factory.
• Implement end-to-end data lineage tracing, including source-to-target mapping, column-level lineage, and job dependency tracking.
• Automate metadata ingestion and transformation for lineage reporting and quality validation.
• Identify static, orphaned, or unused tables/jobs and contribute to data-quality improvement.
• Collaborate with data architects, analysts, and AI strategist to align metadata ingestion with modernization goals.
• Participate in the deployment and hardening of solutions in AWS (and Azure as applicable).
• Contribute to modernization and AI-readiness initiatives through metadata standardization and automation.

Required Skills & Experience

• 8–10 years of experience in Data Engineering / ETL Development.

• Strong experience with:

  • AWS Glue, S3, Redshift, Lambda

  • SQL (SQL Server, Oracle, or Snowflake) – complex queries, stored procedures, performance tuning

  • Python / PySpark for metadata parsing and ETL automation

  • ETL Tools: Informatica, AWS Glue, Azure Data Factory

  • Data Lineage / Metadata Management: Collibra, Alation, or custom lineage repositories

  • CI/CD & Orchestration: GitHub, Jenkins, or Azure DevOps

Nice to Have

• Experience with BladeBridge or similar lineage analysis tools
• Familiarity with Client Batch Processing
• Exposure to Airflow, dbt, or Databricks for orchestration and transformation
• Basic understanding of AI/ML data pipeline readiness
• Azure data platform exposure (ADF, Synapse, Data Lake)