Senior Data Engineer

Senior Data Engineer

We are seeking a highly skilled Senior Data Engineer with 8–12+ years of experience in designing and building enterprise-scale data platforms. The ideal candidate will have deep expertise in Snowflake, DBT, Python, PySpark, Qlik Replicate, Astronomer Airflow, and CI/CD to develop scalable, secure, and high-performance data pipelines supporting modern analytics and business intelligence initiatives.

The successful candidate will play a key role in architecting data ingestion frameworks, implementing data governance and security, managing schema evolution, and optimizing cloud-based data platforms.

Key Responsibilities

• Design, develop, and maintain scalable, resilient data pipelines using Snowflake features, including Snowpipe, Streams, Tasks, Dynamic Tables, and advanced SQL.
• Build and maintain DBT models with testing, documentation, lineage, and reusable transformations.
• Develop Python and PySpark-based ingestion frameworks for files, APIs, and enterprise data sources.
• Build ingestion pipelines for structured and semi-structured data formats, including CSV, JSON, XML, Excel, fixed-width files, and multi-record layouts.
• Design ingestion solutions for Mainframe VSAM/EBCDIC data sources.
• Implement metadata-driven schema evolution and schema drift detection strategies.
• Configure and manage Qlik Replicate for Change Data Capture (CDC) and database replication from Oracle, SQL Server, and DB2.
• Develop robust, auditable, and recoverable data replication pipelines.
• Implement Snowflake data governance using RBAC, masking policies, and security best practices.
• Apply Protegrity (or similar tokenization platforms) for sensitive data protection.
• Develop and orchestrate workflows using Astronomer Airflow with dependency management, scheduling, retries, and monitoring.
• Integrate data pipelines with GitLab and Azure DevOps CI/CD pipelines.
• Optimize Snowflake performance, compute resources, storage utilization, and query execution.
• Implement monitoring, alerting, logging, and operational dashboards for data pipelines.
• Collaborate with Data Architects, Analysts, Data Scientists, and business stakeholders to deliver reliable data solutions.

Required Skills

• 8–12+ years of experience in Data Engineering.
• Strong expertise in Snowflake architecture and development.
• Hands-on experience with: Snowpipe, Streams, Tasks, Dynamic Tables, Advanced SQL, Data Masking, RBAC, Performance Tuning.
• Strong experience with DBT (Data Build Tool).
• Proficiency in Python and PySpark.
• Experience with Qlik Replicate for CDC and database replication.
• Hands-on experience with Astronomer Airflow workflow orchestration.
• Experience implementing schema drift detection and schema evolution.
• Knowledge of GitLab, Git workflows, and CI/CD automation.
• Experience with REST APIs and enterprise data integration.
• Strong understanding of data governance, metadata management, and data quality.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong communication and collaboration skills.

Preferred Qualifications

• Experience with Protegrity or similar data security/tokenization platforms.
• Experience working with Oracle, SQL Server, and DB2 databases.
• Knowledge of Azure DevOps and cloud-native data engineering practices.
• Experience with enterprise-scale data lake and data warehouse implementations.
• Exposure to Agile/Scrum development methodologies.

Experience Required

• 8–12+ years of hands-on Data Engineering experience.