Senior Data Engineer

Role Summary

We are looking for a driven and hands-on Data Engineer to design, implement, and support scalable, production-level data pipelines within our AWS-based data ecosystem. This position is responsible for full lifecycle pipeline delivery-from ingesting data from enterprise systems to producing analytics-ready datasets-while collaborating closely with Data Architects, Analytics teams, and business partners. The role is critical in ensuring reliable, high-quality data is available across the organization to power analytics and informed decision-making. As a member of the core data platform team, you will also contribute to advancing our AWS Lakehouse architecture and establishing engineering best practices.


Key Responsibilities

• Pipeline Engineering: Develop and maintain scalable ELT pipelines using reusable ingestion frameworks that support both batch and event-driven processing across multiple data sources such as ERPs, APIs, vendor feeds, and relational systems.
• AWS Data Platform Development: Design and enhance AWS-native data solutions aligned with a Medallion (Bronze/Silver/Gold) Lakehouse architecture, with an emphasis on performance optimization and cost management.
• Infrastructure & DevOps: Build and manage AWS infrastructure using Terraform, and support CI/CD processes through GitOps methodologies, including automated testing, monitoring, alerting, and system recovery capabilities.
• Data Modeling & Transformation: Create and maintain dimensional models and Gold-layer datasets using SQL, Python, and PySpark. Implement scalable ingestion processes with strong handling of schema evolution, auditing, and performance tuning techniques such as partitioning, clustering, and materialization.
• Data Quality & Governance: Integrate automated data validation, anomaly detection, and lineage tracking into pipelines, while contributing to metadata management practices.
• Reporting & BI Enablement: Diagnose and resolve complex data issues, ensuring pipelines deliver data optimized for Power BI. Work closely with BI developers to align data models with reporting needs and troubleshoot dashboard-related challenges.
• Cross-Functional Collaboration: Partner with architecture and analytics teams to translate business requirements into technical solutions, and actively participate in code reviews, sprint planning, and architectural discussions.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related discipline, or equivalent practical experience.
• At least 7 years of experience in data engineering, with strong hands-on expertise in AWS services and distributed data technologies.
• Advanced proficiency in Python and SQL, including experience with Spark/PySpark.
• Proven experience building and maintaining AWS-based data pipelines using services such as Glue, Step Functions, Lambda, S3, Athena, SNS, SQS, and Redshift.
• Experience with event-driven data architectures.
• Hands-on experience processing and managing large-scale vendor data feeds.
• Practical knowledge of Medallion architecture within a data lake or Lakehouse environment.
• Experience using Terraform for infrastructure-as-code deployments.
• Familiarity with CI/CD tools such as Bitbucket, GitHub, or AWS CodePipeline for pipeline deployment.
• Strong understanding of data warehousing principles, including star schema design, dimensional modeling, and slowly changing dimensions (SCD).
• Experience integrating data with Power BI or comparable business intelligence tools.
• Working knowledge of UNIX/Linux environments, including shell scripting.
• Experience supporting production systems, including monitoring, troubleshooting, and on-call responsibilities.
• Familiarity with Agile development methodologies.

Preferred Qualifications

• Experience integrating data from Oracle EBS.
• Familiarity with data quality tools such as Great Expectations or dbt testing frameworks.
• Experience with AWS CDK or CloudFormation alongside Terraform.
• Knowledge of data cataloging and lineage tools such as Alation or Collibra.
• AWS certifications (e.g., AWS Certified Data Engineer, AWS Certified Solutions Architect).