AWS Data Engineer / Data Governance

ROPES AI - TORC TEST !

About the Role

We are seeking a Senior Data Engineer specializing in Data Quality and Data Governance to design, build, and scale our next-generation data platform. In this role, you will lead the transition from ad-hoc data processing to a production-grade, highly reliable data platform.

You will work at the intersection of platform engineering and governance, establishing robust data contracts, automated data testing, and continuous monitoring across our enterprise systems.

Key Responsibilities

• Data Platform Development: Architect, implement, and maintain scalable, production-grade ETL/ELT data pipelines and platforms using Python, PySpark, and SQL on AWS.
• Data Quality & Checks: Design and deploy automated data quality testing, proactive monitoring, and anomaly detection across all data layers to ensure accuracy, completeness, and freshness.
• Data Contracts & Governance: Standardize and enforce data contracts between data producers and consumers to eliminate schema drift and ensure clear operational ownership.
• Governance Tooling Integration: Implement and manage modern data quality and governance frameworks using tools such as Soda , Gable , or similar modern data stack utilities (e.g., Great Expectations, Monte Carlo).
• Productionization & Optimization: Transition legacy or prototype data pipelines into reliable, maintainable, and cost-effective production systems adhering to CI/CD and software engineering best practices.
• Technical Leadership: Partner with cross-functional analytics, data science, and product teams to embed data quality standards into every stage of the software development lifecycle.

Required Qualifications

• Experience: 7+ years of hands-on data engineering experience, with a strong focus on data quality, data governance, and data pipeline reliability.
• Core Tech Stack: Expert-level proficiency in Python, SQL, and PySpark.
• Cloud Infrastructure: Extensive hands-on experience building production data architectures on AWS (e.g., S3, Glue, EMR, Redshift, Athena, Lambda).
• Data Quality Frameworks: Direct experience developing data checks and using data quality/contract tools like Soda , Gable , Great Expectations, or custom automated frameworks.
• Data Modeling & Architecture: Strong knowledge of data warehousing, lakehouse architectures, schema evolution, and data contract implementations.
• Engineering Standards: Experience with software engineering best practices, including Git, CI/CD pipelines, containerization (Docker), and orchestration tools (e.g., Apache Airflow, Prefect, or Dagster).

Preferred Qualifications

• Experience with automated data lineage and metadata management platforms.
• Familiarity with streaming architecture technologies (e.g., Apache Kafka, AWS Kinesis).
• Background in establishing SLAs, SLOs, and error budgets specifically tailored for data quality metrics.