Data Engineer

Data Engineer We are seeking a skilled Data Engineer to design, build, and maintain high-quality, scalable data pipelines with a strong focus on Silver and Gold layer transformations, data quality automation, and CI/CD-driven deployments. The ideal candidate will have hands-on experience with Python, PySpark, Git-based CI/CD pipelines, ARM templates, and data quality libraries such as Great Expectations. Roles & Responsibilities: Design and implement Silver and Gold layer data models following medallion architecture best practices. Perform data cleansing, standardization, enrichment, and aggregation to support analytics and reporting. Build optimized PySpark-based transformations for large-scale data processing. Ensure data consistency, performance, and scalability across datasets. Build and maintain CI/CD pipelines using Git-based workflows (Azure DevOps / GitHub). Use ARM templates (or IaC equivalents) for automated infrastructure provisioning. Enable automated deployment of data pipelines, notebooks, and configurations. Follow DevOps best practices for version control, branching, and release management. Develop reusable Python-based automation scripts for pipeline orchestration, validation, and monitoring. Create modular, maintainable, and testable Python code. Support automation of metadata, logging, alerting, and operational tasks. Implement data quality libraries such as Great Expectations. Define and automate data quality rules (completeness, accuracy, freshness, consistency). Integrate data quality checks into data pipelines and CI/CD workflows. Monitor, log, and troubleshoot data quality issues proactively. Work closely with data architects, analysts, QA, and business stakeholders. Translate business and analytical requirements into robust data engineering solutions. Participate in Agile ceremonies and support sprint-based delivery. Provide production support, performance tuning, and troubleshooting. JConnect Infotech