Data Engineer

Data EngineerWe are looking for a highly skilled Data Engineer with 10+ years of experience in designing, building, and supporting large-scale data platforms and pipelines. The ideal candidate will have deep expertise in Python, PySpark, Apache Airflow, SQL, and AWS, along with hands-on experience in troubleshooting production data pipeline issues and optimizing distributed data processing workloads.The candidate will be responsible for developing scalable data solutions, orchestrating workflows, managing cloud-based datasets, and ensuring reliable delivery of business-critical data across enterprise systems.Key ResponsibilitiesDesign, develop, and maintain robust data pipelines using Python and PySpark.Build containerized data processing applications that ingest, transform, and publish data across enterprise platforms.Develop and support Apache Airflow DAGs, custom operators, scheduling frameworks, and workflow orchestration processes.Implement complex business rules, data transformations, reconciliation processes, and effective-dating logic.Work with AWS S3, Delta Lake, Parquet datasets, and PostgreSQL databases for scalable data storage and processing.Optimize Spark applications by identifying and resolving memory, performance, and scalability bottlenecks.Manage schema changes, database migrations, and data quality validation processes.Develop and maintain Docker containers and Kubernetes-based deployments.Support CI/CD and multi-environment deployment pipelines.Troubleshoot production incidents, perform root cause analysis, and implement preventive solutions.Integrate with external platforms and APIs for data import/export and enterprise data exchange.Create runbooks, technical documentation, monitoring processes, and data lineage artifacts.Collaborate closely with business stakeholders, architects, and cross-functional engineering teams.Required Skills & Qualifications10+ years of experience in Data Engineering and software development.Strong hands-on expertise in Python development, including modular, configuration-driven, and unit-tested applications.Extensive experience with Apache Spark / PySpark for building and tuning distributed data processing workloads.Proven experience developing and maintaining Apache Airflow workflows and DAGs.Strong SQL skills with expertise in relational database design and data modeling.Experience with PostgreSQL, schema migrations, and data reconciliation processes.Hands-on experience with AWS S3 and cloud-based data platforms.Experience troubleshooting production data pipeline failures and performing root cause analysis.Strong understanding of ETL/ELT frameworks and large-scale data processing architectures.Experience with Git, CI/CD processes, and software development best practices.Preferred QualificationsExperience with Docker and Kubernetes.Knowledge of Delta Lake and Parquet-based data architectures.Experience integrating enterprise applications through APIs.Familiarity with data governance, lineage, and monitoring frameworks.Exposure to Anaplan integrations and bulk data import/export processes.Experience working within Agile development environments.