Senior Data Engineer

Senior Data EngineerWe are seeking an experienced Senior Data Engineer to design, develop, and maintain scalable, reliable, and high-performance data platforms and pipelines that support business intelligence, analytics, and data-driven decision-making.The ideal candidate will have strong hands-on expertise in Python, Apache Spark, Apache Airflow, Amazon Redshift, SQL, and modern data engineering practices, along with mandatory experience in the Healthcare domain. The candidate should have experience building end-to-end ETL/ELT pipelines, processing large-scale datasets, implementing data quality frameworks, and supporting enterprise analytics and reporting needs.This role requires a technically strong professional who can collaborate with business stakeholders, data analysts, architects, and application teams to deliver robust and scalable data solutions.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines and ETL/ELT workflows for structured and unstructured data.Build and optimize data ingestion, transformation, and data integration processes using Python and Apache Spark.Develop, schedule, and monitor workflows using Apache Airflow.Design and manage enterprise data warehouse solutions using Amazon Redshift.Implement data models, data marts, and reporting datasets to support analytics and business intelligence requirements.Optimize data processing jobs and database performance for scalability and efficiency.Work with large healthcare datasets, ensuring data accuracy, consistency, and compliance with industry standards.Collaborate with business stakeholders to understand reporting and data requirements.Implement data quality checks, validation frameworks, and monitoring processes.Troubleshoot data pipeline issues and perform root cause analysis.Ensure data security, governance, and compliance requirements are adhered to.Participate in code reviews, development best practices, and continuous improvement initiatives.Create and maintain technical documentation, data flow diagrams, and data mapping documents.Work closely with cross-functional teams including Data Science, Analytics, Product, and Engineering teams.Skills, Knowledge & ExperienceMandatory Skills6+ years of experience in Data Engineering, Big Data Engineering, or related roles.Mandatory experience working in the Healthcare domain.Strong hands-on experience in Python development for data engineering solutions.Strong hands-on experience with Apache Spark (PySpark) for large-scale data processing.Hands-on experience with Apache Airflow for workflow orchestration and scheduling.Strong experience designing, developing, and optimizing solutions on Amazon Redshift.Advanced SQL skills with experience in query tuning and performance optimization.Experience building ETL/ELT pipelines and modern data integration frameworks.Strong understanding of data warehousing concepts, dimensional modelling, and data architecture.Experience handling large-scale datasets and complex data transformation requirements.Strong debugging, troubleshooting, and performance tuning skills.Experience working with cloud-based data platforms and services (AWS or Azure) is preferred.Excellent analytical and problem-solving skills.Strong communication and stakeholder management capabilities.Preferred SkillsExperience with AWS data services such as S3, Glue, EMR, Athena, Lambda, or KinesisExperience with Azure Data Factory, Azure Databricks, ADLS, Synapse Analytics, or other Azure data servicesExposure to Databricks, Snowflake, or modern Lakehouse architecturesExperience with CI/CD for Data Engineering pipelinesKnowledge of Healthcare data standards such as HL7, FHIR, EHR/EMR, Claims, Provider, Member, or Clinical DataExperience with data governance, data quality, and compliance frameworksExperience with Git and DevOps best practices for data engineeringThis job description is not intended to cover or contain a comprehensive listing of all responsibilities, duties, or activities that are required. Responsibilities, duties, and/or activities may change, or new ones may be added at any time with or without notice.If you are a motivated professional with a passion for delivering impactful solutions, we'd love to hear from you. Apply today to be part of a dynamic and forward-thinking team at NextGen Invent.