Senior Data Engineer - Spark / Kafka / NiFi
Full-time
Mastercard
Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive. Job Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi.
The ideal candidate will have a strong background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing.
The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments. Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.
Contribute to architecture discussions, technical designs, and engineering standards. Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling. Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operational tooling to improve platform reliability.
Monitor production environments and proactively address operational concerns.
Participate in troubleshooting, root-cause analysis, and incident resolution activities. Engineering Excellence
Participate in code reviews and promote engineering best practices.
Maintain high-quality documentation for systems, APIs, and platform components.
Collaborate closely with Product, Architecture, Platform, and DevOps teams.
Contribute to CI/CD processes and continuous improvement initiatives.
Mentor junior engineers and foster a culture of technical excellence and innovation. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related technical field.
6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
Strong expertise in Apache Spark (Batch and Structured Streaming).
Strong experience with Apache Kafka and event-driven architectures.
Experience developing, testing and supporting Apache NiFi data flows.
Proficiency in Scala, pyspark or Python.
Strong SQL and data modeling skills.
Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores.
Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
Strong analytical, problem-solving, and communication skills.
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments. Preferred Qualifications
Experience building observability solutions using monitoring and logging platforms.
Knowledge of data governance, metadata management, and data platform best practices.
Experience with containerization and orchestration technologies (Docker, Kubernetes).
Knowledge of performance engineering, resilience testing, and production readiness assessments.
Experience building engineering automation frameworks and reusable validation platforms.
Familiarity with observability tools, monitoring systems, and operational analytics.
Experience in highly regulated, transaction-processing, or large-scale enterprise environments. Corporate Security Responsibility
Senior Data Engineer - Spark / Kafka / NiFi
Job Posting Title: Senior Data Engineer - Spark / Kafka / NiFi
Who is Mastercard?Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive. Job Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi.
The ideal candidate will have a strong background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing.
The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments. Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.
Contribute to architecture discussions, technical designs, and engineering standards. Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling. Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operational tooling to improve platform reliability.
Monitor production environments and proactively address operational concerns.
Participate in troubleshooting, root-cause analysis, and incident resolution activities. Engineering Excellence
Participate in code reviews and promote engineering best practices.
Maintain high-quality documentation for systems, APIs, and platform components.
Collaborate closely with Product, Architecture, Platform, and DevOps teams.
Contribute to CI/CD processes and continuous improvement initiatives.
Mentor junior engineers and foster a culture of technical excellence and innovation. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related technical field.
6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
Strong expertise in Apache Spark (Batch and Structured Streaming).
Strong experience with Apache Kafka and event-driven architectures.
Experience developing, testing and supporting Apache NiFi data flows.
Proficiency in Scala, pyspark or Python.
Strong SQL and data modeling skills.
Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores.
Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
Strong analytical, problem-solving, and communication skills.
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments. Preferred Qualifications
Experience building observability solutions using monitoring and logging platforms.
Knowledge of data governance, metadata management, and data platform best practices.
Experience with containerization and orchestration technologies (Docker, Kubernetes).
Knowledge of performance engineering, resilience testing, and production readiness assessments.
Experience building engineering automation frameworks and reusable validation platforms.
Familiarity with observability tools, monitoring systems, and operational analytics.
Experience in highly regulated, transaction-processing, or large-scale enterprise environments. Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard’s security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
Vacancy posted 26 days ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer - Spark / Kafka / NiFi in India vacancy
- ...We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance... ..., with hands-on expertise in Spark and Scala . You will work on designing... ...Familiarity with streaming technologies such as Kafka . Experience working on large-scale...SeniorFull time
Prox Works
Indiaa month ago - ...their greatest potential. Title and Summary Senior Data Engineering Overview Ethoca, a Mastercard company, is... ...Snowflake, Databricks, Apache Airflow, Apache NiFi, Azure Data Factory (ADF), UDAP, SQL, Python, Spark, Azure Cloud Services, File Transfer Technologies...SeniorFull timeWorldwide
Mastercard
India15 days ago - ...their greatest potential. Title and Summary Senior Data Engineer Overview We are the global technology company... ...• High proficiency in using Python or Scala, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi, Scoop), SQL to build Big Data products &...SeniorFull timeWorldwideFlexible hours
Mastercard
Indiaa month ago - ...of Gen. About the Role We are seeking an experienced Senior Principal Data Engineer to serve as a technical leader responsible for architecting... ...Streaming & Real-time Processing: Experience with Apache Kafka, event-driven architectures, and real-time analytics pipelines...SeniorFull timeFlexible hours
MoneyLion
India25 days ago - ...governments realize their greatest potential. Title and Summary Senior Data Engineer Data Engineer / Machine Learning Engineer Level 7 |... ...building data pipelines using distributed processing (e.g., Spark,PySpark,HADOOP). • Experience in using data technology...SeniorFull timeWorldwide
Mastercard
India22 days ago - ...governments realize their greatest potential. Title and Summary Senior Data Engineer, Data Engineering Job Description Summary Who is... ...have) with data integration tools – ETL/ELT tools (i.e. Apache NiFi, Azure Data Factory, Pentaho, Talend) • Experience with Graph...SeniorFull timeWorldwide
Mastercard
India15 days ago - ...of Gen. About the Role We are seeking an experienced Senior Manager, Data Engineering to serve as a technical people leader responsible for owning... ...and cost perspectives. Expertise with Apache Kafka, real-time data processing, and event-driven architectures...SeniorFull timeFlexible hours
MoneyLion
India25 days ago - ...Title and Summary Principal Data Engineer Overview We are seeking... ...technologies including Apache Spark, Apache Iceberg, and AWS... ...driven architectures, Apache Kafka, and enterprise data integration... ...Leadership • Mentor and guide senior engineers and technical teams...Full timeWorldwide
Mastercard
India15 days ago - ...potential. Title and Summary Principal Data Engineer Principal Data Engineer | Data... ...daily. This role serves as a senior technical leader responsible for... ...technologies such as: Apache Spark Databricks Hadoop Ecosystem Kafka or Streaming Platforms...Full timeWorldwide
Mastercard
India26 days ago - ...Required Qualifications Strong hands-on Data Engineering experience. Expert-level experience with Databricks . Strong hands-on... ...environments. Ideal Candidate The ideal candidate is a senior-level , hands-on Databricks Data Engineer . They should be...Senior
Enable-data
India5 days ago - ...designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data... ...optimize distributed data applications using Spark and Scala . Develop reliable, high-performance...SeniorFull time
Proximity
India28 days ago - ...of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Data Engineer Role: Senior Data Engineer Role Overview We are seeking a Senior Data Engineer to drive AI-led transformation across...SeniorFull timeWorldwide
Mastercard
Indiaa month ago - ...greatest potential. Title and Summary Sr Data Engineer Overview The AI Innovation at... ...continuous improvement. Role As a Senior AI Engineer, you will: • Be an integral... ...data platforms (e.g., Databricks, Apache Spark) at petabyte scale, pushing the boundaries...SeniorFull timeWorldwide
Mastercard
India15 days ago - ...of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Data Engineer Job Description Summary We believe in power of data. We think creatively, design smart, simple to use, cost-effective...SeniorFull timeTemporary workWork experience placementWorldwide
Mastercard
India8 days ago - ...potential. Title and Summary Senior Data Scientist Our Purpose:... ...in Python and experience with Spark, Hive, Scala, Impala, or similar... ...such as Apache Airflow, NiFi, or equivalent technologies.... ...Science, Statistics, Mathematics, Engineering, Economics, Finance, or a...SeniorFull timeWorldwide
Mastercard
Indiaa month ago - ...potential. Title and Summary Analyst, Big Data Analytics & Engineering About Mastercard Mastercard is a... ...data processing jobs using PySpark/Spark, Python, and SQL (including engines such... ...tools such as Apache Airflow, Apache NiFi, Azure Data Factory, Pentaho, or...Full timeWorldwideFlexible hours
Mastercard
India20 days ago - ...and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Analyst, Big Data Analytics & Engineering About Mastercard: Mastercard is a global technology leader in the payments industry, committed to...SeniorFull timeWorldwide
Mastercard
India8 days ago - ...potential. Title and Summary Principal Data Engineer Who is Mastercard? Mastercard is a... ...processing frameworks (e.g., Apache Spark or equivalent) • Deep hands-on experience... ...skills / experience communicating with senior leaders and executives • Previous leadership...Full timeWorldwide
Mastercard
Indiaa month ago - ...We are looking for an experienced Data Engineer to design, develop, and optimize scalable data platforms and modern cloud-based data solutions... ...integration solutions. ~ Proficiency in Python , Apache Spark, or similar technologies for large-scale data processing. ~...Full time
Weekday
Indiaa month ago - ...greatest potential. Title and Summary Data Engineer II Data Engineer, DPO Who is Mastercard... ...and global teams. Collaborate with senior and peer engineers on multiple projects... ...pipelines. • Experience with Databricks and Spark is a plus. • Demonstrate curiosity...Full timeLocal areaWorldwideFlexible hours
Mastercard
India8 days ago - ...seeking a highly skilled, execution-focused Senior Data Scientist with a minimum of 5 years of... ...and frameworks across Python, SQL, Spark, and AWS. The role requires strong technical... ...and orchestration. Collaborate with engineering teams to integrate models into cloud-based...SeniorFull time
Proximity
Indiaa month ago - ...As a Lead Data Engineer , you will build and lead the data infrastructure powering... ...and intelligently. As the first senior data hire, you will define the data... ...technologies is highly desirable: Apache Spark , Apache Airflow , Kafka , and Elasticsearch or...Full time
Smartworking
India15 days ago - ...potential. Title and Summary Senior Software Engineer Overview Mastercard... ...Software Engineer in Data Collection & Engineering Services... ...Stacks ( Hadoop ecosystem, Spark is a plus) ~ Good to have... ...relevant frameworks ( Kafka and NiFi is a plus) ~ A deep expertise...SeniorFull timeWork experience placementWorldwide
Mastercard
India15 days ago - ...someone with ETL tool experience, since they don't use any ETL tools. Their platform is homegrown, so they need someone with strong data platform development skills, particularly in SQL and Python, who can easily navigate a large codebase and take ownership of managing...Senior
- ...potential. Title and Summary Data Scientist II Data Scientist... .... We are a team on analytics engineers, data architects, BI developers... ...Python, Scala, Hive, Impala, Spark, Cloud, and other related technologies... ...tools such as Apache Airflow, NIFI, and other related tools....Full timeWorldwide
Mastercard
India15 days ago - ...Req ID: 383431 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want... ...organization, apply now. We are currently seeking a Data Engineer Senior Consultant to join our team in Remote or PAN India NTT Offices...SeniorWork at officeRemote workFlexible hours
- ...We are seeking an experienced Snowflake Data Engineer with 5–8 years of hands-on experience in data engineering, data warehousing, and large-scale data migration. The ideal candidate will have strong expertise in Snowflake Data Warehousing, ETL/ELT development, and...Full time
Weekday
India9 days ago - ...set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Data Engineer ( AI/ML ) Overview- Mastercard is a global technology company powering one of the world’s fastest payment networks. Our Data...Full timeWorldwide
Mastercard
India15 days ago - ...Title and Summary Manager, Data Scientist About... ...scalability, performance, and engineering excellence. • Drive... ...driven insights for senior business and product stakeholders... ...Proficiency in Python, Spark, Hadoop platforms (Hive, Impala, Airflow, NiFi), and SQL. •...Full timeWork experience placementWorldwide
Mastercard
Indiaa month ago - The Role We’re seeking a highly skilled, execution-focused Senior Data Scientist with a minimum of 5 years of experience. This role demands... ..., leveraging advanced tools and frameworks across Python, SQL, Spark, and AWS. The role requires strong technical depth, problem-...SeniorFull time
Proximity
India28 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer - Spark / Kafka / NiFi. Be the first to apply!
