Senior Data Engineer - Spark / Kafka / NiFi
Full-time
Mastercard
Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive. Job Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi.
The ideal candidate will have a strong background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing.
The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments. Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.
Contribute to architecture discussions, technical designs, and engineering standards. Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling. Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operational tooling to improve platform reliability.
Monitor production environments and proactively address operational concerns.
Participate in troubleshooting, root-cause analysis, and incident resolution activities. Engineering Excellence
Participate in code reviews and promote engineering best practices.
Maintain high-quality documentation for systems, APIs, and platform components.
Collaborate closely with Product, Architecture, Platform, and DevOps teams.
Contribute to CI/CD processes and continuous improvement initiatives.
Mentor junior engineers and foster a culture of technical excellence and innovation. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related technical field.
6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
Strong expertise in Apache Spark (Batch and Structured Streaming).
Strong experience with Apache Kafka and event-driven architectures.
Experience developing, testing and supporting Apache NiFi data flows.
Proficiency in Scala, pyspark or Python.
Strong SQL and data modeling skills.
Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores.
Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
Strong analytical, problem-solving, and communication skills.
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments. Preferred Qualifications
Experience building observability solutions using monitoring and logging platforms.
Knowledge of data governance, metadata management, and data platform best practices.
Experience with containerization and orchestration technologies (Docker, Kubernetes).
Knowledge of performance engineering, resilience testing, and production readiness assessments.
Experience building engineering automation frameworks and reusable validation platforms.
Familiarity with observability tools, monitoring systems, and operational analytics.
Experience in highly regulated, transaction-processing, or large-scale enterprise environments. Corporate Security Responsibility
Senior Data Engineer - Spark / Kafka / NiFi
Job Posting Title: Senior Data Engineer - Spark / Kafka / NiF
Who is Mastercard?Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive. Job Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi.
The ideal candidate will have a strong background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing.
The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments. Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.
Contribute to architecture discussions, technical designs, and engineering standards. Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling. Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operational tooling to improve platform reliability.
Monitor production environments and proactively address operational concerns.
Participate in troubleshooting, root-cause analysis, and incident resolution activities. Engineering Excellence
Participate in code reviews and promote engineering best practices.
Maintain high-quality documentation for systems, APIs, and platform components.
Collaborate closely with Product, Architecture, Platform, and DevOps teams.
Contribute to CI/CD processes and continuous improvement initiatives.
Mentor junior engineers and foster a culture of technical excellence and innovation. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related technical field.
6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
Strong expertise in Apache Spark (Batch and Structured Streaming).
Strong experience with Apache Kafka and event-driven architectures.
Experience developing, testing and supporting Apache NiFi data flows.
Proficiency in Scala, pyspark or Python.
Strong SQL and data modeling skills.
Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores.
Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
Strong analytical, problem-solving, and communication skills.
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments. Preferred Qualifications
Experience building observability solutions using monitoring and logging platforms.
Knowledge of data governance, metadata management, and data platform best practices.
Experience with containerization and orchestration technologies (Docker, Kubernetes).
Knowledge of performance engineering, resilience testing, and production readiness assessments.
Experience building engineering automation frameworks and reusable validation platforms.
Familiarity with observability tools, monitoring systems, and operational analytics.
Experience in highly regulated, transaction-processing, or large-scale enterprise environments. Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard’s security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
Vacancy posted 8 days ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer - Spark / Kafka / NiFi in India vacancy
- ...greatest potential. Title and Summary Senior AI Data Engineer Overview As a Senior AI Data... ...Strong proficiency in Python, SQL, Spark, and distributed data processing frameworks... ...Experience with streaming technologies such as Kafka, Kinesis, or Azure Event Hubs....SeniorFull timeWorldwide
- ...potential. Title and Summary Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data... ...processing technologies such as Apache Spark and modern lakehouse platforms. • Experience... ...platforms. • Experience with Kafka or other event-streaming technologies....SuggestedFull timeWorldwide
Mastercard
India6 days ago - ...greatest potential. Title and Summary Lead Data Engineer Position Overview Have you ever... ..., distributed data platforms (Cloudera/Spark), and cloud based architectures, with a... ...streaming and big‑data technologies: Kafka Hadoop ecosystem Hive/Impala Exposure...SuggestedFull timeImmediate startWorldwide
Mastercard
India8 days ago - ...of Gen. About the Role We are seeking an experienced Senior Principal Data Engineer to serve as a technical leader responsible for architecting... ...Streaming & Real-time Processing: Experience with Apache Kafka, event-driven architectures, and real-time analytics pipelines...SeniorRemote jobFull timeFlexible hours
- ...governments realize their greatest potential. Title and Summary Senior Data Engineer Data Engineer / Machine Learning Engineer Level 7 |... ...building data pipelines using distributed processing (e.g., Spark,PySpark,HADOOP). • Experience in using data technology...SeniorFull timeWorldwide
- ...governments realize their greatest potential. Title and Summary Senior Data Engineer, Data Engineering Job Description Summary Who is... ...have) with data integration tools – ETL/ELT tools (i.e. Apache NiFi, Azure Data Factory, Pentaho, Talend) • Experience with Graph...SeniorFull timeWorldwide
- ...and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Analyst, Big Data Analytics & Engineering About Mastercard: Mastercard is a global technology leader in the payments industry, committed to...SeniorFull timeWorldwide
Mastercard
India8 days ago - ...of Gen. About the Role We are seeking an experienced Senior Manager, Data Engineering to serve as a technical people leader responsible for owning... ...and cost perspectives. Expertise with Apache Kafka, real-time data processing, and event-driven architectures...SeniorRemote jobFull timeFlexible hours
- ...greatest potential. Title and Summary Data Engineer II Data Engineer, DPO Who is Mastercard... ...and global teams. Collaborate with senior and peer engineers on multiple projects... ...pipelines. • Experience with Databricks and Spark is a plus. • Demonstrate curiosity...Full timeLocal areaWorldwideFlexible hours
Mastercard
India8 days ago - ...greatest potential. Title and Summary Lead Data Engineer Who is Mastercard? Mastercard is a... ...and scalable data pipelines using spark, Scala/ python on Hadoop or object storage... ...storage • Experience of building Nifi pipelines • Experience of working with...Full timeWorldwide
- ...potential. Title and Summary Analyst, Big Data Analytics & Engineering About Mastercard Mastercard is a... ...data processing jobs using PySpark/Spark, Python, and SQL (including engines such... ...tools such as Apache Airflow, Apache NiFi, Azure Data Factory, Pentaho, or...Full timeWorldwideFlexible hours
- ...revolutionising the global payments industry — and data sits at the core of everything we do. We’... ...for a passionate and skilled Data Engineer to join our India-based Data Engineering... ...such as Airflow, Redshift, dbt , and Kafka , collaborating closely with product,...Full timeWork at officeLocal areaWorldwideFlexible hours3 days per week
- ...governments realize their greatest potential. Title and Summary Senior Data Scientist Job Title: Senior Data Scientist, Executive... ...requirement gathering, conceptualization, KPI framework design, data engineering, dashboard development, stakeholder alignment, finance...SeniorFull timeWorldwide
Mastercard
India6 days ago - ...set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Data Scientist-2 Overview Product Data & Analytics is a centralized global team that enables Mastercard product business units to...SeniorFull timeWorldwideFlexible hours
Mastercard
India15 days ago - ...governments realize their greatest potential. Title and Summary Senior Data Scientist Who is Mastercard? Mastercard is a global... ...AI, and apply them to real-world problems. • Promote engineering best practices and contribute to a culture of innovation and...SeniorFull timeWorldwide
Mastercard
India8 days ago - ...realize their greatest potential. Title and Summary Principal Data Engineer (Data Platforms) Who is Mastercard? Mastercard is a... ...processing and query architectures using Databricks, Snowflake, Spark, and Trino, with open table and file formats such as Iceberg,...Full timeContract workLocal areaWorldwide
- ...greatest potential. Title and Summary Data Scientist II Job Description – Data &... ...using big data batch and streaming tools (Spark, AWS tools, Hadoop) • Experience with function... ...Science, Information Technology, Engineering, Mathematics, Statistics, M.S./M.B.A. preferred...Full timeWorldwide
Mastercard
India8 days ago - ...their greatest potential. Title and Summary Data Scientist II-2 Job Description Summary... ..., Prepaid) and services (recommendation engine, anti-money laundering, fraud risk management... ...•Big Data platforms such as Hadoop, Hive, Spark, GPU Clusters for deep learning •...Full timeWorldwide
Mastercard
India8 days ago - ...greatest potential. Title and Summary Data Scientist II Role Overview Build and... ...Mastercard. This role combines strong software engineering with model fine-tuning, Databricks-based... ...Hands-on experience with Databricks, Spark/PySpark, MLflow, model registries, and...Worldwide
Mastercard
India15 days ago - ...of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Lead Data Engineer (Data Platforms) Role Summary We are looking for a Lead Software Engineer to design, build, and scale cloud-based data and...Full timeWorldwide
Mastercard
India5 days ago - Branch Relationship Manager, Growth, Distribution (RE0038) Location: Koramangala, Bangalore Job: Relationship Management Schedule: Regular Employee Status: Full timeSeniorFull time
DBS Bank Ltd
India8 days ago - ...governments realize their greatest potential. Title and Summary Senior Data Scientist Job Description Summary - Who is Mastercard?... .... Leverage best practices in machine learning and data engineering to develop scalable solutions. Identify areas where...SeniorFull timeWorldwide
- ...set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Data Scientist, Business Insights Job Title: Senior Data Scientist Overview: Our Purpose: Mastercard powers economies and...SeniorFull timeWorldwide
- ...people, businesses and governments realize their greatest potential. Title and Summary Senior Data Scientist Overview Mastercard company is seeking a Senior Software Engineer, which serves various client-tele such as Merchants, Financial Institutions (Issuer,...SeniorFull timeWork experience placementWorldwide
- ...Req ID: 389196 NTT DATA strives to hire exceptional, innovative... ...seeking a Business Intelligence Senior Specialist to join our team in... ...(IN-UP), India (IN). Data Engineer Skills: · Tools: Strong... ...Airflow composer, Big Data,Hadoop, Spark, SQL, Python, Java, Shell...SeniorWork at officeRemote workFlexible hours
NTT DATA, Inc.
India28 days ago - ...their greatest potential. Title and Summary Analyst, Big Data Analytics & Engineering Overview: About Mastercard Mastercard is a global... ...value through compelling presentations and dashboards to senior leadership and internal teams, ensuring tool adoption and...Full timeWorldwide
- ...governments realize their greatest potential. Title and Summary Data Scientist II Job Description –Product Data & Analytics –... ...Master’s Degree in a Computer Science, Information Technology, Engineering, Mathematics, Statistics, M.S./M.B.A. preferred Additional Competencies...Full timeWorldwide
Mastercard
India8 days ago - ...governments realize their greatest potential. Title and Summary Data Scientist (Data Visualisation) Overview • The Mastercard... ...Economics Institute, reporting to the team lead for Data Systems Engineering. An individual who will: • create clear, compelling data...Full timeWorldwide
Mastercard
India8 days ago - ...a team to achieve sales targets and customer satisfaction Strong analytical and problem-solving skills, with the ability to make data-driven decisions Excellent communication and interpersonal skills, with the ability to build relationships with customers and stakeholders...SeniorFull timeWork at office
DBS Bank Ltd
India8 days ago - ...and implement models in production. ● Leverage programming and data science tools to design and backtest systematic trading... ...years of experience in quantitative research, trading, financial engineering, or a related role. ● Strong foundation in statistics, probability...Full timeRemote work
Delta Exchange
India6 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer - Spark / Kafka / NiFi. Be the first to apply!
Related searches