Senior Data Engineer - Spark / Kafka / NiFi
Full-time
Mastercard
Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary
Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive. Job Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi.
The ideal candidate will have a strong background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing.
The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments. Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.
Contribute to architecture discussions, technical designs, and engineering standards. Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling. Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operational tooling to improve platform reliability.
Monitor production environments and proactively address operational concerns.
Participate in troubleshooting, root-cause analysis, and incident resolution activities. Engineering Excellence
Participate in code reviews and promote engineering best practices.
Maintain high-quality documentation for systems, APIs, and platform components.
Collaborate closely with Product, Architecture, Platform, and DevOps teams.
Contribute to CI/CD processes and continuous improvement initiatives.
Mentor junior engineers and foster a culture of technical excellence and innovation. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related technical field.
6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
Strong expertise in Apache Spark (Batch and Structured Streaming).
Strong experience with Apache Kafka and event-driven architectures.
Experience developing, testing and supporting Apache NiFi data flows.
Proficiency in Scala, pyspark or Python.
Strong SQL and data modeling skills.
Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores.
Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
Strong analytical, problem-solving, and communication skills.
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments. Preferred Qualifications
Experience building observability solutions using monitoring and logging platforms.
Knowledge of data governance, metadata management, and data platform best practices.
Experience with containerization and orchestration technologies (Docker, Kubernetes).
Knowledge of performance engineering, resilience testing, and production readiness assessments.
Experience building engineering automation frameworks and reusable validation platforms.
Familiarity with observability tools, monitoring systems, and operational analytics.
Experience in highly regulated, transaction-processing, or large-scale enterprise environments. Corporate Security Responsibility
Senior Data Engineer - Spark / Kafka / NiFi
Job Posting Title: Senior Data Engineer - Spark / Kafka / NiF
Who is Mastercard?Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments, and businesses realize their greatest potential. Our culture is guided by the Mastercard Way—own it, simplify it, sense of urgency, thoughtful risk taking, unlock potential, and be inclusive. Job Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi.
The ideal candidate will have a strong background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing.
The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments. Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.
Contribute to architecture discussions, technical designs, and engineering standards. Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling. Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operational tooling to improve platform reliability.
Monitor production environments and proactively address operational concerns.
Participate in troubleshooting, root-cause analysis, and incident resolution activities. Engineering Excellence
Participate in code reviews and promote engineering best practices.
Maintain high-quality documentation for systems, APIs, and platform components.
Collaborate closely with Product, Architecture, Platform, and DevOps teams.
Contribute to CI/CD processes and continuous improvement initiatives.
Mentor junior engineers and foster a culture of technical excellence and innovation. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related technical field.
6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications.
Strong expertise in Apache Spark (Batch and Structured Streaming).
Strong experience with Apache Kafka and event-driven architectures.
Experience developing, testing and supporting Apache NiFi data flows.
Proficiency in Scala, pyspark or Python.
Strong SQL and data modeling skills.
Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores.
Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
Strong analytical, problem-solving, and communication skills.
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments. Preferred Qualifications
Experience building observability solutions using monitoring and logging platforms.
Knowledge of data governance, metadata management, and data platform best practices.
Experience with containerization and orchestration technologies (Docker, Kubernetes).
Knowledge of performance engineering, resilience testing, and production readiness assessments.
Experience building engineering automation frameworks and reusable validation platforms.
Familiarity with observability tools, monitoring systems, and operational analytics.
Experience in highly regulated, transaction-processing, or large-scale enterprise environments. Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard’s security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior Data Engineer - Spark / Kafka / NiFi in India vacancy
- ...We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance... ..., with hands-on expertise in Spark and Scala . You will work on designing... ...Familiarity with streaming technologies such as Kafka . Experience working on large-scale...Senior
Prox Works
Indiaa month ago - ...designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data... ...optimize distributed data applications using Spark and Scala . Develop reliable, high-performance...SeniorFull time
Proximity
Indiaa month ago - ...someone with ETL tool experience, since they don't use any ETL tools. Their platform is homegrown, so they need someone with strong data platform development skills, particularly in SQL and Python, who can easily navigate a large codebase and take ownership of managing...Senior
NTT DATA, Inc.
India23 days ago - ...seeking a highly skilled, execution-focused Senior Data Scientist with a minimum of 5 years of... ...and frameworks across Python, SQL, Spark, and AWS. The role requires strong technical... ...and orchestration. Collaborate with engineering teams to integrate models into cloud-based...SeniorFull time
Proximity
Indiaa month ago - ...Required Qualifications Strong hands-on Data Engineering experience. Expert-level experience with Databricks . Strong hands-on... ...environments. Ideal Candidate The ideal candidate is a senior-level , hands-on Databricks Data Engineer . They should be...SeniorFull time
Enable-data
India11 days ago - ...As a Lead Data Engineer , you will build and lead the data infrastructure powering... ...and intelligently. As the first senior data hire, you will define the data... ...technologies is highly desirable: Apache Spark , Apache Airflow , Kafka , and Elasticsearch or...Full time
Smartworking
India22 days ago - ...We are seeking an experienced Snowflake Data Engineer with 5–8 years of hands-on experience in data engineering, data warehousing, and large-scale data migration. The ideal candidate will have strong expertise in Snowflake Data Warehousing, ETL/ELT development, and...Full time
Weekday
India16 days ago - ...global brands do business? Join us! We are looking for a Senior Data Scientist to join our Agentic Solutions team, focused on driving... ...work closely with cross-functional teams, including product, engineering, solutions, and customer-facing stakeholders, to prototype,...SeniorFull time
Netomi
Indiaa month ago - The Role We’re seeking a highly skilled, execution-focused Senior Data Scientist with a minimum of 5 years of experience. This role demands... ..., leveraging advanced tools and frameworks across Python, SQL, Spark, and AWS. The role requires strong technical depth, problem-...SeniorFull time
Proximity
Indiaa month ago - ...governments realize their greatest potential. Title and Summary Senior Data Scientist Who is Mastercard? Mastercard is a global... ...AI, and apply them to real-world problems. • Promote engineering best practices and contribute to a culture of innovation and...SeniorFull timeWorldwide
Mastercard
India1 day ago - ...empathy and values inclusivity. As a Senior Data Scientist , you will build and own... ...outcomes. You will work closely with product, engineering, and operations teams, hands-on building... ...with real-time/streaming data ( Kafka ) Exposure to LLM/GenAI applications...SeniorFull time
Fourkites
India15 days ago - ...their greatest potential. Title and Summary Analyst, Big Data Analytics & Engineering Overview: About Mastercard Mastercard is a global... ...value through compelling presentations and dashboards to senior leadership and internal teams, ensuring tool adoption and...Full timeWorldwide
Mastercard
India1 day ago - ...centers outside the U.S. As a hub for technological convergence, our engineering talent is a catalyst for innovation in multimedia network and... ...Minimum 14 years of experience with 8+ years of experience in data science, machine learning, or AI, and at least 2+ years in a...Local area
Dish
India12 days ago - ...seeking a highly experienced Senior Platform Architect – Microsoft... ...evolution of a modern enterprise data platform built on Microsoft... ...who enjoy solving complex data engineering challenges, building scalable... ...technologies such as PySpark, Spark SQL, Python, T-SQL, or KQL for...SeniorFull time
Weekday
Indiaa month ago - We are looking for a skilled Data Engineer with strong experience in AWS, SQL, Snowflake, BI Connector to snowflake migration . The candidate will be responsible for building data pipelines, migrating BI workloads to Snowflake, optimizing data solutions, and ensuring seamless...Full time
3pillarglobal
Indiaa month ago - We are looking for an experienced Data Engineer to design, develop, and optimize scalable data platforms and modern cloud-based data solutions. This role is ideal for professionals with strong expertise in Microsoft Azure Fabric and Databricks , along with a passion for...Full time
Weekday
Indiaa month ago - ...Combinator , and Index Ventures , we help enterprises drive efficiency, lower costs, and deliver higher quality customer experiences. Want to be part of the AI revolution and transform how the world’s largest global brands do business? Join us! We are looking for a Senior D...SeniorFull time
Netomi
Indiaa month ago - ...founders and team members have industry experience and have held senior positions in a wide variety of companies - from emerging... ...transparency, and dependability. About the Role We are looking for a Senior BI Engineer to join our team and lead the analytics and repor...SeniorTemporary work
Tech Holding
Indiaa month ago - ...Req ID: 389196 NTT DATA strives to hire exceptional, innovative... ...seeking a Business Intelligence Senior Specialist to join our team in... ...(IN-UP), India (IN). Data Engineer Skills: · Tools: Strong... ...Airflow composer, Big Data,Hadoop, Spark, SQL, Python, Java, Shell...SeniorWork at officeRemote workFlexible hours
NTT DATA, Inc.
India1 day ago - The Role: We are seeking a Senior Principal Engineer to drive the architectural strategy and technical execution across our 80+ engineer Application Portfolio, encompassing all Web, Mobile, and Headless surfaces. This is a high-impact, individual contributor role where...SeniorFull time
Alphasense
Indiaa month ago - ...We are looking for a highly skilled Data Scientist with expertise in Vision Language Models (VLMs), Computer Vision, and Multimodal... ...and video. You will collaborate closely with machine learning engineers, data engineers, and product teams to build intelligent AI...Full time
Weekday
Indiaa month ago - We are seeking an experienced Senior Consultant - Databricks to design... ...develop, and optimize modern data platforms on Databricks . The... ...have strong expertise in data engineering, cloud technologies, ETL... ...ELT workflows using PySpark and Spark SQL . Develop and manage Delta...SeniorFull time
Unison Group
Indiaa month ago - We are seeking an experienced and innovative Senior AI Engineer. In this remote role, you will be responsible for designing and developing autonomous AI agents that automate virtual pre-construction, 3D digital twin generation, and estimation workflows. You'll play a key...SeniorFull timeRemote work
Pear Tree
Indiaa month ago - ...Greenhouse is looking for a Senior Manager, Integrations Consulting to join our team! As a part of the Professional Services organization... ...manage deep technical escalations without needing to pull in engineering for every issue A collaborator – you build alignment...SeniorFull timeImmediate start
Greenhouse
Indiaa month ago - ...and ambitious individuals to join our company. The Role The Senior Director of Customer Success will lead a global Customer... ...customer journeys through automation, segmentation, AI , and data-driven decision making. Partner closely with Sales, Marketing...SeniorFull timeRemote work
JoVE
Indiaa month ago - ...governments realize their greatest potential. Title and Summary Senior Analyst, Business Development • This role supports the ‘... ...transaction flows, ability to build financial models, competence in data analysis to gather useful insights. • A candidate high on energy...SeniorFull timeWorldwide
Mastercard
India1 day ago - ...About the Role The Senior Test Lead will be responsible for independently leading the end-to-end testing and quality assurance activities... ...’s degree in Computer Science, Information Technology, Engineering, Banking, Finance, or a related discipline. ~8+ years of experience...SeniorTemporary workImmediate start
Unison Group
India15 days ago - ...About the Role As a Senior Business Development Representative , you will play a pivotal... ...generation and maintaining excellent data quality. You will collaborate on account... ...personalised outreach to CTOs , Heads of Engineering and senior stakeholder buyers. Build...SeniorFull timeRemote work
Smartworking
Indiaa month ago - ...responsible for submitting any material information around business Opportunities and Industry/Customer Risk, same shall be reported to the senior functional management at regular interval. Requirements ~ Overall 3- 7 years of experience in sales. ~2-3 years of...SeniorFull timeLocal area
DBS Bank Ltd
India1 day ago - ...requirements. Develop strategies for prompt engineering, model routing, fine-tuning, and... ...Functional Collaboration Partner with Product, Data Science, Platform, Security, and... ...Communicate complex technical concepts clearly to senior leadership and business stakeholders....SeniorFull time
Weekday
India14 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Data Engineer - Spark / Kafka / NiFi. Be the first to apply!
Related searches