Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)
Full-time
Mastercard
Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary
What You'll Do
• Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
• Design and build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
• Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
• Build and operate platforms that support analytics, machine learning, generative AI, and agentic AI workloads.
• Create reusable data products, services, APIs, and self-service capabilities that accelerate experimentation and production delivery.
• Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines.
• Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
• Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
• Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance.
• Apply security and privacy-by-design principles for sensitive and regulated data environments.
• Evaluate emerging data and AI technologies through prototypes, proof-of-concepts, and technical assessments.
• Translate loosely defined research, innovation, or product requirements into practical architectures and incremental delivery plans.
• Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, and engineering mentorship.
• Partner with cross-functional teams to mature successful R&D initiatives into enterprise-grade production solutions.
• Communicate technical decisions, trade-offs, risks, dependencies, and roadmap recommendations to both technical and business stakeholders.
• Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines.
All About You
Required Qualifications
• Typically 12-18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions.
• Extensive experience designing, building, and operating large-scale production data platforms.
• Expert programming skills in Python, PySpark, and modern software engineering practices.
• Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large-scale analytical processing.
• Strong hands-on experience with distributed data processing technologies such as Apache Spark and modern lakehouse platforms.
• Experience building and operating cloud-native data platforms in AWS, or Azure, or other enterprise cloud environments.
• Strong knowledge of modern data architecture patterns, including data lakes, lakehouses, data meshes, data products, and event-driven architectures.
• Experience with cloud storage, data integration, streaming, serverless, observability, and security services across public cloud platforms.
• Experience building scalable batch and real-time data pipelines.
• Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar technologies.
• Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
• Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
• Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges.
• Ability to make sound architectural decisions while balancing delivery speed, innovation, maintainability, security, and cost.
• Proven ability to thrive in R&D and innovation-focused environments where priorities and requirements may evolve through experimentation.
• Strong communication, collaboration, technical leadership, and mentoring skills.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience. Preferred Qualifications
• Experience supporting machine learning, generative AI, or agentic AI systems.
• Experience with MLflow or comparable ML lifecycle tooling.
• Experience with Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
• Experience with Unity Catalog, Microsoft Purview, or comparable catalog and governance platforms.
• Experience with Kafka or other event-streaming technologies.
• Experience with containerized and cloud-native platforms, including Docker and Kubernetes.
• Experience with Terraform or another Infrastructure as Code framework.
• Experience integrating structured, semi-structured, unstructured, and streaming data.
• Knowledge of responsible AI, model evaluation, and AI platform observability.
• Experience in payments, financial services, or another regulated industry.
Success in This Role
Success requires someone who:
• Remains strongly hands-on while providing technical direction.
• Can build production-quality systems without introducing unnecessary platform complexity.
• Converts experimentation into reusable engineering capabilities.
• Designs for security, governance, reliability, and observability from the outset.
• Challenges assumptions and validates architectural choices through evidence and prototypes.
• Enables AI and product teams rather than taking ownership of data science or model research.
• Influences across teams without depending on formal people-management authority. Corporate Security Responsibility
Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)
Overview
Mastercard Foundry R&D develops emerging technology solutions and transforms promising concepts into scalable products and platforms.
We are seeking a Principal Data Engineer to design and build the data foundations that power advanced analytics, machine learning, generative AI, and agentic AI solutions. This role requires a hands-on technical leader who can turn ambiguous R&D objectives into secure, scalable, and production-ready data systems.
You will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to create reusable data capabilities that accelerate experimentation and the delivery of innovative products.
What You'll Do
• Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
• Design and build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
• Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
• Build and operate platforms that support analytics, machine learning, generative AI, and agentic AI workloads.
• Create reusable data products, services, APIs, and self-service capabilities that accelerate experimentation and production delivery.
• Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines.
• Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
• Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
• Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance.
• Apply security and privacy-by-design principles for sensitive and regulated data environments.
• Evaluate emerging data and AI technologies through prototypes, proof-of-concepts, and technical assessments.
• Translate loosely defined research, innovation, or product requirements into practical architectures and incremental delivery plans.
• Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, and engineering mentorship.
• Partner with cross-functional teams to mature successful R&D initiatives into enterprise-grade production solutions.
• Communicate technical decisions, trade-offs, risks, dependencies, and roadmap recommendations to both technical and business stakeholders.
• Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines.
All About You
Required Qualifications
• Typically 12-18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions.
• Extensive experience designing, building, and operating large-scale production data platforms.
• Expert programming skills in Python, PySpark, and modern software engineering practices.
• Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large-scale analytical processing.
• Strong hands-on experience with distributed data processing technologies such as Apache Spark and modern lakehouse platforms.
• Experience building and operating cloud-native data platforms in AWS, or Azure, or other enterprise cloud environments.
• Strong knowledge of modern data architecture patterns, including data lakes, lakehouses, data meshes, data products, and event-driven architectures.
• Experience with cloud storage, data integration, streaming, serverless, observability, and security services across public cloud platforms.
• Experience building scalable batch and real-time data pipelines.
• Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar technologies.
• Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
• Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
• Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges.
• Ability to make sound architectural decisions while balancing delivery speed, innovation, maintainability, security, and cost.
• Proven ability to thrive in R&D and innovation-focused environments where priorities and requirements may evolve through experimentation.
• Strong communication, collaboration, technical leadership, and mentoring skills.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience. Preferred Qualifications
• Experience supporting machine learning, generative AI, or agentic AI systems.
• Experience with MLflow or comparable ML lifecycle tooling.
• Experience with Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
• Experience with Unity Catalog, Microsoft Purview, or comparable catalog and governance platforms.
• Experience with Kafka or other event-streaming technologies.
• Experience with containerized and cloud-native platforms, including Docker and Kubernetes.
• Experience with Terraform or another Infrastructure as Code framework.
• Experience integrating structured, semi-structured, unstructured, and streaming data.
• Knowledge of responsible AI, model evaluation, and AI platform observability.
• Experience in payments, financial services, or another regulated industry.
Success in This Role
Success requires someone who:
• Remains strongly hands-on while providing technical direction.
• Can build production-quality systems without introducing unnecessary platform complexity.
• Converts experimentation into reusable engineering capabilities.
• Designs for security, governance, reliability, and observability from the outset.
• Challenges assumptions and validates architectural choices through evidence and prototypes.
• Enables AI and product teams rather than taking ownership of data science or model research.
• Influences across teams without depending on formal people-management authority. Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard’s security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
Vacancy posted 18 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow) in India vacancy
- ...people, businesses and governments realize their greatest potential. Title and Summary Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow) Overview Mastercard Foundry R&D develops emerging technology solutions and transforms...DatabricksAmazon Web ServiceFull timeWorldwide
Mastercard
India10 days ago - ...Summary Lead Data Engineer Position Overview... ...is part of the AI&DPE Data... ...and cloud based architectures, with a strong... ...compute workloads in AWS environments Leverage Databricks for large scale... ...diagrams, data flows, and design... ...engineering or ML workloads Experience...DatabricksAmazon Web ServiceFull timeImmediate startWorldwide
Mastercard
India12 days ago - ...Title and Summary Senior Data Engineer, Data Engineering Job Description... ...all. Our Team: The AI Network Services Team (AI &... ...(CDE), Cloudera AI (CAI), Databricks, AWS to deliver the product engineering... ...tools – ETL/ELT tools (i.e. Apache NiFi, Azure Data Factory,...DatabricksAmazon Web ServiceFull timeWorldwide
Mastercard
India12 days ago - ...Summary Analyst, Big Data Analytics & Engineering About Mastercard... ...downstream analytics and AI use cases. •... ...publish analysis-ready and ML-ready datasets, including... ...tools such as Apache Airflow, Apache NiFi, Azure Data... ...cloud services (Azure/AWS/GCP) for data lakes,...Amazon Web ServiceFull timeWorldwideFlexible hours
Mastercard
India12 days ago - ...Title and Summary Senior AI Data Engineer Overview As a Senior... ...Build and optimize data architectures supporting LLMs, RAG, embeddings... ...native data platforms using AWS, Azure, or Google Cloud... ...orchestration tools such as Apache Airflow, Azure Data Factory, or similar...Amazon Web ServiceFull timeWorldwide
- ...potential. Title and Summary Senior Data Engineer Data Engineer / Machine Learning Engineer... ...pipelines, curated data models, and ML-ready feature datasets while... ...using data technology technology like AWS, S3, Databricks. • Solid understanding of data modeling...DatabricksAmazon Web ServiceFull timeWorldwide
Mastercard
India12 days ago - ...Title and Summary Principal Data Engineer (Data Platforms) Who... ...cloud and modern data-architecture expertise. This role... ...architectures using Databricks, Snowflake, Spark, and... ...using Unity Catalog, AWS Glue Data Catalog, and... ...catalog patterns such as Apache Polaris. Design...DatabricksAmazon Web ServiceFull timeContract workLocal areaWorldwide
- ...and Summary Lead Data Engineer (Data Platforms)... ...capabilities using Databricks, Snowflake, AWS, and Azure, while ensuring... ...Architecture & Development Design... ...including RBAC, service principals, and enterprise identity... ...formats (Delta Lake, Apache Iceberg) Experience...DatabricksAmazon Web ServiceFull timeWorldwide
Mastercard
India9 days ago - .... Title and Summary Data Scientist II Role Overview... ...enterprise-grade AI and Generative AI... ...combines strong software engineering with model fine-tuning, Databricks-based ML engineering, AWS deployment, and end-to... ...including secure cloud architecture, container deployment,...DatabricksAmazon Web ServiceFull timeWorldwide
Mastercard
India19 days ago - ...markets, and as a principal card issuer on... ...industry — and data sits at the... ...and skilled Data Engineer to join our India... ...systems to AWS cloud . You’ll... ...tools such as Airflow, Redshift, dbt... ...evolve the data architecture with new ideas... ...Interest in exploring AI/ML use cases...Amazon Web ServiceFull timeWork at officeLocal areaWorldwideFlexible hours3 days per week
- ...Title and Summary Senior Data Engineer Overview... ...data pipelines using Apache Spark, Java/Scala across... ...enterprise standards. Architecture & Platform Engineering... ...solutions using: Apache Airflow/ Apache NiFi and Similar... ...platforms, preferably AWS: Amazon S3, EMR, AWS...Amazon Web ServiceFull timeWorldwide
Mastercard
India3 days ago - ...Title and Summary Data Scientist I Who... ...at implementing an AI based solution to support... ...cloud platforms (AWS, Azure) and... ...data platforms like Databricks. • Integrate AI models... ...advancements in AI/ML research,... ...problems. • Promote engineering best practices and...DatabricksAmazon Web ServiceFull timeWorldwide
- ...impact talent who see AI as a teammate – leveraging... ...an exceptional Staff Data Engineer to join our team as a... ...design complex data architectures, mentor engineering teams... ...Expertise with Apache Kafka, real-time data... ...Advanced experience with AWS cloud infrastructure and...Amazon Web ServiceFull timeFlexible hours
MoneyLion
India4 days ago - ...Summary Vice President, Data Engineering Our Purpose... ...core components such as Apache NiFi, Apache Spark, and... ...pipeline ecosystem that uses AI agents, orchestration frameworks... ...(e.g., Apache Airflow, Nifi, dbt, Spark, Kafka... ...ecosystems (Oracle, AWS Glue, Azure Data Factory...Amazon Web ServiceFull timeWorldwide3 days per week
- ...Title and Summary AI Engineer Mastercard’s... ...providing unparalleled data-driven insights... ...This role will lead ML engineering team... ...transformer-based architectures (BERT, CLIP, LLaVA... ...knowledge graphs (Neo4j, AWS Neptune) to enable... ...pipelines on Databricks (AWS) integrating...DatabricksAmazon Web ServiceFull timeWork experience placementWorldwide
- ...technological convergence, our engineering talent is a catalyst... ...scaling of end-to-end AI/ML and GenAI initiatives,... ...OpenAI, Anthropic, AWS Bedrock, etc.) into key... ...years of experience in data science, machine learning... ...stores, RAG, agentic architecture Deep understanding of...Amazon Web ServiceLocal area
- ...Summary Manager, Agentic AI Product Management... ...deliverables for engineering and AI/ML teams. Responsibilities... ...requirements, data flows, exception-handling logic... ...embeddings, and RAG-based architectures Understanding of... ...-native platforms (AWS, Azure) and modern...Amazon Web ServiceFull timeWorldwide
Mastercard
India12 days ago - ...and Summary Senior Data Scientist Job... ...framework design, data engineering, dashboard... ...into data-driven and AI-enabled solutions for... ...define AI solution architecture, integration requirements... ...of AI/ML and Generative AI concepts... ...architecture, data flows, API integrations,...Full timeWorldwide
Mastercard
India10 days ago - ...Title and Summary Data Scientist II-1 We... ...of transactions are flowing to Mastercard in... ...apply and leverage AI at scale. The AI team... ...services (recommendation engine, anti-money... ...challenges • Develop AI/ML applications... ...RPD and neural net architecture evaluation), PyTorch...Full timeWorldwide
Mastercard
India12 days ago - ...Title and Summary Senior Data Engineer Job Posting Title: Senior... ...time data platforms built on Apache Spark, Apache Kafka, and Apache... ...systems, streaming architectures, event-driven platforms, and... ...supporting Apache NiFi data flows. Proficiency in Scala, pyspark...Full timeWorldwide
Mastercard
India13 days ago - ...their greatest potential. Title and Summary Senior Data Engineer Who is Mastercard? Mastercard is a global... ...organization • Experience with Cloud tech stack, Big Data, Databricks • Extensive experience with Spark, Airflow • Strong analytical skills required for...DatabricksFull timeWorldwide
- ...governments realize their greatest potential. Title and Summary Data Engineer II Data Engineer, DPO Who is Mastercard? Mastercard... ...implement, and maintain data pipelines. • Experience with Databricks and Spark is a plus. • Demonstrate curiosity about emerging...DatabricksFull timeLocal areaWorldwideFlexible hours
Mastercard
India12 days ago - ...Manager, Software Engineering Overview We... ...development of scalable data platforms, backend... ...leadership across architecture, delivery, and... ..., and leverages AI-powered engineering... ...such as Snowflake, Databricks, Hadoop, or... ...platforms such as AWS, Azure, or GCP....DatabricksAmazon Web ServiceFull timeTemporary workFor contractorsWorldwide
- ...potential. Title and Summary Analyst, Big Data Analytics & Engineering Overview: About Mastercard... ...maintaining advanced business analytics and AI-driven solutions for pre-sales value... ...certifications in analytics, data science, or AI/ML are preferred Why Us? At...Full timeWorldwide
- ...greatest potential. Title and Summary Senior Analyst, Big Data Analytics & Engineering About Mastercard: Mastercard is a global technology... ...degrees or certifications in analytics, data science, AI/ML, or an MBA are preferred. Why Us? At Mastercard, you...Full timeWorldwide
- ...greatest potential. Title and Summary Senior Data Scientist Overview Mastercard company is seeking a Senior Software Engineer, which serves various client-tele such as... ...Mandatory skill set: Data Science including AI ML skillset Are you motivated by speeding business...Full timeWork experience placementWorldwide
Mastercard
India12 days ago - ...governments realize their greatest potential. Title and Summary Senior Data Scientist-1 Overview Product Data & Analytics is a... ...to provide stakeholders with actionable insights. • Integrate AI/ML models to enhance the accuracy and efficiency of analytical tools...Full timeWorldwideFlexible hours
Mastercard
India12 days ago - ...governments realize their greatest potential. Title and Summary Data Engineer II We believe in power of data. We think creatively, design... ...ontributor capacity. • Develop new queries and Alteryx flows for extracting, transforming and analysing data to address project...Full timeTemporary workWork experience placementWorldwide
Mastercard
India12 days ago - ...potential. Title and Summary Data Scientist II Job... ...Experience in building automation flow using Alteryx/ Tableau prep/ Python... ...batch and streaming tools (Spark, AWS tools, Hadoop) • Experience... ..., Information Technology, Engineering, Mathematics, Statistics, M.S....Amazon Web ServiceFull timeWorldwide
Mastercard
India12 days ago - ...Product Management – Data and AI Platform WHO IS... ...a data mesh architecture, advanced technologies... ...to partnering with engineering for execution. You... ...on experience with AWS, Azure, or Google Cloud... ...familiarity with AI/ML platforms such as Databricks and Snowflake (...DatabricksAmazon Web ServiceFull timeWorldwide
Mastercard
India2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow). Be the first to apply!
