Manager, AI Engineering (Tester )
$140k - $231kMastercard
Manager, AI Engineering (Tester )
Mastercard's Business & Market Insights (B&MI) group delivers unparalleled data-driven intelligence and frontier AI solutions that help organizations make smarter, faster, and more impactful decisions. We are currently looking for a AI Tester for the Operational Intelligence Program within B&MI. This is a highly specialized, hands-on AI testing leadership position dedicated to ensuring our Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. This role will lead AI quality engineering efforts — defining evaluation frameworks, red-teaming strategies, and LLMOps quality gates — while fostering a culture of rigorous, first-class AI testing across the program.
Roles and Responsibilities:• Design and own end-to-end LLM evaluation frameworks — including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection across model versions and prompt variations.
• Build comprehensive test suites for agentic AI systems — validating tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling across multi-step reasoning workflows.
• Develop RAG pipeline evaluation frameworks assessing retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using tools like RAGAS, TruLens, and DeepEval.
• Lead structured red-teaming and adversarial testing exercises targeting prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation — building and maintaining an evolving adversarial test library.
• Execute fairness, bias, and Responsible AI audits — testing for demographic bias, sentiment skew, representation gaps, and validating explainability mechanisms, citations, and confidence score accuracy.
• Design and run inference performance benchmarks — measuring latency, throughput, token efficiency, and degradation under peak load — and enforce LLM quality gates within CI/CD pipelines on Databricks (AWS).
• Build production monitoring and drift detection pipelines tracking semantic output drift, embedding shifts, retrieval degradation, and anomalous agent behaviors using observability tooling (Grafana, Datadog, CloudWatch).
• Define the AI testing roadmap and quality standards for the program — establishing evaluation metrics, tooling choices, and documentation practices across all Gen AI workstreams.
• Partner with Gen AI engineers, ML engineers, and product stakeholders to embed quality from day one — reviewing prompt architectures, agent designs, and system workflows for testability and risk.
• Continuously research and adopt frontier evaluation benchmarks (RAGAS, MMLU, TruthfulQA, MT-Bench) and emerging AI testing methodologies to keep quality practices at the cutting edge. All About You:
• Master's/Bachelor's degree in Computer Science, AI/ML, or Software Engineering, with considerable hands-on experience leading AI/ML quality engineering or LLM testing programs in production environments.
• Demonstrated expertise testing LLM and Gen AI systems — including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation in real production settings.
• Deep hands-on knowledge of AI evaluation frameworks and tooling: RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, Weights & Biases Evals, or equivalent platforms.
• Strong understanding of Gen AI failure modes — hallucination, prompt injection, retrieval grounding failures, context drift, agent loop failures — and proven methods to surface and document them systematically.
• Strong Python programming skills with the ability to independently build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.
• Working knowledge of LLM ecosystems — OpenAI, Anthropic, Hugging Face, LangChain/LangGraph — sufficient to understand model behavior, prompt structure, and agent architecture deeply enough to test them rigorously.
• Familiarity with MLOps/LLMOps pipelines (MLflow, Databricks, SageMaker) and experience integrating automated quality gates into CI/CD workflows for AI systems.
• Experience with cloud AI infrastructure (AWS, Azure, or GCP) and observability tooling for monitoring live AI system behavior and output quality in production.
• Strong analytical, communication, and stakeholder management skills — with the ability to translate complex AI failure patterns into clear risk assessments and remediation recommendations for both technical and business audiences.
Mastercard is a merit-based, inclusive, equal opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact View email address on decentrajobs.com and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.
Corporate Security Responsibility All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard’s security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
In line with Mastercard’s total compensation philosophy and assuming that the job will be performed in the US, the successful candidate will be offered a competitive base salary and may be eligible for an annual bonus or commissions depending on the role. The base salary offered may vary depending on multiple factors, including but not limited to location, job-related knowledge, skills, and experience. Mastercard benefits for full time (and certain part time) employees generally include: insurance (including medical, prescription drug, dental, vision, disability, life insurance); flexible spending account and health savings account; paid leaves (including 16 weeks of new parent leave and up to 20 days of bereavement leave); 80 hours of Paid Sick and Safe Time, 25 days of vacation time and 5 personal days, pro-rated based on date of hire; 10 annual paid U.S. observed holidays; 401k with a best-in-class company match; deferred compensation for eligible roles; fitness reimbursement or on-site fitness facilities; eligibility for tuition reimbursement; and many more. Mastercard benefits for interns generally include: 56 hours of Paid Sick and Safe Time; jury duty leave; and on-site fitness facilities in some locations.
Pay RangesO'Fallon, Missouri: $140,000 - $231,000 USD
$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering We are seeking a visionary and results-driven Engineering Manager to lead a service group in our critical Decision...SuggestedFull timePart timeWorldwideFlexible hours$790 per week
Try products before they hit shelves — and get paid for your feedback. Focus Group Panel matches you with paid product tests and remote research from brands you already know. Active members earn up to $790 a week on the studies they qualify for. No experience needed....SuggestedFull timePart timeRemote work$500 per week
...assignment. Up to $500 per week. Location: Remote (USA) Company: ProductReviewJobs Thank you for your interest in becoming a Paid Product Tester. This opportunity is for completing market research opportunities with independent brands via online or phone. Online studies...SuggestedRemote work$99k - $225k
Job Number: R0241538 Agentic AI Developer Opportunity As an experienced engineer, you know that machine learning is critical to understanding and processing... ...and through the development, deployment, and management of AI agents. You'll be part of a talented team of...SuggestedLocal area$99k - $225k
Agentic AI Developer Your growth matters to us - explore our career development opportunities... ...us. The Opportunity As an experienced engineer, you know that machine learning is... ...through the development, deployment, and management of AI agents. You’ll be part of a talented...SuggestedFull timeContract workPart timeWork at officeLocal areaRemote work$115k - $184k
...realize their greatest potential. Title and Summary Senior AI Engineer Mastercard's Business & Market Insights (B&MI) group empowers... ...Gen AI operations. • Engineer state, memory, and context management subsystems that enable agents to reason temporally, maintain...Full timePart timeWorldwideFlexible hours- ...Job Description Job Description Job Title : AI Developer Location: O’Fallon, MO (On-site) Company : Milestone Marketing... ...., customer segmentation, predictive analytics, personalization engines) Build and maintain data pipelines to support AI initiatives...
$140k - $266k
...governments realize their greatest potential. Title and Summary Lead AI Engineer Overview: This role seeks an established AI engineering... ...capable of reasoning, planning, and safely invoking tools; managing memory; and continuously improving through systematic...Full timePart timeWorldwideFlexible hours$140k - $231k
...governments realize their greatest potential. Title and Summary Lead AI Engineer Job Description Summary Overview As a Lead AI... ...• Excellent communication, interpersonal, and stakeholder management skills, with the ability to effectively articulate complex technical...Full timePart timeWorldwideFlexible hours$115k - $184k
...greatest potential. Title and Summary Senior Software Engineer Who is Mastercard? Mastercard is a global technology... ...and identity verification solutions on Mastercard's Decision Management Platform. Our high-performance Java platform processes billions...Full timePart timeWorldwideFlexible hours$115k - $184k
...Senior Software Development Engineer Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together... ..., cloud-native applications, and CI/CD. Experience building AI-assisted applications, autonomous agents, or agentic workflows....Worldwide- ...Identify and implement complex automation efforts, including refactoring of automation code where needed. Work closely with software engineers to build quality in, ensuring proper test and requirement coverage, configuration quality, monitoring, early performance testing....
- ...Software Development Engineer II Location: O'Fallon, Missouri (1000 Technology) Duration: 12 Months Job Overview • Responsible... .... • Perform system maintenance, production incident problem management, identification of root cause, remediation of the problems and...Work experience placement
$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering About the Role This is an exciting opportunity to join Mastercard's global credit card authorization program. We...Full timePart timeWorldwideFlexible hours$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering Who is Mastercard? We work to connect and power an inclusive, digital economy that benefits everyone, everywhere,...Full timePart timeWorldwideFlexible hours$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering We are seeking a visionary and results-driven Engineering Manager to lead a service group in our critical Decision...Full timePart timeWorldwideFlexible hours$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager Software Engineering Overview Mastercard is seeking a Manager, Software Engineering to lead a high-performing team responsible for...Full timePart timeWorldwideFlexible hours$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering Our Purpose We connect and power an inclusive digital economy that benefits everyone, everywhere. We enable safe,...Full timePart timeWorldwideFlexible hours$140k - $231k
...governments realize their greatest potential. Title and Summary Manager, Software Engineering Job Description Summary Job Overview: Providing... ...-to-day support duties. Embraces and champions use of AI across the group. All About You: ~ Proven experience...Full timePart timeWorldwideFlexible hours$212k - $339k
...and Summary Vice President, Software Engineering - DMP Overview: Mastercard is seeking... ...Decisioning domain within the Decision Management program. This role is accountable for... ...spanning real-time decisioning, rules engines, AI/ML inference, data pipelines, and...Full timePart timeWorldwideFlexible hours3 days per week$96k - $163k
...greatest potential. Title and Summary Senior Site Reliability Engineer Who is Mastercard? At Mastercard technology, we work to... ...proactive and upfront in the development process, and to proactively manage production and change activities to maximize customer...Full timePart timeWorldwideFlexible hours$75k
...EC2, and Lambda, with strong experience in Python and shell scripting to automate cloud operations and streamline infrastructure management. Possession of an AWS Certified Developer – Associate certification is a strong plus. Experience in software development or integration...Work at officeWork from homeHome office- Sr. Java Developer Key Skills and Experience: ~ Strong hands on experience in Java(J2ee) ~ Strong in webservices(microservices), Rest API's, Oracle DB knowledge, Cloud native(preferably AWS or Azure) ~ Ability to drive end to end development, performance code...
- ...Senior Java Engineer Senior Java Engineer O’Fallon MO – Day 1 Onsite BS in CS 10+ yrs. experience Experience in object-oriented programming and concepts. Experienced in Java backend application development. Comfortable enough in handling algorithms,...
- Java Developer Location: O’Fallon, MO – Day 1 Onsite – No Remote Duration: Long Term Passport copy mandatory to submit the profile Job Description: • Overall 8+ years of experience must • Experience in object-oriented programming and concepts • Experienced...Remote work
$140k - $231k
...Title and Summary Lead Software Development Engineer Overview The Lead Software Development... ...initiatives, and adoption of Agentic AI-powered engineering practices. What You... ...workflows. - Strong communication, stakeholder management, and technical leadership skills....Full timePart timeWorldwideFlexible hours- ...Lead Software Development Engineer Location: O’Fallon, MO Duration: 24 months Interview process: First round phone, second round... .... What are your top 3 required technical skills? Manage and develop system integration and functional testing strategies...
- ...preferred. • Advanced knowledge and understanding of Software Engineering Concepts and Methodologies is required. • Strong... ...written; able to communicate with technical people as well as management and business Has skills to document and coach team on the...Work experience placement
- ...Products Team Lead Be part of the Operations & Technology Fraud Products team developing new capabilities for MasterCard's Decision Management Platform, which serves as the core for multiple business solutions to combat fraud and validate cardholder identity. Our patented...Local areaRemote workRelocation
- ...Amyx is seeking a Software Engineering Lead to join our Dept. of Defense client at Scott Air Force Base in O'Fallon, IL. Responsibilities... ...delivery activities. Supports Agile ceremonies, backlog management, and documentation. Coordinates software development support...Temporary workFor contractorsFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Manager, AI Engineering (Tester ). Be the first to apply!



