Manager, AI Engineering (Tester )
$140k - $231kMastercard
Manager, AI Engineering (Tester )
Mastercard's Business & Market Insights (B&MI) group delivers unparalleled data-driven intelligence and frontier AI solutions that help organizations make smarter, faster, and more impactful decisions. We are currently looking for a AI Tester for the Operational Intelligence Program within B&MI. This is a highly specialized, hands-on AI testing leadership position dedicated to ensuring our Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. This role will lead AI quality engineering efforts — defining evaluation frameworks, red-teaming strategies, and LLMOps quality gates — while fostering a culture of rigorous, first-class AI testing across the program.
Roles and Responsibilities:• Design and own end-to-end LLM evaluation frameworks — including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection across model versions and prompt variations.
• Build comprehensive test suites for agentic AI systems — validating tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling across multi-step reasoning workflows.
• Develop RAG pipeline evaluation frameworks assessing retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using tools like RAGAS, TruLens, and DeepEval.
• Lead structured red-teaming and adversarial testing exercises targeting prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation — building and maintaining an evolving adversarial test library.
• Execute fairness, bias, and Responsible AI audits — testing for demographic bias, sentiment skew, representation gaps, and validating explainability mechanisms, citations, and confidence score accuracy.
• Design and run inference performance benchmarks — measuring latency, throughput, token efficiency, and degradation under peak load — and enforce LLM quality gates within CI/CD pipelines on Databricks (AWS).
• Build production monitoring and drift detection pipelines tracking semantic output drift, embedding shifts, retrieval degradation, and anomalous agent behaviors using observability tooling (Grafana, Datadog, CloudWatch).
• Define the AI testing roadmap and quality standards for the program — establishing evaluation metrics, tooling choices, and documentation practices across all Gen AI workstreams.
• Partner with Gen AI engineers, ML engineers, and product stakeholders to embed quality from day one — reviewing prompt architectures, agent designs, and system workflows for testability and risk.
• Continuously research and adopt frontier evaluation benchmarks (RAGAS, MMLU, TruthfulQA, MT-Bench) and emerging AI testing methodologies to keep quality practices at the cutting edge. All About You:
• Master's/Bachelor's degree in Computer Science, AI/ML, or Software Engineering, with considerable hands-on experience leading AI/ML quality engineering or LLM testing programs in production environments.
• Demonstrated expertise testing LLM and Gen AI systems — including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation in real production settings.
• Deep hands-on knowledge of AI evaluation frameworks and tooling: RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, Weights & Biases Evals, or equivalent platforms.
• Strong understanding of Gen AI failure modes — hallucination, prompt injection, retrieval grounding failures, context drift, agent loop failures — and proven methods to surface and document them systematically.
• Strong Python programming skills with the ability to independently build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.
• Working knowledge of LLM ecosystems — OpenAI, Anthropic, Hugging Face, LangChain/LangGraph — sufficient to understand model behavior, prompt structure, and agent architecture deeply enough to test them rigorously.
• Familiarity with MLOps/LLMOps pipelines (MLflow, Databricks, SageMaker) and experience integrating automated quality gates into CI/CD workflows for AI systems.
• Experience with cloud AI infrastructure (AWS, Azure, or GCP) and observability tooling for monitoring live AI system behavior and output quality in production.
• Strong analytical, communication, and stakeholder management skills — with the ability to translate complex AI failure patterns into clear risk assessments and remediation recommendations for both technical and business audiences.
Mastercard is a merit-based, inclusive, equal opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact View email address on decentrajobs.com and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.
Corporate Security Responsibility All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard’s security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.
In line with Mastercard’s total compensation philosophy and assuming that the job will be performed in the US, the successful candidate will be offered a competitive base salary and may be eligible for an annual bonus or commissions depending on the role. The base salary offered may vary depending on multiple factors, including but not limited to location, job-related knowledge, skills, and experience. Mastercard benefits for full time (and certain part time) employees generally include: insurance (including medical, prescription drug, dental, vision, disability, life insurance); flexible spending account and health savings account; paid leaves (including 16 weeks of new parent leave and up to 20 days of bereavement leave); 80 hours of Paid Sick and Safe Time, 25 days of vacation time and 5 personal days, pro-rated based on date of hire; 10 annual paid U.S. observed holidays; 401k with a best-in-class company match; deferred compensation for eligible roles; fitness reimbursement or on-site fitness facilities; eligibility for tuition reimbursement; and many more. Mastercard benefits for interns generally include: 56 hours of Paid Sick and Safe Time; jury duty leave; and on-site fitness facilities in some locations.
Pay RangesO'Fallon, Missouri: $140,000 - $231,000 USD
$115k - $184k
...realize their greatest potential. Title and Summary Senior AI Engineer Overview The Security Solutions Data Science team develops... ...Qualifications -Fraud, cybersecurity, payments, or risk management domains. -Experience with Generative AI, LLMs, RAG, or agentic...SuggestedFull timePart timeWorldwideFlexible hours- ...Job Description Job Description Job Title : AI Developer Location: O’Fallon, MO (On-site) Company : Milestone Marketing... ...., customer segmentation, predictive analytics, personalization engines) Build and maintain data pipelines to support AI initiatives...Suggested
$99k - $225k
Phase2 Technology in O'Fallon, Illinois, is seeking an Agentic AI Developer who will be a part of the Defense Technology team. The role requires expertise in machine learning to develop and deploy AI solutions that tackle real-world challenges faced by the Department of...Suggested$99k - $225k
Booz Allen Hamilton is seeking an Agentic AI Developer in O'Fallon, Illinois. This role involves applying... ...expertise in ML algorithms to develop, deploy, and manage AI solutions in collaboration with a team of engineers. The ideal candidate will have at least 3 years of...Suggested$140k - $231k
...and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering We are seeking a visionary and results-driven Engineering Manager to lead a service group in our critical Decision Management...SuggestedFull timePart timeWorldwideFlexible hours$99k - $225k
Job Number: R0241538 Agentic AI Developer Opportunity As an experienced engineer, you know that machine learning is critical to understanding and processing... ...and through the development, deployment, and management of AI agents. You'll be part of a talented team of...Local area$99k - $225k
Agentic AI Developer Your growth matters to us - explore our career development opportunities... ...us. The Opportunity As an experienced engineer, you know that machine learning is... ...through the development, deployment, and management of AI agents. You’ll be part of a talented...Full timeContract workPart timeWork at officeLocal areaRemote work$115k - $184k
...realize their greatest potential. Title and Summary Senior AI Engineer Mastercard's Business & Market Insights (B&MI) group empowers... ...Gen AI operations. • Engineer state, memory, and context management subsystems that enable agents to reason temporally, maintain...Full timePart timeWorldwideFlexible hours$45 per hour
...Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies. We guarantee 15-25 hours per week with an hourly pay of between $25/hr. and $45/hr., depending on the In-Home Usage Test project....Hourly payWeekly payExtra incomeTemporary workPart timeFor contractorsSeasonal workRemote workWork from homeFlexible hours$500 per week
...assignment. Up to $500 per week. Location: Remote (USA) Company: ProductReviewJobs Thank you for your interest in becoming a Paid Product Tester. This opportunity is for completing market research opportunities with independent brands via online or phone. Online studies...Remote work$790 per week
Try products before they hit shelves — and get paid for your feedback. Focus Group Panel matches you with paid product tests and remote research from brands you already know. Active members earn up to $790 a week on the studies they qualify for. No experience needed....Full timePart timeRemote work$25 - $45 per hour
...Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies. We guarantee 15-25 hours per week with an hourly pay of between $25/hr. and $45/hr., depending on the In-Home Usage Test project....Hourly payWeekly payExtra incomeTemporary workPart timeFor contractorsSeasonal workRemote workWork from homeFlexible hours$400 per week
Type: Flexible, project-based Location: Remote (US) Pay: Earn up to $400/week, depending on the number and type of studies you complete. Share honest feedback on products and services from independent brands. Assignments include short online surveys ...Remote workFlexible hours- AI Big Data Engineer We are seeking a highly skilled and experienced AI Big Data Engineer to design, develop, and optimize large-scale data... ...including Test Automation, Build Automation and Configuration Management frameworks. Strong written and verbal technical...Work experience placement3 days per week
- ManpowerGroup Global, Inc. seeks an AI Big Data Engineer to design, develop, and optimize large-scale data processing systems across platforms. The role requires 3 days per week onsite in Rockville, MD or Tysons Corner, VA, with strong focus on Java/Scala/Python, Hadoop...3 days per week
$140k - $231k
...realize their greatest potential. Title and Summary Lead, AI Security Engineer Job Description Summary As a Lead Information... ...technologies, application security, secure system design, and risk management, along with the ability to coordinate work across teams and...Full timePart timeWorldwideFlexible hours$170k - $281k
...Software EngineeringDirector, Software Engineering Responsible for leading a team of Java... ...improvement. • Make presentations to senior management, business owners, and other stakeholders... ..., clearing, and settlement • AI/GenAI exposure and familiarity with AI-assisted...Full timePart timeWorldwideFlexible hours$92k - $147k
...realize their greatest potential. Title and Summary Software Engineer II Who is Mastercard? Mastercard is a global technology... ...such as OWASP, CWE, and SEI CERT. Knowledge of vulnerability management and experience utilizing code quality and security scanning...Full timePart timeWorldwideFlexible hours$115k - $184k
...software applications while applying software engineering best practices, secure coding standards,... ...system maintenance, production incident management, troubleshooting, root cause analysis,... ...learning mindset with interest in AI-driven automation, innovation, and operational...Full timePart timeWork experience placementWorldwideFlexible hours- ...Software Engineer II Our client, a credit card company, is looking for a Software Engineer... ...to communicate with all levels of management Ability to build rapport and relationships... ...and JPA You have experience in building AI assisted application, autonomous and semi...Live in
$115k - $184k
...their greatest potential. Title and Summary Senior Software Engineer Who is Mastercard? We work to connect and power an inclusive... ...standards (e.g., OWASP, CWE, SEI CERT) and vulnerability management • Understands the basic engineering principles used in building...Full timeContract workPart timeWork experience placementWorldwideFlexible hours$115k - $184k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Software Engineer Who is Mastercard? Mastercard is a global technology company in the payments industry. Our mission is to connect and power...Full timePart timeWorldwideFlexible hours$72k - $115k
...realize their greatest potential. Title and Summary Software Engineer I AI and Decision Product Enablement Be part of the Shared... ...team developing new capabilities for Mastercard’s Decision Management Platform, which serves as the core for multiple business solutions...Full timePart timeWorldwideFlexible hours$115k - $184k
...greatest potential. Title and Summary Senior Software Development Engineer Overview The Senior Software Development Engineer is a senior... ..., cloud-native applications, and CI/CD. •Experience building AI-assisted applications, autonomous agents, or agentic workflows....Full timePart timeWorldwideFlexible hours$92k - $147k
...services that help people, businesses and governments realize their greatest potential. Title and Summary Software Engineer II - Decision Management Program Position Summary The Notification Manager team is responsible for Mastercard's enterprise notification platform...Full timePart timeWorldwideFlexible hours$115k - $184k
...greatest potential. Title and Summary Senior Software Engineer Who is Mastercard? Mastercard is a global technology... ...and identity verification solutions on Mastercard's Decision Management Platform. Our high-performance Java platform processes billions...Full timePart timeWorldwideFlexible hours$115k - $184k
...Title and Summary Senior Software Engineer Who is Mastercard? Mastercard is a global... ...for all. Overview The Decision Management Program (DMP) delivers industry‑leading intelligence... ...• Demonstrated track record with AI • Familiarity with performance tuning...Full timePart timeWork experience placementWorldwideFlexible hours$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager Software Engineering Overview Mastercard is seeking a Manager, Software Engineering to lead a high-performing team responsible for...Full timePart timeWorldwideFlexible hours$140k - $231k
...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering Who is Mastercard? We work to connect and power an inclusive, digital economy that benefits everyone, everywhere,...Full timePart timeWorldwideFlexible hours$140k - $231k
...and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering We are seeking a visionary and results-driven Engineering Manager to lead a service group in our critical Decision Management...Full timePart timeWorldwideFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Manager, AI Engineering (Tester ). Be the first to apply!



