Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Manager, AI Engineering (Tester )

$140k - $231k
Full-time

Mastercard

Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary

Manager, AI Engineering (Tester )

Mastercard's Business & Market Insights (B&MI) group delivers unparalleled data-driven intelligence and frontier AI solutions that help organizations make smarter, faster, and more impactful decisions. We are currently looking for a AI Tester for the Operational Intelligence Program within B&MI. This is a highly specialized, hands-on AI testing leadership position dedicated to ensuring our Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. This role will lead AI quality engineering efforts — defining evaluation frameworks, red-teaming strategies, and LLMOps quality gates — while fostering a culture of rigorous, first-class AI testing across the program.

Roles and Responsibilities:
• Design and own end-to-end LLM evaluation frameworks — including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection across model versions and prompt variations.
• Build comprehensive test suites for agentic AI systems — validating tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling across multi-step reasoning workflows.
• Develop RAG pipeline evaluation frameworks assessing retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using tools like RAGAS, TruLens, and DeepEval.
• Lead structured red-teaming and adversarial testing exercises targeting prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation — building and maintaining an evolving adversarial test library.
• Execute fairness, bias, and Responsible AI audits — testing for demographic bias, sentiment skew, representation gaps, and validating explainability mechanisms, citations, and confidence score accuracy.
• Design and run inference performance benchmarks — measuring latency, throughput, token efficiency, and degradation under peak load — and enforce LLM quality gates within CI/CD pipelines on Databricks (AWS).
• Build production monitoring and drift detection pipelines tracking semantic output drift, embedding shifts, retrieval degradation, and anomalous agent behaviors using observability tooling (Grafana, Datadog, CloudWatch).
• Define the AI testing roadmap and quality standards for the program — establishing evaluation metrics, tooling choices, and documentation practices across all Gen AI workstreams.
• Partner with Gen AI engineers, ML engineers, and product stakeholders to embed quality from day one — reviewing prompt architectures, agent designs, and system workflows for testability and risk.
• Continuously research and adopt frontier evaluation benchmarks (RAGAS, MMLU, TruthfulQA, MT-Bench) and emerging AI testing methodologies to keep quality practices at the cutting edge.

All About You:
• Master's/Bachelor's degree in Computer Science, AI/ML, or Software Engineering, with considerable hands-on experience leading AI/ML quality engineering or LLM testing programs in production environments.
• Demonstrated expertise testing LLM and Gen AI systems — including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation in real production settings.
• Deep hands-on knowledge of AI evaluation frameworks and tooling: RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, Weights & Biases Evals, or equivalent platforms.
• Strong understanding of Gen AI failure modes — hallucination, prompt injection, retrieval grounding failures, context drift, agent loop failures — and proven methods to surface and document them systematically.
• Strong Python programming skills with the ability to independently build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.
• Working knowledge of LLM ecosystems — OpenAI, Anthropic, Hugging Face, LangChain/LangGraph — sufficient to understand model behavior, prompt structure, and agent architecture deeply enough to test them rigorously.
• Familiarity with MLOps/LLMOps pipelines (MLflow, Databricks, SageMaker) and experience integrating automated quality gates into CI/CD workflows for AI systems.
• Experience with cloud AI infrastructure (AWS, Azure, or GCP) and observability tooling for monitoring live AI system behavior and output quality in production.
• Strong analytical, communication, and stakeholder management skills — with the ability to translate complex AI failure patterns into clear risk assessments and remediation recommendations for both technical and business audiences.

Mastercard is a merit-based, inclusive, equal opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact View email address on decentrajobs.com and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

  • Abide by Mastercard’s security policies and practices;

  • Ensure the confidentiality and integrity of the information being accessed;

  • Report any suspected information security violation or breach, and

  • Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

In line with Mastercard’s total compensation philosophy and assuming that the job will be performed in the US, the successful candidate will be offered a competitive base salary and may be eligible for an annual bonus or commissions depending on the role. The base salary offered may vary depending on multiple factors, including but not limited to location, job-related knowledge, skills, and experience. Mastercard benefits for full time (and certain part time) employees generally include: insurance (including medical, prescription drug, dental, vision, disability, life insurance); flexible spending account and health savings account; paid leaves (including 16 weeks of new parent leave and up to 20 days of bereavement leave); 80 hours of Paid Sick and Safe Time, 25 days of vacation time and 5 personal days, pro-rated based on date of hire; 10 annual paid U.S. observed holidays; 401k with a best-in-class company match; deferred compensation for eligible roles; fitness reimbursement or on-site fitness facilities; eligibility for tuition reimbursement; and many more. Mastercard benefits for interns generally include: 56 hours of Paid Sick and Safe Time; jury duty leave; and on-site fitness facilities in some locations.

Pay Ranges

O'Fallon, Missouri: $140,000 - $231,000 USD

Vacancy posted 24 days ago
Similar jobs that could be interesting for youBased on the Manager, AI Engineering (Tester ) in O Fallon, MO vacancy
  • $115k - $184k

     ...potential. Title and Summary Senior AI EngineerOverview The Security...  ...activity in real time. As a Senior AI Engineer, you will work at the intersection of AI...  ...-Fraud, cybersecurity, payments, or risk management domains. -Experience with Generative AI... 
    Suggested
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    3 days ago
  •  ...Job Description Job Description Job Title : AI Developer Location: O’Fallon, MO (On-site) Company : Milestone Marketing...  ...., customer segmentation, predictive analytics, personalization engines) Build and maintain data pipelines to support AI initiatives... 
    Suggested

    Automotive Product Consultants LLC

    O Fallon, MO
    more than 2 months ago
  • $99k - $225k

    Phase2 Technology in O'Fallon, Illinois, is seeking an Agentic AI Developer who will be a part of the Defense Technology team. The role requires expertise in machine learning to develop and deploy AI solutions that tackle real-world challenges faced by the Department of... 
    Suggested

    Phase2 Technology

    O Fallon, MO
    1 day ago
  • $99k - $225k

    Booz Allen Hamilton is seeking an Agentic AI Developer in O'Fallon, Illinois. This role involves applying...  ...expertise in ML algorithms to develop, deploy, and manage AI solutions in collaboration with a team of engineers. The ideal candidate will have at least 3 years of... 
    Suggested

    Booz Allen Hamilton

    O Fallon, MO
    17 hours ago
  • $140k - $231k

     ...Manager Software Engineering Overview Mastercard is seeking a Manager, Software Engineering to lead a high-performing team responsible for building, enhancing, and supporting enterprise software solutions, internal engineering platforms, and workflow automation... 
    Suggested

    Dynamic Yield

    O Fallon, MO
    3 days ago
  • $140k - $231k

     ...and services that help people, businesses and governments realize their greatest potential. Title and Summary Manager, Software Engineering We are seeking a visionary and results-driven Engineering Manager to lead a service group in our critical Decision Management... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    17 hours ago
  • $25 - $45 per hour

     ...Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies. We guarantee 15-25 hours per week with an hourly pay of between $25/hr. and $45/hr., depending on the In-Home Usage Test project.... 
    Hourly pay
    Weekly pay
    Extra income
    Temporary work
    Part time
    For contractors
    Seasonal work
    Remote work
    Work from home
    Flexible hours

    Online Consumer Panels America

    O Fallon, MO
    17 hours ago
  • $25 - $45 per hour

     ...Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies. We guarantee 15-25 hours per week with an hourly pay of between $25/hr. and $45/hr., depending on the In-Home Usage Test project.... 
    Hourly pay
    Weekly pay
    Extra income
    Temporary work
    Part time
    For contractors
    Seasonal work
    Remote work
    Work from home
    Flexible hours

    Online Consumer Panels America

    Dardenne Prairie, MO
    17 hours ago
  • $500 per week

     ...assignment. Up to $500 per week. Location: Remote (USA) Company: ProductReviewJobs Thank you for your interest in becoming a Paid Product Tester. This opportunity is for completing market research opportunities with independent brands via online or phone. Online studies... 
    Remote work

    Arroyo Media

    O Fallon, MO
    4 days ago
  • $790 per week

    Try products before they hit shelves — and get paid for your feedback. Focus Group Panel matches you with paid product tests and remote research from brands you already know. Active members earn up to $790 a week on the studies they qualify for. No experience needed....
    Full time
    Part time
    Remote work

    Focus Group Panel

    O Fallon, MO
    1 day ago
  • $99k - $225k

    Agentic AI Developer Your growth matters to us - explore our career development opportunities...  ...us. The Opportunity As an experienced engineer, you know that machine learning is...  ...through the development, deployment, and management of AI agents. You’ll be part of a talented... 
    Full time
    Contract work
    Part time
    Work at office
    Local area
    Remote work

    Booz Allen Hamilton

    O Fallon, MO
    17 hours ago
  • $99k - $225k

    Job Number: R0241538 Agentic AI Developer Opportunity As an experienced engineer, you know that machine learning is critical to understanding and processing...  ...and through the development, deployment, and management of AI agents. You'll be part of a talented team of... 
    Local area

    Phase2 Technology

    O Fallon, MO
    1 day ago
  • $400 per week

    Type: Flexible, project-based Location: Remote (US) Pay: Earn up to $400/week, depending on the number and type of studies you complete. Share honest feedback on products and services from independent brands. Assignments include short online surveys ...
    Remote work
    Flexible hours

    BuzzTesters

    O Fallon, MO
    17 hours ago
  • AI Big Data Engineer We are seeking a highly skilled and experienced AI Big Data Engineer to design, develop, and optimize large-scale data...  ...including Test Automation, Build Automation and Configuration Management frameworks. Strong written and verbal technical... 
    Work experience placement
    3 days per week

    ManpowerGroup Global, Inc.

    Lake Saint Louis, MO
    2 days ago
  • ManpowerGroup Global, Inc. seeks an AI Big Data Engineer to design, develop, and optimize large-scale data processing systems across platforms. The role requires 3 days per week onsite in Rockville, MD or Tysons Corner, VA, with strong focus on Java/Scala/Python, Hadoop... 
    3 days per week

    ManpowerGroup Global, Inc.

    Lake Saint Louis, MO
    2 days ago
  • $115k - $184k

     ...realize their greatest potential. Title and Summary Senior AI Engineer Mastercard's Business & Market Insights (B&MI) group empowers...  ...Gen AI operations. • Engineer state, memory, and context management subsystems that enable agents to reason temporally, maintain... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    24 days ago
  • $170k - $281k

     ...Software EngineeringDirector, Software Engineering Responsible for leading a team of Java...  ...improvement. • Make presentations to senior management, business owners, and other stakeholders...  ..., clearing, and settlement • AI/GenAI exposure and familiarity with AI-assisted... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    4 days ago
  • $140k - $231k

     ...realize their greatest potential. Title and Summary Lead, AI Security Engineer Job Description Summary As a Lead Information...  ...technologies, application security, secure system design, and risk management, along with the ability to coordinate work across teams and... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    8 hours ago
  •  ...Director, Platform Engineering Mastercard is a global technology company in the payments...  ..., the pace of change across technology, AI, and the nature of work requires that we...  ...services, including CI/CD pipelines, secrets management, artifact repositories, and... 

    Dynamic Yield

    O Fallon, MO
    2 hours ago
  • $115k - $184k

     ...software applications while applying software engineering best practices, secure coding standards,...  ...system maintenance, production incident management, troubleshooting, root cause analysis,...  ...learning mindset with interest in AI-driven automation, innovation, and operational... 
    Full time
    Part time
    Work experience placement
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    3 days ago
  • $92k - $147k

     ...realize their greatest potential. Title and Summary Software Engineer II Who is Mastercard? Mastercard is a global technology...  ...such as OWASP, CWE, and SEI CERT. Knowledge of vulnerability management and experience utilizing code quality and security scanning... 
    Full time
    Part time
    Worldwide
    Flexible hours

    MasterCard

    O Fallon, MO
    2 hours ago
  •  ...Senior Software Engineer Location: O'Fallon, Missouri - onsite (LOCAL ONLY) Rate - $65/hr. on C2C Java/JEE 6-10 years Essential responsibilities of the position: Execute and own the baseline architecture implementations and design activities collaborating... 
    Work experience placement
    Internship
    Local area

    Diverse Lynx

    O Fallon, MO
    3 hours ago
  • $72k - $115k

     ...realize their greatest potential. Title and Summary Software Engineer I AI and Decision Product Enablement Be part of the Shared...  ...team developing new capabilities for Mastercard’s Decision Management Platform, which serves as the core for multiple business solutions... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    7 days ago
  • $92k - $147k

     ...Software Engineer II Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our...  ...diagnose and resolve technical issues systematically. • Ability to manage multiple priorities and meet deadlines in a fast-paced... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Dynamic Yield

    O Fallon, MO
    3 days ago
  • $115k - $184k

     ...their greatest potential. Title and Summary Senior Software Engineer Who is Mastercard? We work to connect and power an inclusive...  ...standards (e.g., OWASP, CWE, SEI CERT) and vulnerability management • Understands the basic engineering principles used in building... 
    Full time
    Contract work
    Part time
    Work experience placement
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    14 days ago
  • $115k - $184k

     ...products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Software Engineer Who is Mastercard? Mastercard is a global technology company in the payments industry. Our mission is to connect and power... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    4 days ago
  • Job Title About Us: At *** (*** | Connected World, Connected Experiences), we live the philosophy of connected world and connected experiences. We thrive on change that is powered by the intelligent symphony of technology and humans designing meaningful and sustainable...

    Yantran LLC

    O Fallon, MO
    2 days ago
  • $115k - $184k

     ...greatest potential. Title and Summary Senior Software Development Engineer Overview The Senior Software Development Engineer is a senior...  ..., cloud-native applications, and CI/CD. •Experience building AI-assisted applications, autonomous agents, or agentic workflows.... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    13 days ago
  • $92k - $147k

     ...services that help people, businesses and governments realize their greatest potential. Title and Summary Software Engineer II - Decision Management Program Position Summary The Notification Manager team is responsible for Mastercard's enterprise notification platform... 
    Full time
    Part time
    Worldwide
    Flexible hours

    Mastercard

    O Fallon, MO
    18 days ago
  •  ...Job Title Bachelor’s degree in information systems, information technology, computer science, engineering, or equivalent work experience. More than 7 years of industry experience. Excellent communication skills, with the ability to interact across multiple organizational... 
    Work experience placement

    Saxon Global

    O Fallon, MO
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Manager, AI Engineering (Tester ). Be the first to apply!