Quality Assurance Tester

Global Markets GenAI QA Tester

Location: New York, NY (2BP – onsite interview required)

Duration: 18 months

Work Arrangement: W2 contract only

Experience: 5–10 years

**W2 ONLY NO SPONSORSHIP ALLOWED**

Job Description

Bank of America is seeking a QA Tester with experience in Generative AI, test automation, and Python-based applications to support its Global Markets GenAI platform. This role will focus on validating LLM outputs, evaluating model quality, and testing the underlying AI infrastructure to ensure reliability, accuracy, and performance.

The platform, Orchestra, enables secure access to external large language models (LLMs) while enforcing user authorization and controlled integrations. The ideal candidate has hands-on experience applying AI to testing, building automation solutions, and evaluating LLM responses.

Key Responsibilities

• Develop and execute test strategies, test cases, and automated test suites for AI-powered applications and infrastructure.
• Evaluate LLM outputs and establish methods to measure response quality, accuracy, and reliability.
• Apply LLM-as-a-Judge techniques to assess model responses and identify quality issues.
• Develop or leverage AI agents to automate testing workflows and improve testing efficiency.
• Automate Python application testing using PyTest and Playwright.
• Use tools such as LangChain and LangSmith to inspect, trace, and evaluate model outputs.
• Perform database testing and data validation across MongoDB and other data platforms.
• Manage defects, report testing progress, and escalate risks and issues.
• Collaborate with developers, data scientists, product owners, and platform teams to support releases and continuous improvement.
• Ensure testing practices comply with enterprise security, data privacy, and governance requirements.

Required Qualifications

• 5–10 years of experience in QA, software testing, test automation, or a related technical role.
• Hands-on experience testing Python applications using PyTest .
• Experience with Playwright for test automation.
• Experience with MongoDB or other NoSQL databases.
• Practical experience using LLMs in testing, including LLM-as-a-Judge techniques or AI-agent-based test automation.
• Understanding of how to evaluate LLM responses and measure model quality.
• Familiarity with AI testing tools and frameworks such as LangChain and/or LangSmith .
• Strong analytical, troubleshooting, communication, and problem-solving skills.

Preferred Qualifications

• Experience testing AI/ML applications or Generative AI platforms.
• Exposure to SQL, NoSQL databases, and data validation.
• Familiarity with Azure, Google Cloud Platform (GCP), or AWS.
• Experience working in Agile environments using Jira and DevOps practices.
• Financial services or banking industry experience.
• Experience collaborating with cross-functional and onshore/offshore teams.