Remote AI Test Lead: GenAI & AI App QA Leader

AI Test Lead

Experience: 7+ Years
Location: Remote

Position Summary

We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands-on exposure to AI/ML, Generative AI, LLM, and RAG-based applications .
The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.

Key Responsibilities

Lead end-to-end testing activities for AI/ML and Generative AI applications.
Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
Design and implement automated testing frameworks for web, API, and AI-based applications.
Test LLM, RAG, prompt-based applications, chatbots, and AI agents .
Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
Develop test datasets and evaluation methodologies for AI/GenAI applications.
Perform API, functional, regression, integration, and end-to-end testing.
Collaborate with developers, data scientists, product owners, and business stakeholders.
Identify and troubleshoot defects and quality issues across application and AI components.
Integrate automated tests into CI/CD pipelines .
Mentor QA engineers and provide technical leadership to the testing team.
Establish quality metrics, reporting, and continuous improvement practices.
Ensure testing standards and best practices are followed across projects.

Required Skills

7+ years of experience in Software Testing / QA / Test Automation .
Hands-on experience in AI/GenAI/LLM testing .
Strong experience with Selenium, Playwright, Cypress , or similar automation tools.
Strong programming skills in Python or Java .
Experience in API testing using REST, Postman, Rest Assured, or similar tools.
Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications .
Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality .
Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals , or equivalent.
Good understanding of SQL and database testing.
Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.

Good to Have

Experience testing LangChain / LangGraph applications.
Knowledge of vector databases and RAG architectures.
Exposure to cloud platforms such as AWS, Azure, or GCP .
Knowledge of AI security testing and responsible AI principles.
Experience in performance and security testing.
Previous experience leading or mentoring QA teams.