AI TEST LEAD

AI Test Lead

Experience: 7+ Years
Location: Remote

Position Summary

We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands‑on exposure to AI/ML, Generative AI, LLM, and RAG-based applications .
The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.

Key Responsibilities

• Lead end-to-end testing activities for AI/ML and Generative AI applications.
• Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
• Design and implement automated testing frameworks for web, API, and AI-based applications.
• Test LLM, RAG, prompt-based applications, chatbots, and AI agents .
• Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
• Develop test datasets and evaluation methodologies for AI/GenAI applications.
• Perform API, functional, regression, integration, and end-to-end testing.
• Collaborate with developers, data scientists, product owners, and business stakeholders.
• Identify and troubleshoot defects and quality issues across application and AI components.
• Integrate automated tests into CI/CD pipelines .
• Mentor QA engineers and provide technical leadership to the testing team.
• Establish quality metrics, reporting, and continuous improvement practices.
• Ensure testing standards and best practices are followed across projects.

Required Skills

• 7+ years of experience in Software Testing / QA / Test Automation .
• Hands-on experience in AI/GenAI/LLM testing .
• Strong experience with Selenium, Playwright, Cypress , or similar automation tools.
• Strong programming skills in Python or Java .
• Experience in API testing using REST, Postman, Rest Assured, or similar tools.
• Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications .
• Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality .
• Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals , or equivalent.
• Good understanding of SQL and database testing.
• Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
• Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.

Good to Have

• Experience testing LangChain / LangGraph applications.
• Knowledge of vector databases and RAG architectures.
• Exposure to cloud platforms such as AWS, Azure, or GCP .
• Knowledge of AI security testing and responsible AI principles.
• Experience in performance and security testing.
• Previous experience leading or mentoring QA teams.