AI QA Engineer

AI QA EngineerWe are seeking an AI QA Engineer to ensure the quality, reliability, safety, and performance of AI-powered applications and machine learning solutions. The ideal candidate will design and execute testing strategies for AI systems, including large language model (LLM) applications, generative AI features, machine learning models, and AI-driven workflows. This role requires expertise in traditional software quality assurance combined with an understanding of AI model behavior, prompt engineering, and automated testing.Key ResponsibilitiesDesign and implement test strategies for AI and machine learning applications.Develop and maintain automated test frameworks for AI-powered systems.Validate LLM responses for accuracy, consistency, relevance, and safety.Create and manage test datasets, prompts, and evaluation benchmarks.Test AI features for functionality, usability, performance, security, and reliability.Identify, document, and track defects using bug management tools.Perform regression, integration, API, end-to-end, and performance testing.Evaluate AI systems for hallucinations, bias, toxicity, and prompt injection vulnerabilities.Collaborate with AI engineers, data scientists, product managers, and developers to resolve issues.Analyze model outputs and recommend improvements to prompts or workflows.Monitor AI application performance after deployment.Ensure compliance with organizational quality standards and AI governance policies.Required QualificationsBachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.3–5+ years of experience in software quality assurance or test automation.Experience testing web, mobile, or API-based applications.Familiarity with AI, machine learning, or generative AI applications.Strong understanding of software testing methodologies and QA best practices.Experience with Agile or Scrum development environments.Preferred QualificationsExperience testing LLM-based or generative AI applications.Knowledge of AI evaluation frameworks and benchmarking techniques.Familiarity with prompt engineering and AI model behavior.AI or cloud certifications (AWS, Microsoft Azure, Google Cloud, etc.).Experience with MLOps or AI deployment pipelines.Technical SkillsManual and automated testingTest case design and executionAPI testing (Postman, REST Assured)Automation tools (Selenium, Playwright, Cypress)Programming (Python, Java, JavaScript, or C#)SQL and database validationGit and CI/CD pipelinesTest management tools (Jira, TestRail, Zephyr)Performance testing (JMeter, k6)AI model evaluation techniquesPrompt engineeringLLM testing and validationAI safety testing (hallucinations, bias, toxicity, prompt injection)Basic machine learning conceptsJSON and REST APIsSoft SkillsAnalytical thinkingStrong attention to detailProblem-solvingCommunication and documentationCollaboration with cross-functional teamsCritical thinkingContinuous learningTime managementPreferred ExperienceAI-powered enterprise applicationsGenerative AI productsMachine learning platformsSaaS applicationsConversational AI or chatbot testingCloud-native applicationsSuccess MetricsDefect detection rateTest automation coverageAI response quality and accuracyReduction in production defectsTest execution efficiencyAI safety and compliance metricsRelease qualityCustomer satisfactionMean time to detect (MTTD) and resolve (MTTR) issuesNice-to-Have SkillsLangChain or similar AI orchestration frameworksAI evaluation frameworks (e.g., DeepEval, Ragas, LangSmith)Vector databasesDocker and KubernetesMLOps toolsCloud platforms (AWS, Azure, Google Cloud)Data annotation and synthetic data generationSecurity testing for AI systemsAccessibility testingExperience with Retrieval-Augmented Generation (RAG) systems