AI Quality Engineer

AI Quality EngineerThis is a HYBRID position requiring 2 in-office days at local client office per week. Travel costs are not reimbursable. Duration: 6+ Months Only W2.We are seeking an AI Quality Engineer to join an innovative and high-impact team building AI capabilities within enterprise systems for a leading wealth management client. In this role, you will define, execute, and continuously improve quality engineering practices for AI-powered applications, including LLM-based assistants, retrieval solutions, and intelligent workflows. You will help ensure these solutions are reliable, accurate, secure, and fit for enterprise use by building robust testing strategies across model behavior, system integrations, user experience, and operational performance. This is an opportunity to shape the next generation of enterprise AI in a highly visible environment, where innovation, rigor, and business value all matter. The ideal candidate brings a quality-first mindset, strong testing discipline, and practical experience validating complex systems in regulated environments.Key Responsibilities:Design and execute test strategies for AI applications, including functional, integration, regression, usability, and non-functional testingValidate LLM-based solutions for accuracy, relevance, consistency, safety, latency, and task completion across realistic business scenariosCreate test cases, evaluation datasets, and quality benchmarks for prompts, retrieval pipelines, orchestration flows, and end-user interactionsPartner with AI engineers, product owners, and business stakeholders to define acceptance criteria and measurable quality standardsIdentify model failure modes, edge cases, hallucination risks, and workflow breakdowns, and recommend corrective actionsSupport automation of AI testing, monitoring, and quality reporting across development and production environmentsTest data flows, APIs, integrations, and system dependencies that support AI-enabled business processesContribute to governance and release readiness by documenting defects, risks, test results, and quality decisionsMonitor production feedback, user behavior, and operational metrics to help drive continuous improvementHelp establish repeatable AI quality engineering practices, tooling, and controls for enterprise deliveryRequired Qualifications:Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or related field6+ years of experience in quality engineering, software testing, QA automation, or related technical rolesExperience testing enterprise applications, APIs, data pipelines, or workflow-based systemsExperience evaluating AI or ML-based systems, including prompt behavior, output quality, or model-driven workflowsStrong understanding of test strategy, defect management, root cause analysis, and quality metricsFamiliarity with automation frameworks, test management tools, and CI/CD-aligned quality processesAbility to work across technical and business teams to define and validate expected outcomesPreferred Qualifications:Experience with LLM evaluation, prompt testing, RAG validation, or AI observability toolsFamiliarity with Python, SQL, APIs, or scripting for test automation and data validationExperience building synthetic datasets, golden test sets, or evaluation frameworks for AI use casesExposure to responsible AI, model risk, privacy, or governance controls in enterprise settingsExperience in financial services, wealth management, consulting, or other regulated environmentsCore Competencies:Analytical rigor and attention to detailStructured testing and defect analysisCuriosity about model behavior and failure modesCross-functional collaborationClear communication of quality risks and trade-offsFocus on reliability, user trust, and measurable outcomes Apolis