Senior AI Quality Engineer

Role: AI QE engineering

Location: Chicago, IL or Irving, TX (Onsite)

Experience: 10+ Years

Engagement: Contract

Job Description:

We are looking for an experienced Sr. AI Quality & Reliability Engineer to drive the design, development, and execution of AI Quality Engineering initiatives supporting enterprise AI transformation. The candidate will be responsible for AI validation, AI-assisted testing, runtime quality controls, and scalable testing frameworks to ensure the reliability and quality of AI-powered applications.

Key Responsibilities:

AI Quality Engineering & Technical Execution

• Design and execute quality engineering activities for AI-powered applications, LLM-enabled workflows, intelligent automation solutions, agentic systems, and enterprise AI platforms.
• Develop AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and scalable testing workflows.
• Modernize traditional QA practices to support AI-enabled applications and healthcare workflows.
• Establish AI quality standards, validation approaches, release readiness processes, and governance-aligned testing practices.
• Manage AI validation activities, defect management, test execution, release support, and quality improvement initiatives.

AI Validation, Testing & Runtime Assurance

• Perform functional testing, prompt testing, workflow testing, regression testing, and runtime quality assurance.
• Collaborate with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, and Data teams.
• Support runtime monitoring, distributed tracing, observability, incident management, and production reliability.
• Develop AI evaluation frameworks, validation datasets, quality scoring approaches, and automated testing workflows.
• Support drift detection, human-in-the-loop validation, and AI monitoring initiatives.
• Drive AI-assisted testing, intelligent automation, and reusable test automation frameworks.
• Validate healthcare workflows, payer operations, and AI-enabled business processes.

Delivery Coordination & Operational Support

• Coordinate testing activities, sprint execution, issue tracking, risk identification, and release support.
• Evaluate testing readiness, implementation dependencies, and quality risks.
• Support tooling evaluations, automation initiatives, and testing modernization.
• Establish scalable testing processes and reusable quality engineering frameworks.

Collaboration, Mentorship & Continuous Improvement

• Mentor quality engineers and analysts and provide technical guidance to delivery teams.
• Communicate testing risks, implementation issues, and recommendations to technical and business stakeholders.
• Promote engineering best practices, continuous improvement, and responsible AI adoption.
• Evaluate emerging AI testing, observability, automation, and runtime assurance technologies.

Required Qualifications:

• 10+ years of overall IT experience.
• 5+ years of experience in Quality Engineering, Quality Assurance, Software Testing, or Enterprise Application Delivery.
• 2+ years of technical leadership experience in testing initiatives, automation, or enterprise delivery projects.
• Strong experience in Quality Engineering / QA across enterprise platforms, APIs, operational workflows, healthcare applications, or integrated business systems.
• Experience with test automation frameworks, SDLC, Agile methodologies, defect management, regression testing, release validation, and production support.
• Understanding of AI/LLM validation, prompt testing, AI-assisted testing, and runtime quality practices.
• Strong analytical, communication, collaboration, and leadership skills.

Preferred Skills:

• GenAI and LLM Testing.
• AI Model Evaluation and Validation.
• Prompt Testing and Engineering.
• Agentic AI Testing.
• AI Observability and Runtime Monitoring.
• AIOps / LLMOps.
• Healthcare / Payer Domain Experience.
• Test Automation and API Testing.