Quality Assurance Engineer

We've made a lot of progress since opening the doors in 1942, but one thing has never changed - our commitment to serve, heal, lead, educate, and innovate. We believe that every award earned, every record broken and every patient helped is because of the dedicated employees who fill our hallways. At Ochsner, whether you work with patients every day or support those who do, you are making a difference and that matters. Come make a difference at Ochsner Health and discover your future today! Owns the quality engineering function for an agentic conversational AI platform — serving as the guardian of the customer experience across every voice and chat interaction. This role spans two mandates: hands-on assurance that every release is effective and defect-free before it reaches end users, and building, scaling, and maintaining the evaluation infrastructure that makes quality continuous, measurable, and increasingly self-improving.The role plans and directs the development, application, and maintenance of quality standards for conversational AI applications and their supporting systems and owns the automated evaluation platform — synthetic and adversarial agent-to-agent conversation testing, DeepEval test suites, proprietary LLM-as-judge evaluators, and deterministic rule-based checks — as the system of record for shipping model, prompt, and agent changes with confidence. Because LLM systems are non-deterministic, this role designs test plans and evaluation rubrics that turn ambiguous notions of quality into clear, defensible metrics, and correlates those metrics with real-world performance and user experience.The role is weighted toward quality infrastructure and evaluation engineering (~70%), balanced with hands-on manual and exploratory QA (~30%), with an explicit mandate to advance the platform toward recursive self-improvement: a closed-loop quality flywheel in which evaluation findings automatically generate new test cases, refine judge rubrics, surface model and prompt improvement candidates, and re-validate fixes — so the evaluators, and the agents they evaluate, get better with every conversation.To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable qualified individuals with disabilities to perform the essential duties.This job description is a summary of the primary duties and responsibilities of the job and position. It is not intended to be a comprehensive or all-inclusive listing of duties and responsibilities. Contents are subject to change at the company’s discretion.Work ExperienceRequired - 3 years Software; quality assurance and testing methodologies; and defect-tracking tools. Requires conversational AI testing experience (intent accuracy, fallback handling, escalation logic)Preferred - Familiarity with AI/machine learning and natural language processing concepts, particularly for conversational AI or chatbot testing.Knowledge Skills and Abilities (KSAs)Understanding of the software development lifecycle and API/integration testing.Ability to design and execute detailed test cases across multiple scenarios.Strong attention to detail and quality standards.Ability to collaborate and communicate effectively with engineers and product teams to resolve defects.Documentation and reporting of test outcomes.Ability to balance speed of testing with thoroughness and accuracy while maintaining consistency and discipline in the testing process.Job DutiesDevelops and implements quality standards, tests plans, and methods for inspection, testing, and evaluation across the conversational AI platform and supporting systems.Defines functional and non-functional test strategies, including conversation flow accuracy, response quality, latency, and fallback and escalation behavior.Designs and executes test cases covering user scenarios, edge cases, and failure points, including regression, negative, usability, and integration testing.Builds and maintains automated test scripts and frameworks, and perform manual testing where appropriate.Identifies, documents, and tracks defects through resolution, and confirm fixes.Tests the product both as intended and as misused to anticipate incorrect usage and reduce operational and compliance risk.Maintains test documentation and reports on quality assurance activities, defect patterns, and release readiness.Performs other duties as assignedComplies with all policies and standardsThe above statements describe the general nature and level of work only. They are not an exhaustive list of all required responsibilities, duties, and skills. Other duties may be added, or this description amended at any time.Remains knowledgeable on current federal, state and local laws, accreditation standards or regulatory agency requirements that apply to the assigned area of responsibility and ensures compliance with all such laws, regulations and standards.This employer maintains and complies with its Compliance & Privacy Program and Standards of Conduct, including the immediate reporting of any known or suspected unethical or questionable behaviors or conduct; patient/employee safety, patient privacy, and/or other compliance-related concerns. The employer is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or disability status.Physical and Environmental DemandsThe physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.Sedentary Work- Exerting up to 10 pounds of force occasionally (occasionally means activity or conditions exist up to 1/3 of the work day), and/or, a negligible amount of force frequently (frequently means activity or condition exists from 1/3 to 2/3 of the work day) to lift, carry, push, pull, or otherwise move objects, including the human body. Sedentary work involves sitting most of the time but may involve walking or standing for brief periods of time. Jobs are sedentary if walking and standing are required only occasionally and all other sedentary criteria are met.Normal routine involves no exposure to blood, body fluid or tissue and as part of their employment, incumbents are not called upon to perform or assist in emergency care or first aid.The incumbent has no occupational risk for exposure to communicable diseases .Because the incumbent works within a healthcare setting, there may be occupational risk for exposure to hazardous medications or hazardous waste within the environment through receipt, transport, storage, preparation, dispensing, administration, cleaning and/or disposal of contaminated waste. The risk level of exposure may increase depending on the essential job duties of the role.Are you ready to make a difference? Apply Today!Ochsner Health does not consider an individual an applicant until they have formally applied to the open position on this careers website.Please refer to the job description to determine whether the position you are interested in is remote or on-site.Individuals who reside in and will work from the following areas are not eligible for remote work position: Colorado, California, Hawaii, Illinois, Maine, Maryland, Massachusetts, Minnesota, New Jersey, New York, Vermont, Washington, and Washington D.C.Ochsner Health endeavors to make our site accessible to all users. If you would like to contact us regarding the accessibility of our website, or if you need an accommodation to complete the application process, please contact our HR Employee Solution Center at View phone number on click.appcast.io (select option 1) or View email address on click.appcast.io. This contact information is for accommodation requests only and cannot be used to inquire about the status of applications.Ochsner is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to any legally protected class, including protected veterans and individuals with disabilities.SummaryLocation: New Orleans - New Orleans Region - LouisianaType: Full time