Senior QA Engineer - API & AI/LLM Testing

Senior QA Engineer - API & AI/LLM Testing
Mason, OH or Woodland Hills, CA - 5 days

Candidates should have experience in testing AI projects. We can try for candidates have below experience:
• LM Evaluation & Benchmarking: Design and execute "LLM-as-a-Judge" frameworks to automatically score model outputs on continuous scales for accuracy, relevance, and tone.
• Hallucination & Bias Testing: Develop testing workflows to rigorously stress-test models against data hallucinations, toxic language, and gender/racial biases.
• Test Dataset Curation: Build and maintain high-quality, domain-specific golden datasets (input/output pairs) representing standard user queries and complex edge cases.

Overview We are looking for a Senior QA Engineer with strong experience in API testing, automation, and data validation, along with exposure to AI/LLM-based applications. The role involves working on API-driven healthcare systems, including claims, benefits, and data extraction workflows, as well as chatbot and AI-powered solutions.

Key Responsibilities:
Design and execute test strategies for API-driven applications
Perform end-to-end testing across APIs, databases, and UI layers
Validate data extraction and transformation logic, including AI/LLM outputs
Develop and maintain automation frameworks for API and UI testing
Perform data validation using SQL and NoSQL (DBeaver, MongoDB, or similar tools)
Test chatbot and AI/LLM workflows Integrate automated tests into CI/CD pipelines
Collaborate with cross-functional teams to ensure quality Identify, debug, and track defects through resolution

Required Skills:
Strong experience in API testing (Postman, REST APIs)
Hands-on automation experience (RestAssured, Selenium, Playwright, Cypress, or similar)
Solid SQL and NoSQL skills for data validation
Experience with CI/CD tools (Jenkins, Git, etc.)
Understanding of microservices and integration testing
Experience in healthcare domain (claims, benefits, eligibility)
Exposure to AI/LLM-based applications (chatbots, data extraction, NLP)
Familiarity with cloud platforms (AWS/Azure)