Principal QA Engineer

Principal QA Engineer (AI Systems & Platform) – Remote

Job Title: Principal QA Engineer Work Mode: Remote Shift: 7:30 PM IST – 4:30 AM IST Experience: 7+ Years

About the Role

We are looking for a highly skilled Principal QA Engineer to build and lead the QA function from scratch for cutting-edge AI systems.

This is a founding-level role where you will define quality standards, design testing frameworks, and ensure AI-generated insights are accurate, reliable, and production-ready. You will act as the final quality gate before outputs reach business leaders and clients.

Key Responsibilities

A. QA Foundation & Strategy

• Build QA processes from the ground up (unit, integration, end-to-end testing)
• Define quality benchmarks, coverage standards, and documentation practices
• Audit platform risks and prepare for client deployments

B. AI Systems & Evaluation

• Design evaluation frameworks for non-deterministic AI/LLM outputs
• Implement prompt regression testing and model drift detection
• Build automated testing for AI agent orchestration and pipelines

C. Platform & Integration Testing

• Test end-to-end data ingestion pipelines across multiple systems
• Validate multi-model routing, streaming responses, and system performance
• Ensure reliability of file ingestion pipelines and enterprise data workflows

D. Data & Knowledge Systems Validation

• Validate knowledge graphs, data accuracy, and retrieval quality
• Ensure reliability of AI-generated insights before client delivery

E. Monitoring & Reliability

• Track system performance, failures, and latency thresholds
• Ensure graceful handling of model failures and degraded scenarios

F. Team Building & Leadership

• Hire, mentor, and lead QA engineers
• Establish a strong quality-first engineering culture
• Collaborate with engineering and product teams in Agile environments

Required Skills

• 7+ years in QA Engineering (3+ years in leadership roles)
• Strong experience testing AI/LLM-based applications
• Expertise in Python and automated testing frameworks
• Hands-on experience with Playwright and Vitest
• Experience with API testing, async systems, and distributed architectures
• Strong understanding of CI/CD pipelines and test automation
• Excellent problem-solving and leadership skills

Tech Stack Exposure

• AI/LLM: Anthropic Claude, OpenAI GPT, multi-model routing
• Backend: Python (FastAPI), async systems
• Data: Memgraph, Neo4j, Qdrant, PostgreSQL, Redis
• Frontend: React, Next.js, TypeScript
• Cloud: Google Cloud Platform, Azure
• DevOps: GitHub Actions, Docker
• Testing: Playwright, Vitest

Nice to Have

• Experience with LLM evaluation frameworks (LangSmith, PromptFlow, etc.)
• Exposure to agent-based systems or orchestration frameworks
• Experience in regulated industries (finance, healthcare, etc.)
• Experience working with client-facing or deployment teams

Salary

As per industry standards