GenAI/LLM Test Engineer

We are looking for a GenAI/LLM Test Engineer with strong experience in AI/ML product testing, Python, AI evaluation frameworks, SQL, and test automation . The role will focus on testing LLM-based applications, RAG systems, AI agents, prompts, model outputs, and data pipelines.

The experience expected from applicants, as well as additional skills and qualifications needed for this job are listed below.

Required Skills:

• LLM/GenAI Product Testing
• Python & SQL
• AI Evaluation Frameworks – Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals, TruLens
• Test Automation, Regression & API Testing
• Performance & Adversarial Testing
• Generative AI, LLMs & RAG

Nice to Have:

• Prompt Engineering / Prompt Testing
• CI/CD Automation
• Azure & Snowflake
• Cloud Data Platform experience

Key Responsibilities:

• Design and execute test plans for GenAI/LLM features, prompts, APIs, and applications
• Build automated functional, regression, integration, and API test suites
• Evaluate model outputs for accuracy, relevance, hallucination, bias, toxicity, and robustness
• Develop evaluation frameworks, golden datasets, scoring rubrics, and regression test suites
• Perform adversarial/red-team, performance, scalability, and reliability testing
• Validate AI data pipelines for data quality, schema, and drift
• Integrate automated and AI evaluation testing into CI/CD pipelines
• Collaborate with Data Scientists, ML Engineers, and DevOps teams to define quality metrics and acceptance criteria.

Environment: Primarily Microsoft Azure and Snowflake , with GenAI initiatives leveraging OpenAI and Claude models.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains . With a strong emphasis on innovation, scalability, and client success , BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions.

BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence , enabling professionals to contribute to high-impact global initiatives while advancing their careers. xhyhwjd Through its focus on delivering specialized talent and technology expertise, BrickRed Systems supports organizations in building scalable teams and executing critical business initiatives across diverse industries and functional areas.