Ai QA Architect

AI QE Architect

Duration: Fulltime

Location Onsite - West Palm Beach , Florida

Job description:

We are seeking an experienced AI Quality Engineering (QE) Architect & Governance Leader to drive end-to-end quality, automation, and governance for AI, ML, and GenAI solutions across mission-critical and highly regulated environments. This role combines deep expertise in automation, AI/ML technologies, and agentic workflows with strong leadership in governance, risk, and compliance-especially for Energy/Utilities/Nuclear domains.

You will architect future-state AI QE frameworks, establish testing standards for LLMs and agents, automate validation pipelines, enforce model governance, and lead enterprise-wide assurance initiatives.

Core Experience & Qualifications

• 10 14 years as a Technology Architect, SDET, or QE Leader with strong coding skills in Python, TypeScript, or Java.
• Proven experience designing automation frameworks or developer tools for large engineering organizations.
• Hands-on expertise with Large Language Models (LLMs), prompt engineering, and safety evaluation techniques.
• Exposure to Agentic AI systems and orchestration tools such as LangGraph, AutoGen, CrewAI, or similar agent frameworks.
• Experience implementing Model Context Protocol (MCP) for real-time automation, autonomous workflows, or CI/CD integrations.
• Experience working in highly regulated industries-Energy, Utilities, Nuclear, Healthcare, BFSI, or similar.

AI / ML Technologies

• Practical experience with frameworks and ecosystems:

LangChain, Hugging Face, GPT models, vector databases

• Working knowledge of ML/DL libraries:

Scikit-learn, TensorFlow, Keras, PyTorch, HuggingFace Transformers, OpenCV, NLTK and BART

• Understanding of RAG architectures, embeddings, and semantic search (bonus).

GenAI & AI Agent Development

• Expertise in designing, developing, validating, and deploying Generative AI solutions.
• Experience building AI agents, multi-agent workflows, or autonomous decision systems.
• Ability to define governance for:

oLLM drift detection

oPrompt quality standards

oAgent monitoring & observability

oData lineage & model versioning

Automation & Quality Engineering

• Strong experience building test automation frameworks using Python, PyTest, Selenium, Playwright, and Requests.
• Ability to create automated tests covering:

oFunctional

oAPI

oIntegration

oPerformance

oSecurity

oAI/ML validation (LLM testing, model accuracy, hallucination detection)

• Proficiency in API testing and validation of RESTful services.
• Plus: Experience with performance/load testing tools K6 or JMeter.

AI Governance & Compliance

• Establish AI/ML quality standards, testing guidelines, and risk controls.
• Define governance around:

oData security & privacy

oModel evaluation KPIs (accuracy, bias, toxicity, hallucination rates)

oRegulatory alignment for Energy/Utility operations

oContinuous monitoring & drift alerts

• Experience working with IRB, compliance, or audit teams.

Cloud, DevOps & CI/CD

• Knowledge of deploying AI and automation solutions on AWS.
• Experience implementing CI/CD pipelines for ML and LLM models:

oModel versioning & lifecycle

oRetraining workflows

oAutomated evaluation gates

oInfrastructure-as-code (IAC) familiarity

• Experience implementing observability frameworks for AI/ML systems.

SDLC & Collaboration

• Strong understanding of end-to-end SDLC and QE methodologies.
• Work closely with developers, product managers, data scientists, and business stakeholders.
• Ability to provide clear communication around test strategy, risks, coverage, and governance readiness.

Skilled in defect triage, risk-based testing, and quality strategy leadership