AI Quality Engineer

Position Title: AI Quality Engineer
Location: Vienna, VA / Remote
Clearance Requirements: None
Position Status: Contract
Pay Rate: $53 - $60 per hour

Position Description:
We are seeking an experienced AI Quality Engineer to lead the validation, certification, and production readiness of enterprise Generative AI and AI-powered automation solutions. This is a highly technical engineering role focused on ensuring AI systems are accurate, reliable, secure, explainable, compliant, and ready for enterprise production deployment.

This is not a traditional QA or manual testing position. The ideal candidate will serve as an independent quality authority responsible for evaluating Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) applications, AI agents, orchestration frameworks, developer platforms, and AI-enabled SDLC solutions.

Working alongside AI Engineers, Platform Engineers, Architects, Security, Risk, DevOps, and Product teams, you will develop repeatable validation frameworks, AI evaluation methodologies, production readiness standards, and governance processes that enable the successful deployment of enterprise AI solutions.

This position is ideal for engineers passionate about AI Quality Engineering, Responsible AI, AI Governance, Platform Engineering, DevEx, Azure AI, and enterprise-scale automation.

Key Responsibilities:

• Key Responsibilities:
• Develop and implement AI validation, certification, and production readiness standards for enterprise AI solutions.
• Design evaluation frameworks to measure:
  • AI accuracy
  • Response relevance
  • Groundedness
  • Completeness
  • Hallucination detection
  • Retrieval effectiveness
  • Recommendation quality
  • User satisfaction
• Build and maintain AI validation datasets, benchmark scenarios, regression suites, and golden datasets using tools such as LangSmith and Azure AI Foundry.
• Validate RAG (Retrieval-Augmented Generation) solutions utilizing Azure AI Search, LangChain, LangGraph, Azure AI Foundry, and enterprise knowledge repositories.
• Review AI solution architectures deployed across Azure cloud services including:
  • Azure Container Apps
  • Azure Functions
  • Azure Databricks
  • Azure SQL
  • Cosmos DB
• Evaluate AI agent workflows, orchestration pipelines, prompt execution, tool integrations, guardrails, human-in-the-loop processes, and MCP integrations.
• Assess AI security, governance, auditability, identity management, and compliance controls including Entra ID, RBAC, Managed Identities, Key Vault, and data protection requirements.
• Develop production readiness checklists covering observability, monitoring, resiliency, logging, supportability, recoverability, and operational excellence.
• Analyze AI telemetry, LangSmith traces, execution logs, and evaluation metrics to identify quality issues and optimization opportunities.
• Partner with engineering teams to resolve AI quality, security, and performance concerns before production deployment.
• Produce AI certification reports, quality scorecards, dashboards, and executive summaries for governance reviews.
• Establish independent quality gates and certification criteria for enterprise AI deployments.
• Lead validation and production readiness reviews for Internal Developer Portal (IDP), Developer Experience (DevEx), self-service engineering workflows, and platform automation initiatives.
• Drive continuous improvement of AI testing strategies, evaluation methodologies, and quality engineering practices.

Required Skills/Education

• Bachelor's degree in Computer Science, Software Engineering, Artificial Intelligence, Data Science, Information Systems, or a related technical discipline.
• 5+ years of experience in Software Quality Engineering, Test Architecture, Software Development, Platform Engineering, AI Engineering, Machine Learning Engineering, or related technical roles.
• 2+ years of hands-on experience with Generative AI, Large Language Models (LLMs), AI Agents, or Retrieval-Augmented Generation (RAG) solutions.
• Strong understanding of AI evaluation techniques including:
  • Hallucination detection
  • Groundedness validation
  • Accuracy testing
  • AI quality metrics
  • Model evaluation
• Experience with one or more of the following technologies:
  • Azure AI Foundry
  • Azure OpenAI
  • LangChain
  • LangGraph
  • LangSmith
  • AI Agent frameworks
  • RAG architectures
• Experience with Azure cloud technologies including:
  • Azure AI Search
  • Azure Container Apps
  • Azure Functions
  • Cosmos DB
  • Azure SQL
  • Azure Databricks
  • Azure Key Vault
• Experience supporting Platform Engineering, Internal Developer Portals (IDP), DevEx platforms, DevOps, or CI/CD environments.
• Strong knowledge of automated testing frameworks, regression testing, AI validation methodologies, and quality certification processes.
• Experience with APIs, microservices, distributed systems, and cloud-native architectures.
• Familiarity with DevSecOps, CI/CD pipelines, observability, monitoring, and enterprise SDLC practices.
• Excellent analytical, troubleshooting, documentation, and stakeholder communication skills.
• Ability to work independently while providing objective, data-driven quality assessments.

Preferred Qualifications

• Experience validating enterprise AI agents or multi-agent systems.
• Background in Platform Engineering, Developer Experience (DevEx), Site Reliability Engineering (SRE), or DevOps.
• Experience with AI observability and evaluation platforms such as LangSmith.
• Knowledge of Azure AI Search, vector databases, semantic search, embeddings, and enterprise knowledge retrieval.
• Experience implementing Responsible AI, AI Governance, AI Risk Management, or AI Compliance frameworks.
• Experience in highly regulated industries such as financial services, banking, healthcare, or insurance.
• Familiarity with Azure DevOps, GitHub, GitHub MCP, Azure DevOps MCP, and enterprise SDLC tooling.
• Experience with performance engineering, resiliency testing, chaos engineering, and production readiness reviews.
• Knowledge of Entra ID, RBAC, Managed Identities, Azure Key Vault, and identity governance.
• Experience creating executive dashboards, KPIs, quality scorecards, and AI performance reporting..

About Seneca Resources
At Seneca Resources, we are more than just a staffing and consulting firm, we are a trusted career partner. With offices across the U.S. and clients ranging from Fortune 500 companies to government organizations, we provide opportunities that help professionals grow their careers while making an impact.

When you work with Seneca, you're choosing a company that invests in your success, celebrates your achievements, and connects you to meaningful work with leading organizations nationwide. We take the time to understand your goals and match you with roles that align with your skills and career path. Our consultants and contractors enjoy competitive pay, comprehensive health, dental, and vision coverage, 401(k) retirement plans, and the support of a dedicated team who will advocate for you every step of the way.

Seneca Resources is proud to be an Equal Opportunity Employer, committed to fostering a diverse and inclusive workplace where all qualified individuals are encouraged to apply.