QA Engineer

DescriptionKforce has a client that is seeking a QA Engineer in New York, NY.Responsibilities:* Design, develop, and maintain automated test suites for AI platforms and services* Integrate testing capabilities into CI/CD pipelines to support rapid and reliable releases* Build automated regression, functional, integration, and end-to-end testing frameworks* Collaborate with platform engineering teams to improve testing coverage and release confidence* Monitor test execution results and proactively identify quality risks* Develop and maintain evaluation harnesses used to assess AI model and agent performance* Create and curate golden datasets that represent expected business outcomes and user interactions* Establish deterministic validation methods to verify AI responses against predefined expectations* Design scoring methodologies, evaluation metrics, and performance rubrics aligned to business objectives* Build adversarial and edge-case testing suites to assess model robustness, safety, and resilience* Implement continuous evaluation capabilities that extend beyond pre-production testing into live environments* Monitor AI outputs against established quality thresholds and service-level expectations* Develop dashboards and reporting mechanisms that provide visibility into model quality and drift* Identify performance degradation, anomalous behavior, and emerging risks* Support root-cause analysis and remediation efforts when quality issues are identified* Validate sandbox environments, platform guardrails, and operational control frameworks* Verify that platform controls function as intended across development, testing, and production environments* Support testing of AI safety mechanisms, access controls, content filters, and governance policies* Ensure evaluation standards remain aligned with regulatory, compliance, and organizational requirementsRequirements* 1-3 years of experience in Software Quality Assurance, Test Automation, Quality Engineering, Software Development, or a related technical discipline* Experience building automated tests using Python, Java, JavaScript, or similar programming languages* Familiarity with CI/CD pipelines and modern software delivery practices* Understanding of software testing methodologies, including integration, regression, and performance testing* Strong analytical and troubleshooting skills* Excellent written and verbal communication skillsPreferred Qualifications:* Experience building evaluation frameworks or automated validation tools* Experience with cloud platforms such as Azure, AWS, or Google Cloud* Experience with testing frameworks such as PyTest, Playwright, Selenium, JUnit, or equivalent tools* Exposure to Generative AI, Large Language Models (LLMs), AI agents, or machine learning systems* Exposure to observability and monitoring platforms* Familiarity with prompt engineering, retrieval-augmented generation (RAG), or agentic workflows* Understanding of AI safety, governance, model evaluation, and risk management conceptsTechnical Skills:* Testing & Automation* Python* PyTest* API Testing* Test Automation Frameworks* CI/CD Pipelines* Git/GitHubAI & Data* Generative AI* LLM Evaluation* Prompt Testing* Golden Dataset Creation* Evaluation Metrics* Continuous MonitoringCloud & Platforms:* Azure* Azure DevOps* GitHub Actions* Container Technologies* Monitoring & Observability ToolsJob TypeContractCompensation32 - $38