QA / Automation Engineer Agentic AI
Advanced Tech Placement
Job Description
Job Description
QA Automation Engineer – Agentic AI TestingPosition Overview
We are seeking a QA Automation Engineer to help build and validate a next-generation agentic AI application within a growing AI-driven HCM software ecosystem.
This is not a traditional QA automation role . The application uses large language models (LLMs) and AI agents to dynamically determine application behavior and workflows, creating testing challenges that cannot be solved through conventional scripted automation alone.
The team is building a new Python-based testing and evaluation framework from the ground up specifically for agentic AI systems . This includes LLM-as-judge evaluations, agent state checks, behavioral validation, and testing approaches designed for non-deterministic AI outputs.
Python and hands-on AI/LLM testing experience are the primary requirements for this role. Playwright is used for front-end automation but is a secondary/supporting skill rather than the core focus.
Key Responsibilities- Design, build, and maintain automated testing and evaluation solutions for agentic and LLM-driven applications .
- Help build a new Python-based AI testing and evaluation framework from the ground up , including defining patterns, standards, and approaches as the platform evolves.
- Develop LLM-as-judge evaluators and other automated methods for assessing AI-generated outcomes.
- Validate agent states, conversational flows, tool interactions, outputs, and expected behaviors across multi-step agentic workflows.
- Develop testing strategies for non-deterministic LLM behavior where conventional scripted pass/fail assertions are insufficient.
- Evaluate AI responses for factors such as accuracy, relevance, consistency, groundedness, and appropriate behavior .
- Test agent behavior across expected paths, edge cases, failure scenarios, and variable outputs.
- Use Playwright for front-end/UI automation where appropriate while maintaining a separate Python-based evaluation layer for AI behavior.
- Partner closely with software engineers and AI developers to identify risks, troubleshoot failures, and improve product quality.
- Help establish testing standards, evaluation methodologies, and automation practices for an evolving agentic AI platform.
- Take shared ownership of quality within a collaborative engineering model rather than operating within a traditional siloed QA organization.
- Strong hands-on Python programming and automation experience.
- Experience testing AI/LLM-based, conversational AI, or agentic systems .
- Experience with or strong knowledge of LLM evaluation techniques , including LLM-as-judge, state/behavior validation, or similar approaches.
- Experience validating non-deterministic AI behavior where expected outcomes cannot always be represented through conventional scripted assertions.
- Ability to design and build testing or evaluation frameworks , rather than solely executing within an established automation framework.
- Strong understanding of software testing, automation principles, debugging, and root-cause analysis.
- Ability to design test scenarios for complex, multi-step application and agent workflows.
- Strong problem-solving skills and ability to operate effectively in an early-stage, greenfield environment where standards and processes are still being defined.
- Comfortable working in a distributed quality model , with QA responsibility shared closely across engineering rather than owned solely by a formal QA function.
- Experience with Playwright for front-end automation.
- Experience with PyTest, Behave, or similar Python testing frameworks .
- Experience testing AI agents, tool/function calling, multi-turn conversations, or complex agentic workflows .
- Experience evaluating LLM outputs for groundedness, hallucination, relevance, correctness, consistency, or completeness .
- Experience testing guardrails, failure scenarios, and AI validation controls .
- Experience validating agent tool usage and downstream system interactions.
- API and database testing experience.
- Experience integrating automated testing and evaluation into CI/CD pipelines .
- Experience within HCM, HR technology, payroll, or related enterprise software .
This is an early-stage, greenfield AI initiative where engineering conventions, testing standards, and evaluation frameworks are actively being developed.
The team operates using an AI-SDLC model , with small engineering pods working closely with AI agents rather than relying on traditional, highly structured development and QA workflows. Quality ownership is collaborative and distributed across the team.
The broader product is more than a chatbot. It is an expanding agentic HCM software ecosystem , with a conversational interface serving as one entry point into a growing set of AI-driven HR and workforce capabilities.
Ideal CandidateThe strongest candidate will be a Python-focused QA Automation Engineer with genuine hands-on AI/LLM testing experience who is excited about helping define how agentic applications should be tested.
You should understand that testing an AI agent requires more than confirming whether a predefined workflow completes successfully. You will help determine whether an agent behaved appropriately, reached the correct state, used the right tools, produced an acceptable result, and performed reliably across inherently variable interactions .
This role is particularly well suited for someone who enjoys building frameworks rather than simply using them , solving testing problems that do not yet have established answers, and working closely with engineers to shape how agentic AI applications are evaluated and validated at scale.
$127.5k - $204k
...TitleNetwork Security Detection and Automation EngineerAbout your role: You... ..., telemetry and detection engineering combined with the ability to... ...and develop data-driven agentic responses Lead network-based... ...compliance, and governance outcomes. AI-Enabled Security...SuggestedFull timeTemporary workH1bWork at officeMonday to Friday- ...Job Description Job Description Job Title: QA Automation Engineer-Data/Backend Testing Location: Whippany, NJ (Hybrid) Duration: Contract Job Description ~ We are looking for a hands-on QA Automation Engineer with strong experience in backend and data testing...SuggestedContract work
$120k - $217.5k
Senior AI Security Automation EngineerWho we are looking forState Street's Cyber Data & Analytics (... ...seeking a Senior AI Security Automation Engineer to help shape the next generation of... ...hands-on delivery of scalable, reliable agentic AI platforms for security workflows·...SuggestedFull timeTemporary workWork at officeLocal areaFlexible hoursShift work2 days per week- ...Job Description Job Description Role: QA Automation Engineer Location: Whippany, NJ (Hybrid) Duration: Contract Key Requirements: Karate & API Testing experience must have. Selenium with Cucumber/BDD and UI Automation. Microservices Testing and Postman...SuggestedContract work
$134.5k - $265.1k
...Summary The Lead Managed Services Engineer II leads the development of AI-powered security engineering solutions and endpoint management automation for a large banking client. The role... ..., Go, large language models, and agentic frameworks to improve operational efficiency...SuggestedWork at officeLocal areaVisa sponsorshipNight shift1 day per week$55 per hour
...Role: Infrastructure Automation Engineer Location: Livingston, NJ (hybrid) Duration: Long Term Client needs a hands-on Infrastructure... ...as code, automation, and practical use of approved AI tools. The engineer will assess the current environment, recommend...Contract workRemote work- ...Job Description Job Description Job Title: QA/QE Automation Engineer/Manager Location: Berkeley Heights, NJ 5 days a week onsite Pay Rate: $65 W2 Skills: Playwright, Scripting experience with CI/CD pipeline Skill Years Of experience Self Rating QA Automation...
- ...seeking an experienced Software Development Engineer in Test (SDET) to join a highly... ...hands-on engineering role focused on test automation, quality engineering, and software reliability... ...JavaScript, and related tools. Leverage modern AI-assisted development tools to accelerate...
$134k - $179k
CoreWeave is The Essential Cloud for AI. Built for pioneers by pioneers, CoreWeave delivers... ...the RoleAs a Senior Security Operations Engineer I, you will independently lead the... ...contribute to the development of detections and automation, and help improve the processes and...Permanent employmentFull timeTemporary workCasual workWork at officeFlexible hoursNight shiftWeekend work$65.9k - $152.61k
...longevity bonuses Responsibilities The QA Engineering Manager leads the design, implementation... ...continuous improvement ofenterprise test automation strategies, frameworks, and quality engineering... ...Unauthorized recording, transcription, AI note-taking, or AI interview assistance...Work at officeRemote workWeekend work- ...Job Description Job Description We are looking for a hands-on AI & NetSuite Automation Specialist to improve and automate operational processes across Sales, Purchasing, Warehouse, Finance, and Data. You will help reduce manual work by building reliable workflows...
- ...digital core and unleashing the power of AI to create value at speed for... ...and Enterprise, Song, Supply Chain and Engineering, and Talent, with advanced capabilities... ...us at accenture.com.You are:The Robotics Automation Engineer understands the broader Physical...Work experience placementLive inWork at officeLocal area
$110k - $186k
...confidence.What does a successful Senior SDET Engineer do at Clover?This is an experienced... ...designing, building, and maintaining AI-based test automation frameworks and integration tests for... ..., deploying, and maintaining QA processes, tools, and supporting technologiesTranslate...Full timeTemporary workH1bWorldwide- ...Key Responsibilities Design, develop, and maintain automation solutions using Power Automate Desktop, Power Platform, RPA technologies, and AI-enabled capabilities. Build and support end-to-end workflow automations that improve business processes and reduce manual...
- ...Our client is looking to add an Automation Engineer I with hands-on experience developing PLC and SCADA applications for industrial process equipment to their team. This role involves programming control systems from the ground up, troubleshooting hardware and instrumentation...Night shift
$125k - $150k
Data, BI & Automation Developer North Jersey $125k - $150k A healthcare company is seeking... ...analytics tools, and develop practical AI and automation solutions. Working... ...business intelligence development, data engineering, analytics development, or a related technical...$145k - $185k
...as a subject matter expert for the test engineering team. You will lead, mentor, and grow a... ...of LabVIEW and Python, with expertise in automating complex, multi-day cryogenic and electromagnetic... ...1 We may use artificial intelligence (AI) tools to support parts of the hiring...Night shiftWeekend work- ...intersection of Application Security, DevOps Engineering, and Mainframe Modernization, requiring... ...to promotion between environments. " Automate vulnerability remediation workflows: auto... ...handling, and rollback procedures. AI-Augmented Findings Orchestration & Remediation...Contract workImmediate start
- ...Job Description Job Description Title- sr. automation engineer Location: laurel, MD contract: 6 month contract to hire days... ...robust, scalable automation frameworks and solutions including AI capabilities where applicable. Serve as a subject matter...Contract work
$165k - $242k
CoreWeave is The Essential Cloud for AI. Built for pioneers by pioneers, CoreWeave delivers... ...You’ll Do:CoreWeave’s Fleet Provisioning Automation (FPA) team is responsible for the... ...automation into a cohesive, high-reliability engine of fleet management. This group sits at...Permanent employmentFull timeTemporary workCasual workWork at officeFlexible hours- ...a group of passionate and forward-thinking experts. We're one of the largest engineering and system integration firms in the United States providing value for our clients through IT automation and control solutions for more than 30 years to the Life Sciences, Mission Critical...Local areaFlexible hours
$22 per hour
...Building Automation Specialist Internship Here at Siemens, we take pride in enabling sustainable progress through technology. We do... ...installations and system commissioning. ~ Collaborate with our engineering and project management teams to resolve any discrepancies in...Hourly payPermanent employmentInternshipH1bWork at officeLocal areaImmediate startRelocation- ...Support user-acceptance testing and performance validation. Collaborate closely with cross-functional engineering teams. Mentor junior testers and contribute to QA process development. Mandatory Skills ~3–5 years of hands-on experience in QA/testing mechatronic...
- ...thoroughly before clicking on the apply button at the bottom of this description. Possible 3 Month CTH Role name: Controls & Automation Engineer Work Location: Clarksville, TN (Onsite) Start date: Immediate availability. Background check MANDATORY Position...Immediate start
- ...Are you a hands-on Software QA Engineer who enjoys digging into complex technical issues and ensuring software performs reliably across... ...cases, test plans, and validation procedures. Contribute to automated testing efforts and ongoing improvements to QA processes. Track...
$99k - $143k
Network Automation Engineer American International Group, Inc. (AIG) is a leading global insurance organization. AIG member companies provide a wide range of property casualty insurance in approximately 70 countries and jurisdictions. These diverse offerings include products...$100k - $199k
...healthcare customers to deliver our broad portfolio of analytical solutions, and OneSource services.Job TitleSenior/Manager, Automation Engineering (cGMP) - Project FarmaLocation(s)Boulder, CO, Boulder, CO, Cambridge, MA, Columbus, OH, Concord, NC, Customer Site - AL,...Full timeWork visa- Job Title: Nuclear Engineer (Naval Reactors Engineer) Category / Component: Officer • Active Overview Design, regulate, and oversee the Navy’s nuclear propulsion program, including reactor design, fleet operations, and eventual defueling and decommissioning of nuclear...Contract workApprenticeshipWork at office
$85k - $135k
...be eligible to work in the United States without visa sponsorship now or in the future***Tutor Perini Corporation is seeking a QA/QC Engineer to join our Newark AirTrain project site at Newark International Airport. About Tutor Perini Corporation Extraordinary Projects...Full timeFor subcontractorWork at officeVisa sponsorship- ...and communities. Visit us at . The Global AI Infrastructure team enables resilient,... ...stack. We build reusable operational tools, automation workflows, and platform capabilities... ...including capacity management, reliability engineering, change management, and performance...Work experience placementLive inWork at officeLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to QA / Automation Engineer Agentic AI. Be the first to apply!







