Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

QA / Automation Engineer Agentic AI

Advanced Tech Placement

Job Description

Job Description

QA Automation Engineer – Agentic AI TestingPosition Overview

We are seeking a QA Automation Engineer to help build and validate a next-generation agentic AI application within a growing AI-driven HCM software ecosystem.

This is not a traditional QA automation role . The application uses large language models (LLMs) and AI agents to dynamically determine application behavior and workflows, creating testing challenges that cannot be solved through conventional scripted automation alone.

The team is building a new Python-based testing and evaluation framework from the ground up specifically for agentic AI systems . This includes LLM-as-judge evaluations, agent state checks, behavioral validation, and testing approaches designed for non-deterministic AI outputs.

Python and hands-on AI/LLM testing experience are the primary requirements for this role. Playwright is used for front-end automation but is a secondary/supporting skill rather than the core focus.

Key Responsibilities
  • Design, build, and maintain automated testing and evaluation solutions for agentic and LLM-driven applications .
  • Help build a new Python-based AI testing and evaluation framework from the ground up , including defining patterns, standards, and approaches as the platform evolves.
  • Develop LLM-as-judge evaluators and other automated methods for assessing AI-generated outcomes.
  • Validate agent states, conversational flows, tool interactions, outputs, and expected behaviors across multi-step agentic workflows.
  • Develop testing strategies for non-deterministic LLM behavior where conventional scripted pass/fail assertions are insufficient.
  • Evaluate AI responses for factors such as accuracy, relevance, consistency, groundedness, and appropriate behavior .
  • Test agent behavior across expected paths, edge cases, failure scenarios, and variable outputs.
  • Use Playwright for front-end/UI automation where appropriate while maintaining a separate Python-based evaluation layer for AI behavior.
  • Partner closely with software engineers and AI developers to identify risks, troubleshoot failures, and improve product quality.
  • Help establish testing standards, evaluation methodologies, and automation practices for an evolving agentic AI platform.
  • Take shared ownership of quality within a collaborative engineering model rather than operating within a traditional siloed QA organization.
Required Qualifications
  • Strong hands-on Python programming and automation experience.
  • Experience testing AI/LLM-based, conversational AI, or agentic systems .
  • Experience with or strong knowledge of LLM evaluation techniques , including LLM-as-judge, state/behavior validation, or similar approaches.
  • Experience validating non-deterministic AI behavior where expected outcomes cannot always be represented through conventional scripted assertions.
  • Ability to design and build testing or evaluation frameworks , rather than solely executing within an established automation framework.
  • Strong understanding of software testing, automation principles, debugging, and root-cause analysis.
  • Ability to design test scenarios for complex, multi-step application and agent workflows.
  • Strong problem-solving skills and ability to operate effectively in an early-stage, greenfield environment where standards and processes are still being defined.
  • Comfortable working in a distributed quality model , with QA responsibility shared closely across engineering rather than owned solely by a formal QA function.
Preferred Qualifications
  • Experience with Playwright for front-end automation.
  • Experience with PyTest, Behave, or similar Python testing frameworks .
  • Experience testing AI agents, tool/function calling, multi-turn conversations, or complex agentic workflows .
  • Experience evaluating LLM outputs for groundedness, hallucination, relevance, correctness, consistency, or completeness .
  • Experience testing guardrails, failure scenarios, and AI validation controls .
  • Experience validating agent tool usage and downstream system interactions.
  • API and database testing experience.
  • Experience integrating automated testing and evaluation into CI/CD pipelines .
  • Experience within HCM, HR technology, payroll, or related enterprise software .
Team & Environment

This is an early-stage, greenfield AI initiative where engineering conventions, testing standards, and evaluation frameworks are actively being developed.

The team operates using an AI-SDLC model , with small engineering pods working closely with AI agents rather than relying on traditional, highly structured development and QA workflows. Quality ownership is collaborative and distributed across the team.

The broader product is more than a chatbot. It is an expanding agentic HCM software ecosystem , with a conversational interface serving as one entry point into a growing set of AI-driven HR and workforce capabilities.

Ideal Candidate

The strongest candidate will be a Python-focused QA Automation Engineer with genuine hands-on AI/LLM testing experience who is excited about helping define how agentic applications should be tested.

You should understand that testing an AI agent requires more than confirming whether a predefined workflow completes successfully. You will help determine whether an agent behaved appropriately, reached the correct state, used the right tools, produced an acceptable result, and performed reliably across inherently variable interactions .

This role is particularly well suited for someone who enjoys building frameworks rather than simply using them , solving testing problems that do not yet have established answers, and working closely with engineers to shape how agentic AI applications are evaluated and validated at scale.

Vacancy posted 26 days ago
Similar jobs that could be interesting for youBased on the QA / Automation Engineer Agentic AI in Roseland, NJ vacancy
  • $127.5k - $204k

     ...TitleNetwork Security Detection and Automation EngineerAbout your role: You...  ..., telemetry and detection engineering combined with the ability to...  ...and develop data-driven agentic responses Lead network-based...  ...compliance, and governance outcomes. AI-Enabled Security... 
    Suggested
    Full time
    Temporary work
    H1b
    Work at office
    Monday to Friday

    Fiserv

    Berkeley Heights, NJ
    1 day ago
  •  ...Job Description Job Description Job Title: QA Automation Engineer-Data/Backend Testing Location: Whippany, NJ (Hybrid) Duration: Contract Job Description ~ We are looking for a hands-on QA Automation Engineer with strong experience in backend and data testing... 
    Suggested
    Contract work

    Han IT Staffing

    Whippany, NJ
    6 days ago
  • $120k - $217.5k

    Senior AI Security Automation EngineerWho we are looking forState Street's Cyber Data & Analytics (...  ...seeking a Senior AI Security Automation Engineer to help shape the next generation of...  ...hands-on delivery of scalable, reliable agentic AI platforms for security workflows·... 
    Suggested
    Full time
    Temporary work
    Work at office
    Local area
    Flexible hours
    Shift work
    2 days per week

    State Street Bank

    Clifton, NJ
    3 days ago
  • $134.5k - $265.1k

     ...Summary The Lead Managed Services Engineer II leads the development of AI-powered security engineering solutions and endpoint management automation for a large banking client. The role...  ..., Go, large language models, and agentic frameworks to improve operational efficiency... 
    Suggested
    Work at office
    Local area
    Visa sponsorship
    Night shift
    1 day per week

    Deloitte

    Morristown, NJ
    15 hours ago
  • $55 per hour

     ...Role: Infrastructure Automation Engineer Location: Livingston, NJ (hybrid) Duration: Long Term Client needs a hands-on Infrastructure...  ...as code, automation, and practical use of approved AI tools. The engineer will assess the current environment, recommend... 
    Suggested
    Contract work
    Remote work

    VDart Inc

    Livingston, NJ
    11 days ago
  •  ...Job Description Job Description Job Title: QA/QE Automation Engineer/Manager Location: Berkeley Heights, NJ 5 days a week onsite Pay Rate: $65 W2 Skills: Playwright, Scripting experience with CI/CD pipeline Skill Years Of experience Self Rating QA Automation... 

    Diverse Lynx

    Berkeley Heights, NJ
    24 days ago
  • $134k - $179k

    CoreWeave is The Essential Cloud for AI. Built for pioneers by pioneers, CoreWeave delivers...  ...the RoleAs a Senior Security Operations Engineer I, you will independently lead the...  ...contribute to the development of detections and automation, and help improve the processes and... 
    Permanent employment
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours
    Night shift
    Weekend work

    CoreWeave

    Livingston, NJ
    2 days ago
  •  ...Job Description Job Description Role: QA Automation Engineer Location: Whippany, NJ (Hybrid) Duration: Contract Key Requirements: Karate & API Testing experience must have. Selenium with Cucumber/BDD and UI Automation. Microservices Testing and Postman... 
    Contract work

    Han IT Staffing

    Whippany, NJ
    26 days ago
  •  ...Job Description Job Description We are looking for a hands-on AI & NetSuite Automation Specialist to improve and automate operational processes across Sales, Purchasing, Warehouse, Finance, and Data. You will help reduce manual work by building reliable workflows... 

    Clifton NJ Logistics Company

    Clifton, NJ
    11 days ago
  • $65.9k - $152.61k

     ...longevity bonuses Responsibilities The QA Engineering Manager leads the design, implementation...  ...continuous improvement ofenterprise test automation strategies, frameworks, and quality engineering...  ...Unauthorized recording, transcription, AI note-taking, or AI interview assistance... 
    Work at office
    Remote work
    Weekend work

    Guard Insurance Group

    Parsippany, NJ
    5 days ago
  •  ...digital core and unleashing the power of AI to create value at speed for...  ...and Enterprise, Song, Supply Chain and Engineering, and Talent, with advanced capabilities...  ...us at accenture.com.You are:The Robotics Automation Engineer understands the broader Physical... 
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Morristown, NJ
    1 day ago
  • $110k - $186k

     ...confidence.What does a successful Senior SDET Engineer do at Clover?This is an experienced...  ...designing, building, and maintaining AI-based test automation frameworks and integration tests for...  ..., deploying, and maintaining QA processes, tools, and supporting technologiesTranslate... 
    Full time
    Temporary work
    H1b
    Worldwide

    Fiserv

    Berkeley Heights, NJ
    4 days ago
  •  ...Key Responsibilities Design, develop, and maintain automation solutions using Power Automate Desktop, Power Platform, RPA technologies, and AI-enabled capabilities. Build and support end-to-end workflow automations that improve business processes and reduce manual... 

    Robert Half

    Fairfield, Essex County, NJ
    16 days ago
  •  ...Our client is looking to add an Automation Engineer I with hands-on experience developing PLC and SCADA applications for industrial process equipment to their team. This role involves programming control systems from the ground up, troubleshooting hardware and instrumentation... 
    Night shift

    Germer International - Pharmaceutical Recruiting

    Wayne, NJ
    6 days ago
  • $125k - $150k

    Data, BI & Automation Developer North Jersey $125k - $150k A healthcare company is seeking...  ...analytics tools, and develop practical AI and automation solutions. Working...  ...business intelligence development, data engineering, analytics development, or a related technical... 

    Pro Talent Solutions

    Nutley, NJ
    3 days ago
  •  ...seeking an experienced Software Development Engineer in Test (SDET) to join a highly...  ...hands-on engineering role focused on test automation, quality engineering, and software reliability...  ...JavaScript, and related tools. Leverage modern AI-assisted development tools to accelerate... 

    Saicon Consultants Inc

    Roseland, NJ
    26 days ago
  • $165k - $242k

    CoreWeave is The Essential Cloud for AI. Built for pioneers by pioneers, CoreWeave delivers...  ...You’ll Do:CoreWeave’s Fleet Provisioning Automation (FPA) team is responsible for the...  ...automation into a cohesive, high-reliability engine of fleet management. This group sits at... 
    Permanent employment
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours

    CoreWeave

    Livingston, NJ
    3 days ago
  • $22 per hour

     ...Building Automation Specialist Internship Here at Siemens, we take pride in enabling sustainable progress through technology. We do...  ...installations and system commissioning. ~ Collaborate with our engineering and project management teams to resolve any discrepancies in... 
    Hourly pay
    Permanent employment
    Internship
    H1b
    Work at office
    Local area
    Immediate start
    Relocation

    Siemens

    Morristown, NJ
    5 days ago
  •  ...thoroughly before clicking on the apply button at the bottom of this description. Possible 3 Month CTH Role name: Controls & Automation Engineer Work Location: Clarksville, TN (Onsite) Start date: Immediate availability. Background check MANDATORY Position... 
    Immediate start

    Talent Software Services, Inc.

    Bloomfield, NJ
    3 days ago
  •  ...Are you a hands-on Software QA Engineer who enjoys digging into complex technical issues and ensuring software performs reliably across...  ...cases, test plans, and validation procedures. Contribute to automated testing efforts and ongoing improvements to QA processes. Track... 

    Kane Partners LLC

    Newark, NJ
    5 days ago
  • $99k - $143k

    Network Automation Engineer American International Group, Inc. (AIG) is a leading global insurance organization. AIG member companies provide a wide range of property casualty insurance in approximately 70 countries and jurisdictions. These diverse offerings include products... 

    AIG

    Passaic, NJ
    3 days ago
  • $75k

     ...Are you a hands-on Software QA Engineer who enjoys breaking things, troubleshooting issues, and ensuring software performs as expected?...  ...PTT, or MCPTT. Knowledge of 3GPP standards. Exposure to automated testing or scripting (Python/Java). Experience testing connected... 
    Full time

    Kane Partners LLC

    Newark, NJ
    4 hours ago
  • $100k - $199k

     ...healthcare customers to deliver our broad portfolio of analytical solutions, and OneSource services.Job TitleSenior/Manager, Automation Engineering (cGMP) - Project FarmaLocation(s)Boulder, CO, Boulder, CO, Cambridge, MA, Columbus, OH, Concord, NC, Customer Site - AL,... 
    Full time
    Work visa

    PerkinElmer

    Summit, NJ
    4 days ago
  • Job Title: Nuclear Engineer (Naval Reactors Engineer) Category / Component: Officer • Active Overview Design, regulate, and oversee the Navy’s nuclear propulsion program, including reactor design, fleet operations, and eventual defueling and decommissioning of nuclear... 
    Contract work
    Apprenticeship
    Work at office

    US Navy

    Roseland, NJ
    1 day ago
  •  ...and communities. Visit us at . The Global AI Infrastructure team enables resilient,...  ...stack. We build reusable operational tools, automation workflows, and platform capabilities...  ...including capacity management, reliability engineering, change management, and performance... 
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    Morristown, NJ
    1 day ago
  • $85k - $135k

     ...be eligible to work in the United States without visa sponsorship now or in the future***Tutor Perini Corporation is seeking a QA/QC Engineer to join our Newark AirTrain project site at Newark International Airport. About Tutor Perini Corporation Extraordinary Projects... 
    Full time
    For subcontractor
    Work at office
    Visa sponsorship

    Tutor Perini Corporation

    Newark, NJ
    3 days ago
  •  ...Job Description Job Description Enterprise AI Engineer (GenAI Process Engineering) 80% Remote | Enterprise Transformation | Industry...  ...Generation (RAG) Model Context Protocol (MCP) Autonomous/Agentic AI systems Experience building AI workflows using... 
    Remote work

    Contech Systems

    Newark, NJ
    6 days ago
  • $120k - $202.5k

     ...) team is seeking a Sr.Platform Operations Engineer to help shape the next generation of cybersecurity data, analytics, and AI-powered platforms. Partnering closely with...  ...develops advanced data platforms, intelligent automation solutions, and engineering capabilities... 
    Full time
    Temporary work
    Flexible hours

    State Street Bank

    Clifton, NJ
    5 days ago
  •  ...of manufacturing where your ideas, skills, and passion matter. Weiss-Aug, is a leading provider of advanced manufacturing and engineering solutions with advanced facilities across the U.S. and Mexico for over 50 years. Why Join Us? At Weiss-Aug we are driven by... 
    Temporary work
    Work at office
    Local area
    Worldwide
    Flexible hours

    Weiss-Aug Co., Inc.

    East Hanover, NJ
    5 days ago
  • $83k - $110k

     ...Operations Engineer, Fleet Reliability Plano, TX / Washington, DC / Livingston, NJ CoreWeave is The Essential Cloud for AI. Built for pioneers by pioneers, CoreWeave delivers a platform of technology, tools, and teams that enables innovators to build and scale AI with... 
    Permanent employment
    Full time
    Temporary work
    Casual work
    Work at office
    Remote work
    Flexible hours
    Shift work

    CoreWeave

    Livingston, NJ
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to QA / Automation Engineer Agentic AI. Be the first to apply!