Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior AI Test Automation Engineer

Genuine Parts Company

Senior AI Test Automation Engineer The Senior AI Test Automation Engineer designs, develops, maintains, and executes automated testing and evaluation solutions for both traditional software and LLM-powered applications. Operating in a forward-deployed capacity, this role works directly with users and delivery teams to capture real-world usage patterns and feedback, translating them into a continuously growing evaluation framework that validates LLM behavior against the consistent flows users actually follow. This role partners with delivery teams, QA, development, product, AI engineering, and the QA Center of Excellence (CoE) to establish scalable automation and evaluation practices, increase test and eval coverage, and integrate quality controls throughout the software delivery lifecycle, from pre-deployment regression testing through production observability. You must be eligible to work in the US without Visa Sponsorship. Responsibilities Act as a forward-deployed quality engineer: engage directly with users and stakeholders to collect feedback, observe real usage patterns, and identify the consistent flows users follow through LLM-powered features. Translate user feedback and production traces into curated evaluation datasets in LangSmith, and continuously expand the eval framework as new feedback, edge cases, and failure modes are discovered. Design, build, and maintain offline evaluation suites (regression, benchmarking, and backtesting) that gate prompt, model, and LangGraph workflow changes before deployment. Develop and calibrate evaluators, heuristic/code-based checks, LLM-as-judge evaluators, and pairwise comparisons, and validate judge reliability against human review. Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith, ensuring trace coverage, quality, and useful metadata for debugging and analysis. Manage annotation queues and human-in-the-loop feedback workflows, routing interesting or problematic production runs to reviewers and feeding results back into datasets and evaluator calibration. Analyze agent trajectories and multi-step LangGraph executions (tool calls, state transitions, retrieval steps) to pinpoint failure points and distinguish nondeterministic LLM variance from genuine product defects. Integrate eval runs into CI/CD pipelines so that dataset versions, experiments, and quality thresholds provide automated feedback on every relevant change. Support the adoption of production auditing and monitoring capabilities—such as online evaluations on live traffic, quality drift detection, and alerting—to help teams detect issues in production (supportive to the role, not its core focus). Design, develop, and implement automated test scripts for UI, API, integration, and regression testing, including deterministic E2E coverage of LLM-powered application surfaces. Integrate automated tests into CI/CD pipelines to enable timely feedback and continuous quality validation. Collaborate with QA, development, product, and business teams to translate requirements, acceptance criteria, and expected agent behaviors into effective automated test and eval coverage. Analyze and triage automation test failures, differentiating framework or script issues from valid product defects, including the added dimension of expected LLM nondeterminism. Support test data management (including eval dataset versioning, splits, and provenance) and help identify or resolve test-environment stability issues. Report on test execution results, eval experiment outcomes, automation and eval coverage, quality trends, and risks. Participate in code reviews and contribute to automation and evaluation standards, reusable components, and best practices. Engage with the QA CoE to align automation and AI evaluation practices with enterprise standards while contributing domain-specific feedback, lessons learned, and continuous-improvement opportunities. Required Experience 5+ years of experience in test automation engineering, software quality assurance, or a related role. 1–2+ years of hands-on experience testing or evaluating LLM-powered applications, including building eval datasets, defining pass/fail criteria for nondeterministic outputs, and using LLM-as-judge or heuristic evaluators. Hands-on experience with LangSmith for tracing, evaluation, and observability of LLM applications, including creating datasets, running experiments, and configuring evaluators. Hands-on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling. Hands-on experience building and maintaining automated test suites using Playwright, preferably with TypeScript. Working proficiency in Python and/or TypeScript sufficient to author custom evaluators, tracing instrumentation, and test code. Experience automating UI and API testing, and experience with API testing tools or frameworks. Familiarity with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or equivalent, including integrating eval runs as pipeline gates. Working knowledge of source-code version control, including Git. Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies. Ability to analyze requirements and user feedback, identify test and eval scenarios, and create maintainable automated coverage. Strong problem-solving, troubleshooting, communication, and collaboration skills, including comfort engaging directly with end users to gather feedback in a forward-deployed capacity. Preferred / Nice to Have Ability to understand and document end-to-end business processes, user journeys, and operational workflows. Partner with business stakeholders and end users to translate process requirements into test scenarios, acceptance criteria, and LLM evaluation datasets. Identify process exceptions, edge cases, dependencies, and risks that may affect application or agent behavior. Validate that automated workflows and LLM-powered features produce outcomes aligned with defined business rules and user needs. Use production feedback and observed user behavior to continuously refine process coverage, test automation, and evaluation frameworks. Experience with online evaluations, production monitoring, and quality drift detection for LLM applications. Experience with agent trajectory evaluation, RAG evaluation (retrieval relevance, groundedness, hallucination detection), or guardrails validation. Familiarity with prompt engineering and prompt versioning workflows, and evaluating the impact of prompt or model changes. Understanding of statistical approaches to nondeterministic testing (multiple-run sampling, confidence thresholds, summary metrics across datasets). Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework. Experience with contract testing tools. Familiarity with SAFe and Agile Release Train (ART) practices. Experience with AI-assisted engineering practices (e.g., using AI coding agents to accelerate test and eval development). Experience with performance, accessibility, mobile, or security test automation. Experience with test management and defect-tracking tools, such as Azure DevOps. Not the right fit? Let us know you're interested in a future opportunity by joining our Talent Community on jobs.genpt.com or create an account to set up email alerts as new job postings become available that meet your interest! GPC conducts its business without regard to sex, race, creed, color, religion, marital status, national origin, citizenship status, age, pregnancy, sexual orientation, gender identity or expression, genetic information, disability, military status, status as a veteran, or any other protected characteristic. GPC's policy is to recruit, hire, train, promote, assign, transfer and terminate employees based on their own ability, achievement, experience and conduct and other legitimate business reasons. Genuine Parts Company

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior AI Test Automation Engineer in Birmingham, AL vacancy
  •  ...Curious and Learn.; Win as a Team Job Summary Quality at Moultrie isn't a checkbox - it's built into how we work. As a Senior Test Automation Engineer , you'll own the test infrastructure that keeps our connected camera platform reliable from the mobile app all the way... 
    Senior
    Local area
    Remote work

    EBSCO

    Birmingham, AL
    1 day ago
  •  ...of existing and future electrical and automation equipment at the plant.   RESPONSIBILITIES...  ...system procedures. Work with the engineering and production staff to refine process...  ...CAD system and knowledge of electrical testing meters and methods are preferred. A... 
    Suggested
    For contractors

    Sparrow Inc

    Birmingham, AL
    11 hours ago
  •  ...Job Title - AWS Infrastructure Automation Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham...  ...Position Overview We are seeking a Senior Cloud Automation Engineer to join a...  ...technologies, complex data challenges, AI-enabled automation, and cross-functional... 
    Suggested
    Full time
    Local area

    System One

    Birmingham, AL
    more than 2 months ago
  • $119k - $190.4k

     ...seeking a dynamic and technically proficient Senior Technical Business Developer to lead the expansion of our Grid Automation portfolio in the U.S. market. This role...  ...QualificationsBachelor’s or Master’s degree in Electrical Engineering, Power Systems, or related field.9+ years of... 
    Senior
    Hourly pay
    Full time
    Temporary work
    Remote work
    Home office
    Monday to Friday

    ABB

    Birmingham, AL
    1 day ago
  • $77.2k - $96.5k

     ...working independently in Agile/iterative delivery environments with SLA-driven support ~ Continuous learner who actively explores AI tools and stays current on emerging technologies. Excellent written, verbal, and presentation communication skills ~ Ability to explain... 
    Senior
    Full time
    Shift work

    World Wide Technology

    Birmingham, AL
    2 days ago
  •  ...Senior DevSecOps Engineer Permanent Full-Time Lafayette, LA | Knoxville, TN | Birmingham, AL |...  ...Repository, or JFrog . Build and automate software approval workflows , including...  ...onboarding of emerging ecosystems, including AI/ML frameworks . Build reporting... 
    Senior
    Permanent employment
    Full time

    System One

    Birmingham, AL
    23 hours ago
  •  ...is a fast-growing healthcare technology company building AI voice agents that answer every call for physician practices...  ...routine requests around the clock. They are looking for a Senior Automation Test Engineer who can plug in and build rapidly, owning the test strategy... 
    Senior
    Contract work

    Oxenham Group LLC

    Birmingham, AL
    24 days ago
  •  ...Senior Cloud Security Architect Location: Birmingham, AL, Columbia, SC, Knoxville, TN...  ...product owners, enterprise architecture, engineering, and security teams to protect sensitive...  ...experience evaluating security risks in AI, machine learning, and agentic AI applications... 
    Senior
    Full time

    System One Holdings, LLC

    Birmingham, AL
    3 days ago
  •  ...Senior Data Scientist (Level III) GC & US Citizen only Location...  ...environments. Feature Engineering & Data Preparation Lead...  ...audiences. Experimentation & Testing Design and analyze A/B...  ...standards. Promote ethical AI and responsible data practices... 
    Senior
    Contract work

    4P Consulting Inc

    Vestavia Hills, AL
    5 hours ago
  •  ...Job Description Job Description Job Title: Senior AI Coding Engineer Duration: Full Time / Permanent Position Location: Birmingham...  ...You'll also build out the supporting tooling — APIs, automation scripts, test harnesses — needed to keep these agents reliable and... 
    Senior
    Permanent employment
    Full time

    System One

    Birmingham, AL
    14 days ago
  • You build the systems that actually make AI work in enterprise environments, not demos...  .... The difference between an AI Engineer and what we are looking for is straightforward...  ...systems Lead client engineering engagements at senior level — facilitate architecture design... 
    Senior

    Accenture

    Birmingham, AL
    4 days ago
  • Senior ServiceNow Discovery Developer Seeking a Senior ServiceNow Discovery...  ...and Reconciliation Engine (IRE), CMDB relationships, and...  ...infrastructure visibility. Use Python for automation, troubleshooting, and tooling...  ...resolve discovery gaps. Use AI-assisted development tools... 
    Senior

    System One Holdings, LLC

    Birmingham, AL
    1 day ago
  • $100k

     ...about Evonik:://careers.evonik.com/en/about/meet-the-team/The automation engineer is responsible for maintaining all aspects of the site...  ...position receives technical guidance and assistance from more senior automation engineering positions.RESPONSIBILITIES:Troubleshoot... 
    Full time
    For contractors
    Local area
    Flexible hours

    Evonik Services

    Birmingham, AL
    3 days ago
  • $132.23k - $176.31k

     ...trusted network for the AI‑powered world,...  ...today. The Role The Senior Lead Marketing Data Platform...  ...partnership with engineering, analytics, Adobe, Salesforce...  ...enabling intelligent automation, and creating modern...  ..., observability, testing, security, and environment... 
    Senior
    Full time
    Temporary work

    Lumen

    Birmingham, AL
    1 day ago
  •  ...Job Description Job Description Job Title: Senior GenAI & Data Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL...  ...Work Mode: Hybrid Systemone is seeking a Senior Gen AI Software Engineer with strong hands-on experience in Generative... 
    Senior
    Permanent employment
    Full time

    System One

    Birmingham, AL
    14 days ago
  •  ...Kforce has a client that is seeking a Senior Full Stack Software Engineer (React, .NET, AWS) in Birmingham, AL...  ...groups. The team is focused on automating workflows, improving existing processes...  ...Today? you agree to receive calls, AI-generated calls, text messages or emails... 
    Senior
    Hourly pay
    Contract work

    Kforce Technology Staffing

    Birmingham, AL
    11 hours ago
  •  ...Sigao Software Engineer Position Sigao is a Birmingham, AL based AI and technology consultancy that builds software to...  ...apps with Angular front-ends, to automated AI solutions that enhance workflows...  ...of web APIs, DevOps, Test Driven Development and modern software... 
    Senior
    Remote work
    Visa sponsorship
    Flexible hours

    Sigao Studios

    Birmingham, AL
    1 day ago
  • $89.6k - $123.2k

    Senior AI/DevOps Engineer CGI is seeking a Senior AI/DevOps Engineer to join a fast-paced technology team supporting a Corporate Risk and Compliance...  ...codebase, building GenAI powered assistants, intelligent automation, and data driven applications on AWS. This isn't a role... 
    Senior

    CGI

    Birmingham, AL
    1 day ago
  • $171.6k - $392.1k

     ...cutover, and managing post-go-live support activities. As a Senior Manager in our Technology practice, you will play a pivotal role...  ..., while building trust in capital markets. Enabled by data, AI and advanced technology, EY teams help clients shape the future with... 
    Senior
    Summer holiday
    Remote work
    Flexible hours

    EY

    Birmingham, AL
    4 days ago
  •  ...Senior AI Engineer At Jack Henry, we're more than a technology company, we're a force for good...  ...retirement of the application. Develops, tests and reviews applications based on...  ...Experience with unit testing and end-to-end automated testing. Experience with Kubernetes.... 
    Senior
    Permanent employment
    H1b
    Work at office
    Local area
    Shift work
    1 day per week

    Jack Henry and Associates

    Birmingham, AL
    1 day ago
  •  ...Job Title: Senior Product Analyst/System Analyst Location: Birmingham, AL, Columbia, SC...  ...and resolving production issues. Support testing phases including SIT, UAT, and end-to-end...  ...beneficial. ~ Comfortable using generative AI tools such as Microsoft Copilot or ChatGPT... 
    Senior
    Permanent employment
    Full time
    Local area

    System One

    Birmingham, AL
    a month ago
  • About the roleYou love security, automation, and building systems that...  ...up. As a Security Operations Engineer, you’ll protect the networks behind...  ...internal penetration tests, and scans for unencrypted credit...  ...GDPR.Working proficiency with AI tools relevant to security work... 
    Full time
    Casual work
    Remote work
    Flexible hours

    Daxko

    Birmingham, AL
    3 days ago
  • Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia, SC Work Arrangement: Hybrid Employment...  ...modern AWS technologies, complex data challenges, AI-enabled automation, and cross-functional collaboration within a major... 
    Senior
    Full time

    System One Holdings, LLC

    Birmingham, AL
    1 day ago
  • $130k - $170k

     ...Senior Mechanical Engineer - Data Centers Job Locations US-TX-Austin | US-GA-Atlanta | US-CA-Santa Clara | US-MN-St. Paul | US-MI-Troy...  ...shaping the future of data center facilities and high-performance AI factory design. In this role, you will work closely with... 
    Senior
    Full time
    For contractors
    Local area
    Remote work
    Flexible hours

    SSOE

    Birmingham, AL
    11 hours ago
  • Do not enter a job description in this field. AI will refine the above job description within approximately 15 minutes, after which you may review, edit, approve, and publish the position. Ref: #404-IT Pittsburgh
    Senior

    System One

    Birmingham, AL
    3 days ago
  • $103.71k - $138.28k

     ...Lumen is the trusted network for the AI‑powered world, connecting people, data, and...  ...join us today. The Role The Field Engineering Quality Control Engineer is responsible for...  ...reviews, stakeholder communications, and Senior Leadership-level reporting when required.... 
    Full time
    Temporary work
    For contractors

    Lumen

    Birmingham, AL
    4 days ago
  •  ...Intelligence with Cutting-Edge AI/ML Are you an experienced...  .../camera fusion. As a Senior Machine Learning Engineer , you will build ML models...  ...of new field data, and automated evaluation pipelines What...  ...collection trips at our Pensacola test facility · A culture that... 
    Senior
    Permanent employment
    Full time
    Visa sponsorship
    Work visa

    Tocaro Blue

    Birmingham, AL
    a month ago
  • $106k - $159.5k

    Senior Software Engineer Our not-so-secret sauce. Award-winning, inclusive, Top Workplace culture doesn...  ...full stack capabilities for an AI-powered insurance document analysis platform...  ...bar through thoughtful code reviews, automated testing, observability, and technical... 
    Senior
    Minimum wage
    Local area
    Remote work
    Night shift

    Marsh McLennan Agency

    Birmingham, AL
    1 day ago
  • $167.7k - $245.2k

     ...generation of products for securing workloads and AI infrastructure using advanced...  ...environments.As part of the Windows agent engineering team, you will focus on extending these security...  ...environments. You will work closely with senior engineers, architects, and cross-... 
    Senior
    Full time
    Temporary work
    Local area
    Remote work
    Flexible hours

    CISCO Systems

    Birmingham, AL
    1 day ago
  • $125.8k - $209.7k

     ...working world. The Opportunity As a Cyber AI Engineer at the Senior Consultant level in EY's Cyber...  ...outputs, and build evaluation datasets and test harnesses to measure accuracy, false...  .... Proficiency in Python, APIs, Git, automated testing, and production delivery, plus... 
    Senior
    Immediate start

    EY

    Birmingham, AL
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior AI Test Automation Engineer. Be the first to apply!