Senior QA Automation Engineer
IDT
We are looking for an experienced Senior QA Engineer to own quality for our AI voice virtual agent. This role is for someone who is equally comfortable reasoning about a large language model's behavior and reading a SIP trace. You should be interested in testing non-deterministic systems. A voice agent takes a live phone call, transcribes speech that may be accented or noisy, interprets intent, reasons over an LLM, generates a spoken reply, and decides whether to resolve the request or hand it to a human—all within a latency budget callers will notice. Nothing in that chain can be validated with a simple assertion of equality, so you will design the evaluation approach that makes it testable anyway. You should also be interested in the underlying telephony. Testing this agent means placing real calls: provisioning SIP endpoints and softphones, driving calls programmatically, controlling media conditions, and distinguishing a model failure from a codec, jitter, or call-routing issue. Responsibilities Embed fully within a Scrum team , participating in all ceremonies and maintaining associated artifacts. Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers. Design test scenarios for complex conversational behavior , including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery. Build and maintain an evaluation framework for non-deterministic output , leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes. Adversarially test the agent against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data. Prepare and provision SIP devices , endpoints, and call scenarios to automate call origination end-to-end. Diagnose failures across the entire stack , isolating issues down to the LLM, ASR (speech recognition), TTS (text-to-speech), media path, or call routing layer. Extend automated coverage and integrate test suites into the CI/CD pipeline to serve as release gates. Define and track core quality metrics , including task completion, containment, intent accuracy, transcription accuracy, response latency, and escalation correctness. Maintain documented test coverage in the TMS with full traceability to business requirements. Communicate and elevate quality risks to stakeholders backed by data and evidence. Requirements 5+ years in QA , with recent hands-on ownership of a complex, integrated product. Practical experience testing SIP/VoIP systems , including call flows, SIP signaling, media handling, softphones, or SIP trunking, with the ability to troubleshoot using SIP traces or packet captures. Demonstrated creativity in test design (live scenario design will be a core part of the interview process and heavily weighted). Proficiency in Python, sufficient to independently build and maintain automation frameworks. Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs. Strong analytical skills, including the ability to define quality metrics, justify their calculation, and use data to back release decisions. Hands-on experience with formal TMS tools and Jira. Professional fluency in English (spoken and written), with reliable overlap with US Eastern business hours. We use the following technologies Python with pytest for test automation. Playwright for browser and WebRTC-based call driving. JsSIP / WebRTC for programmatic call origination and media control. SIP softphones and devices for manual and exploratory testing. Allure TestOps as our TMS, linking automated runs to documented test cases. Jira for project management and issue tracking. Jenkins & GitHub Actions for CI/CD pipelines. Grafana for quality and delivery metrics. LLM APIs and evaluation tooling — currently in active development, and we are open to any effective approach. Pluses Direct experience testing LLM-based or conversational AI products , including evaluation frameworks and LLM-as-a-judge approaches. Familiarity with speech technologies, including ASR/transcription accuracy measurement (WER), TTS quality assessment, endpointing, and silence detection. Experience with RAG or retrieval-grounded systems, specifically in evaluating grounding correctness and context retrieval. WebRTC and media-layer testing experience, including codec behavior and performance degradation under jitter or packet loss. Background in testing IVR, contact center, or conversational IVR platforms. Multilingual testing capabilities, with emphasis on Spanish/English code-switching. Literacy in prompt engineering, sufficient to analyze system prompts and identify potential failure modes We are pleased to offer Remote work opportunity! B2B Employment ($, gross). Stable job with long-term growth perspective. Competitive salary with annual performance review. Really good hardware. An exciting and challenging job with talented people around. Continuous learning and career growth opportunities. Compensation for professional training, seminars, and conferences. Referral program: get rewarded for helping us grow the team with talented people. Company-supported English classes to enhance your professional growth. Creativity is a core requirement for this role, not a nice-to-have. The defects that matter here are found by someone who imagines the caller who interrupts mid-sentence, switches languages halfway through, gives a partial account number, answers a different question, or deliberately tries to jailbreak the agent. Scenario design is the primary skill we are hiring for. Current State: We have automated call testing built on JsSIP/WebRTC with Python and Playwright, alongside a growing body of test scenarios. The evaluation layer—how we score responses consistently, catch regressions when a prompt or model changes, and report quality metrics leadership can act on—is where this role will have the most impact. About us IDT Technologies is a part of IDT Corporation —an American telecommunications company founded in 1990 and headquartered in New Jersey. It is an industry leader in prepaid communication and payment services and one of the world's largest international voice carriers. We are listed on the NYSE, employ over 1,500 people across 20+ countries, and have revenues in excess of $1.5 billion. Net2Phone is a global provider of unified cloud communications solutions. It offers an integrated communications suite featuring powerful collaboration tools, including voice, messaging, chat, and unlimited domestic and international calling to 23+ countries. #J-18808-Ljbffr IDT
- ACT DIGITAL poszukuje specjalistów QA do testów automatyzowanych i eksploracyjnych. Pracownik będzie testował aplikacje oparte na React, TypeScript i Java, wykorzystując Playwright w chmurze Google Cloud. Do zadań należeć będzie tworzenie scenariuszy testowych wspieranych...Senior
- epiqsystems is seeking a QA Automation Engineer to enhance the quality of cloud-based web applications through hands-on test automation and broader quality engineering. You will plan and execute testing across features and APIs, investigate failures with developers, and...SeniorRemote job
- Ready is seeking a Senior QA Automation Engineer to own quality for a modern Full Stack loan origination platform. From day one you will join the development team (Shift Left) and build testing infrastructure to ship with confidence. This 100% remote role is based in Mexico...SeniorRemote jobShift work
- O firmie Naszą misją jako firmy świadczącej usługi IT jest dostarczanie najlepszych rozwiązań naszym klientom na całym świecie. Osiągamy to poprzez analizowanie potrzeb naszych klientów i dopasowywanie ich do umiejętności i aspiracji naszych pracowników. Dlatego jedną z...SeniorWork at office
- ...reliability of cloud-based web applications through hands-on test automation and broader quality engineering. You will plan and execute testing across application... ...partners. Requirements At least five years in software QA or testing, including several years of practical...SeniorRemote work
- Senior QA Automation EngineerJob#: 3059300Job Description:Senior QA Automation EngineerLocation: New York, New York (On-site)Role OverviewWe are seeking a Senior QA Automation Engineer to establish and execute the quality engineering strategy for a cutting-edge Digital...Senior
$30 per hour
Apple Store Mesa is seeking an Automation QA Engineer focused on security. The role involves developing and running automated tests in a fast-moving team environment on security-related features. The position pays $30-$41 per hour, is full-time, and sits in Mesa, Arizona...SeniorHourly payFull time- Location: Mexico Senior QA Automation Engineer+ Inglés We are looking for a Senior QA Automation Engineer who will be responsible for the quality of a modern Full Stack loan origination platform. This is not a traditional QA role; you will be part of the development team...SeniorFull timeWork at officeRemote workMonday to FridayEarly shift
- ...most meaningful work of your career and see your impact at scale. Job Description: Epiq is looking for an experienced Senior QA Automation Engineer to help ensure the quality and reliability of our cloud-based applications. This is a hands‑on quality engineering role...SeniorLocal areaRemote work
- ЗащитаИнфоТранс, ФГУП ищет тестировщика-универсала с опытом автотестирования на Java и навыками ручного тестирования ПО, грейд Middle+ (3-5 лет). Обязанности: ручное функциональное тестирование веб, мобильных и десктоп‑приложений; API и микросервисов; подготовка тестовой...SeniorRemote job
- Boardroom Appointments seeks an experienced QA Engineer to ensure product stability by collaborating with QA Lead, Development and Portfolio... ...teams. You will design test plans, write test cases, develop automation, and report results for projects. The role requires strong...Senior
$110k - $130k
...the 21st century and the age of 5G and IoT, with world class engineering, best-in-class user experience, and features that will... ...join our team. Job Description: We are looking for a Senior QA Automation Engineer who can take ownership of our end-to-end automation...SeniorWork at officeFlexible hours- Ciklum is seeking a Senior Automation QA Engineer (JavaScript) to join our team full-time in India. You will own automated test suites across frontend and backend, including non-deterministic AI/non-deterministic system testing, and work with Temporal workflows and multi...SeniorFull time
- ...Description: o Perform functional and automation testing for Capital Markets ensuring... ...model and customer needs and apply/modify QA techniques accordingly o Provide effective... ...science, Information Systems or Engineering is required, or in lieu, a demonstrated...SeniorWork experience placement
- Epiq is seeking an experienced Senior QA Automation Engineer to ensure quality and reliability of cloud-based applications. This hands-on role requires strong automated testing skills and collaboration with developers across the SDLC. You will design and execute automated...Senior
- Epiq is looking for an experienced Senior QA Automation Engineer to help ensure the quality and reliability of our cloud-based applications. This hands-on role requires strong automated testing, understanding of modern web applications and APIs, and collaboration with developers...SeniorRemote job
- Ciklum is looking for a Senior Automation QA (JavaScript) Engineer to join our team full-time in India. We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With...SeniorFull timeContract work
- Alex Staff Agency is seeking a Senior QA Engineer with Automation to join our Engineering team in the United States. The role balances manual testing with automation, aiming for a 70/30 automation/manual split in the mid term and involves testing complex web apps across...Senior
- ...meticulous production standards. You will collaborate with development teams to review specs, plan and execute test coverage, and drive QA practices across product lines. The ideal candidate has 3+ years in QA, strong analytical skills, experience with testing tools, and...Senior
- ...the 21st century and the age of 5G and IoT, with world class engineering, best-in-class user experience, and features that will... ...our team. Job Description: We are looking for a Senior QA Automation Engineer who can take ownership of our end-to-end automation...SeniorFull time
- Future Processing poszukuje Senior QA Engineer z silnym doświadczeniem w Playwright i automatyzacji testów. Szukamy lidera, który potrafi projektować strategie testowe, współpracować z zespołem i mentorować młodszych testerów. Wymagane jest bardzo dobre zrozumienie SQL...Senior
- ...businesses. Through real-time data, automation, and actionable insights delivered... ..., innovation, and agility. As the Senior Salesforce Test Automation Engineer, you will lead the expansion of automated... ...standards. Contribute to QA governance, release management, Agile...SeniorImmediate startRemote work
- Xebia is seeking an experienced Quality Engineer to lead verification for migrations and ensure... ...will design test strategies, implement automation, and validate REST APIs and frontend... ...metrics. Ideal candidates have 5+ years in QA, strong SQL, evidence of mutation testing...SeniorRemote work
- ...Job Summary We are seeking a hands-on Senior Network Test & Automation Engineer with strong expertise in enterprise networking, network testing, Python automation, Linux, and AI-assisted engineering. The candidate will be responsible for testing and certifying network...SeniorContract work
$25 per hour
...hired, no OTP, W-8, or other types of sponsorship Senior Quality Assurance Engineer (Manual + Automation) NextGen Coding Company Location: United States... ...operations. The company is seeking a Senior QA Engineer capable of owning the full quality...SeniorHourly payContract workRemote work- ...commitment to making a difference through technology. The Senior Test Automation Engineer plays a key role in designing, building, and validating... ...scoping. - Review project documentation, provide actionable QA feedback, and participate in formal sign-off processes....SeniorFull timeWork at officeShift workNight shiftWeekend workAfternoon shift
- ...Accelirate’s mission is to help companies accelerate automation within their business processes using Robotic Process Automation... ...Language Processing (NLP). We are currently seeking Senior Automation Engineers with at least 1.5 years of extensive hands- on development...SeniorFull time
- ...As aSenior Automation Engineer, you will be the architect of efficiency. Your goal is to eliminate manual bottlenecks and build seamless, automated ecosystems that empower our commercial teams and clients to scale at speed. Your Responsibilities Build and maintain...SeniorWork at officeWork from homeFlexible hours
- ...objectives. The ideal candidate should have 4 years of experience in QA methodology and validation across SDLC and STLC, with expertise... ...CSS Selectors and XPath expressions, as well as techniques to automate AJAX calls, Child Windows, and IFrames. Required Skills:...
- ...Opportunity We're looking for a QA Developer Co-op to join our R... ...work mainly in Java, writing automated tests around the ways real... ...currently studying Software Engineering, Computer Science, or a related... ...rely on Mentorship from senior developers who'll invest in how...Full timeWork experience placementRemote workFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior QA Automation Engineer. Be the first to apply!
- qa test engineer New York, NY
- software quality engineer New York, NY
- web qa engineer New York, NY
- qa automation engineer remote New York, NY
- junior software test automation engineer New York, NY
- software test engineer New York, NY
- qa automation engineer New York, NY
- qa engineer New York, NY
- sdet qa automation engineer New York, NY
- software quality assurance engineer New York, NY






