Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior QA Automation Engineer

IDT

We are looking for an experienced Senior QA Engineer to own quality for our AI voice virtual agent. This role is for someone who is equally comfortable reasoning about a large language model's behavior and reading a SIP trace. You should be interested in testing non-deterministic systems. A voice agent takes a live phone call, transcribes speech that may be accented or noisy, interprets intent, reasons over an LLM, generates a spoken reply, and decides whether to resolve the request or hand it to a human—all within a latency budget callers will notice. Nothing in that chain can be validated with a simple assertion of equality, so you will design the evaluation approach that makes it testable anyway. You should also be interested in the underlying telephony. Testing this agent means placing real calls: provisioning SIP endpoints and softphones, driving calls programmatically, controlling media conditions, and distinguishing a model failure from a codec, jitter, or call-routing issue. Responsibilities Embed fully within a Scrum team , participating in all ceremonies and maintaining associated artifacts. Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers. Design test scenarios for complex conversational behavior , including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery. Build and maintain an evaluation framework for non-deterministic output , leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes. Adversarially test the agent against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data. Prepare and provision SIP devices , endpoints, and call scenarios to automate call origination end-to-end. Diagnose failures across the entire stack , isolating issues down to the LLM, ASR (speech recognition), TTS (text-to-speech), media path, or call routing layer. Extend automated coverage and integrate test suites into the CI/CD pipeline to serve as release gates. Define and track core quality metrics , including task completion, containment, intent accuracy, transcription accuracy, response latency, and escalation correctness. Maintain documented test coverage in the TMS with full traceability to business requirements. Communicate and elevate quality risks to stakeholders backed by data and evidence. Requirements 5+ years in QA , with recent hands-on ownership of a complex, integrated product. Practical experience testing SIP/VoIP systems , including call flows, SIP signaling, media handling, softphones, or SIP trunking, with the ability to troubleshoot using SIP traces or packet captures. Demonstrated creativity in test design (live scenario design will be a core part of the interview process and heavily weighted). Proficiency in Python, sufficient to independently build and maintain automation frameworks. Experience testing non-deterministic or probabilistic systems, with a clear approach to designing assertions for variable outputs. Strong analytical skills, including the ability to define quality metrics, justify their calculation, and use data to back release decisions. Hands-on experience with formal TMS tools and Jira. Professional fluency in English (spoken and written), with reliable overlap with US Eastern business hours. We use the following technologies Python with pytest for test automation. Playwright for browser and WebRTC-based call driving. JsSIP / WebRTC for programmatic call origination and media control. SIP softphones and devices for manual and exploratory testing. Allure TestOps as our TMS, linking automated runs to documented test cases. Jira for project management and issue tracking. Jenkins & GitHub Actions for CI/CD pipelines. Grafana for quality and delivery metrics. LLM APIs and evaluation tooling — currently in active development, and we are open to any effective approach. Pluses Direct experience testing LLM-based or conversational AI products , including evaluation frameworks and LLM-as-a-judge approaches. Familiarity with speech technologies, including ASR/transcription accuracy measurement (WER), TTS quality assessment, endpointing, and silence detection. Experience with RAG or retrieval-grounded systems, specifically in evaluating grounding correctness and context retrieval. WebRTC and media-layer testing experience, including codec behavior and performance degradation under jitter or packet loss. Background in testing IVR, contact center, or conversational IVR platforms. Multilingual testing capabilities, with emphasis on Spanish/English code-switching. Literacy in prompt engineering, sufficient to analyze system prompts and identify potential failure modes We are pleased to offer Remote work opportunity! B2B Employment ($, gross). Stable job with long-term growth perspective. Competitive salary with annual performance review. Really good hardware. An exciting and challenging job with talented people around. Continuous learning and career growth opportunities. Compensation for professional training, seminars, and conferences. Referral program: get rewarded for helping us grow the team with talented people. Company-supported English classes to enhance your professional growth. Creativity is a core requirement for this role, not a nice-to-have. The defects that matter here are found by someone who imagines the caller who interrupts mid-sentence, switches languages halfway through, gives a partial account number, answers a different question, or deliberately tries to jailbreak the agent. Scenario design is the primary skill we are hiring for. Current State: We have automated call testing built on JsSIP/WebRTC with Python and Playwright, alongside a growing body of test scenarios. The evaluation layer—how we score responses consistently, catch regressions when a prompt or model changes, and report quality metrics leadership can act on—is where this role will have the most impact. About us IDT Technologies is a part of IDT Corporation —an American telecommunications company founded in 1990 and headquartered in New Jersey. It is an industry leader in prepaid communication and payment services and one of the world's largest international voice carriers. We are listed on the NYSE, employ over 1,500 people across 20+ countries, and have revenues in excess of $1.5 billion. Net2Phone is a global provider of unified cloud communications solutions. It offers an integrated communications suite featuring powerful collaboration tools, including voice, messaging, chat, and unlimited domestic and international calling to 23+ countries. #J-18808-Ljbffr IDT

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior QA Automation Engineer in New York, NY vacancy
  • ACT DIGITAL poszukuje specjalistów QA do testów automatyzowanych i eksploracyjnych. Pracownik będzie testował aplikacje oparte na React, TypeScript i Java, wykorzystując Playwright w chmurze Google Cloud. Do zadań należeć będzie tworzenie scenariuszy testowych wspieranych... 
    Senior

    act digital

    New York, NY
    2 days ago
  • epiqsystems is seeking a QA Automation Engineer to enhance the quality of cloud-based web applications through hands-on test automation and broader quality engineering. You will plan and execute testing across features and APIs, investigate failures with developers, and... 
    Senior
    Remote job

    epiqsystems

    New York, NY
    4 days ago
  • Ready is seeking a Senior QA Automation Engineer to own quality for a modern Full Stack loan origination platform. From day one you will join the development team (Shift Left) and build testing infrastructure to ship with confidence. This 100% remote role is based in Mexico... 
    Senior
    Remote job
    Shift work

    Ready

    New York, NY
    2 days ago
  • O firmie Naszą misją jako firmy świadczącej usługi IT jest dostarczanie najlepszych rozwiązań naszym klientom na całym świecie. Osiągamy to poprzez analizowanie potrzeb naszych klientów i dopasowywanie ich do umiejętności i aspiracji naszych pracowników. Dlatego jedną z...
    Senior
    Work at office

    act digital

    New York, NY
    2 days ago
  •  ...reliability of cloud-based web applications through hands-on test automation and broader quality engineering. You will plan and execute testing across application...  ...partners. Requirements At least five years in software QA or testing, including several years of practical... 
    Senior
    Remote work

    epiqsystems

    New York, NY
    4 days ago
  • Senior QA Automation EngineerJob#: 3059300Job Description:Senior QA Automation EngineerLocation: New York, New York (On-site)Role OverviewWe are seeking a Senior QA Automation Engineer to establish and execute the quality engineering strategy for a cutting-edge Digital... 
    Senior

    Apex Systems

    New York, NY
    3 days ago
  • $30 per hour

    Apple Store Mesa is seeking an Automation QA Engineer focused on security. The role involves developing and running automated tests in a fast-moving team environment on security-related features. The position pays $30-$41 per hour, is full-time, and sits in Mesa, Arizona... 
    Senior
    Hourly pay
    Full time

    MaziCTools

    New York, NY
    3 days ago
  • Location: Mexico Senior QA Automation Engineer+ Inglés We are looking for a Senior QA Automation Engineer who will be responsible for the quality of a modern Full Stack loan origination platform. This is not a traditional QA role; you will be part of the development team... 
    Senior
    Full time
    Work at office
    Remote work
    Monday to Friday
    Early shift

    Ready

    New York, NY
    2 days ago
  •  ...most meaningful work of your career and see your impact at scale. Job Description: Epiq is looking for an experienced Senior QA Automation Engineer to help ensure the quality and reliability of our cloud-based applications. This is a hands‑on quality engineering role... 
    Senior
    Local area
    Remote work

    Epiq

    New York, NY
    2 days ago
  • ЗащитаИнфоТранс, ФГУП ищет тестировщика-универсала с опытом автотестирования на Java и навыками ручного тестирования ПО, грейд Middle+ (3-5 лет). Обязанности: ручное функциональное тестирование веб, мобильных и десктоп‑приложений; API и микросервисов; подготовка тестовой...
    Senior
    Remote job

    ЗащитаИнфоТранс, ФГУП

    New York, NY
    4 days ago
  • Boardroom Appointments seeks an experienced QA Engineer to ensure product stability by collaborating with QA Lead, Development and Portfolio...  ...teams. You will design test plans, write test cases, develop automation, and report results for projects. The role requires strong... 
    Senior

    Boardroom Appointments

    New York, NY
    1 day ago
  • $110k - $130k

     ...the 21st century and the age of 5G and IoT, with world class engineering, best-in-class user experience, and features that will...  ...join our team. Job Description: We are looking for a Senior QA Automation Engineer who can take ownership of our end-to-end automation... 
    Senior
    Work at office
    Flexible hours

    US Mobile

    New York, NY
    22 days ago
  • Ciklum is seeking a Senior Automation QA Engineer (JavaScript) to join our team full-time in India. You will own automated test suites across frontend and backend, including non-deterministic AI/non-deterministic system testing, and work with Temporal workflows and multi... 
    Senior
    Full time

    Description Ciklum

    New York, NY
    2 days ago
  •  ...Description: o Perform functional and automation testing for Capital Markets ensuring...  ...model and customer needs and apply/modify QA techniques accordingly o Provide effective...  ...science, Information Systems or Engineering is required, or in lieu, a demonstrated... 
    Senior
    Work experience placement

    TechDigital Corporation

    New York, NY
    22 days ago
  • Epiq is seeking an experienced Senior QA Automation Engineer to ensure quality and reliability of cloud-based applications. This hands-on role requires strong automated testing skills and collaboration with developers across the SDLC. You will design and execute automated... 
    Senior

    Epiq

    New York, NY
    2 days ago
  • Epiq is looking for an experienced Senior QA Automation Engineer to help ensure the quality and reliability of our cloud-based applications. This hands-on role requires strong automated testing, understanding of modern web applications and APIs, and collaboration with developers... 
    Senior
    Remote job

    Remote Queer

    New York, NY
    2 days ago
  • Ciklum is looking for a Senior Automation QA (JavaScript) Engineer to join our team full-time in India. We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With... 
    Senior
    Full time
    Contract work

    Description Ciklum

    New York, NY
    2 days ago
  • Alex Staff Agency is seeking a Senior QA Engineer with Automation to join our Engineering team in the United States. The role balances manual testing with automation, aiming for a 70/30 automation/manual split in the mid term and involves testing complex web apps across... 
    Senior

    Employers from Greece

    New York, NY
    2 days ago
  •  ...meticulous production standards. You will collaborate with development teams to review specs, plan and execute test coverage, and drive QA practices across product lines. The ideal candidate has 3+ years in QA, strong analytical skills, experience with testing tools, and... 
    Senior

    Bilions LLC

    New York, NY
    16 hours ago
  •  ...the 21st century and the age of 5G and IoT, with world class engineering, best-in-class user experience, and features that will...  ...our team.   Job Description:   We are looking for a Senior QA Automation Engineer who can take ownership of our end-to-end automation... 
    Senior
    Full time

    USMobile

    New York, NY
    5 days ago
  • Future Processing poszukuje Senior QA Engineer z silnym doświadczeniem w Playwright i automatyzacji testów. Szukamy lidera, który potrafi projektować strategie testowe, współpracować z zespołem i mentorować młodszych testerów. Wymagane jest bardzo dobre zrozumienie SQL... 
    Senior

    Future Processing

    New York, NY
    2 days ago
  •  ...businesses. Through real-time data, automation, and actionable insights delivered...  ..., innovation, and agility. As the Senior Salesforce Test Automation Engineer, you will lead the expansion of automated...  ...standards. Contribute to QA governance, release management, Agile... 
    Senior
    Immediate start
    Remote work

    Calix

    New York, NY
    2 days ago
  • Xebia is seeking an experienced Quality Engineer to lead verification for migrations and ensure...  ...will design test strategies, implement automation, and validate REST APIs and frontend...  ...metrics. Ideal candidates have 5+ years in QA, strong SQL, evidence of mutation testing... 
    Senior
    Remote work

    Poland and Central-Eastern Europe

    New York, NY
    3 days ago
  •  ...Job Summary We are seeking a hands-on Senior Network Test & Automation Engineer with strong expertise in enterprise networking, network testing, Python automation, Linux, and AI-assisted engineering. The candidate will be responsible for testing and certifying network... 
    Senior
    Contract work

    PB consulting

    New York, NY
    1 day ago
  • $25 per hour

     ...hired, no OTP, W-8, or other types of sponsorship Senior Quality Assurance Engineer (Manual + Automation) NextGen Coding Company Location: United States...  ...operations. The company is seeking a Senior QA Engineer capable of owning the full quality... 
    Senior
    Hourly pay
    Contract work
    Remote work

    NextGen Coding Company

    New York, NY
    16 hours ago
  •  ...commitment to making a difference through technology. The Senior Test Automation Engineer plays a key role in designing, building, and validating...  ...scoping. - Review project documentation, provide actionable QA feedback, and participate in formal sign-off processes.... 
    Senior
    Full time
    Work at office
    Shift work
    Night shift
    Weekend work
    Afternoon shift

    Office of Technology and Innovation

    New York, NY
    1 day ago
  •  ...Accelirate’s mission is to help companies accelerate automation within their business processes using Robotic Process Automation...  ...Language Processing (NLP). We are currently seeking Senior Automation Engineers with at least 1.5 years of extensive hands- on development... 
    Senior
    Full time

    Accelirate Inc

    New York, NY
    5 days ago
  •  ...As aSenior Automation Engineer, you will be the architect of efficiency. Your goal is to eliminate manual bottlenecks and build seamless, automated ecosystems that empower our commercial teams and clients to scale at speed. Your Responsibilities Build and maintain... 
    Senior
    Work at office
    Work from home
    Flexible hours

    LogIQ Curve Pvt. Ltd.

    New York, NY
    4 days ago
  •  ...objectives. The ideal candidate should have 4 years of experience in QA methodology and validation across SDLC and STLC, with expertise...  ...CSS Selectors and XPath expressions, as well as techniques to automate AJAX calls, Child Windows, and IFrames. Required Skills:... 

    Macpower Digital Assets Edge Private Limited

    New York, NY
    20 days ago
  •  ...Opportunity We're looking for a QA Developer Co-op to join our R...  ...work mainly in Java, writing automated tests around the ways real...  ...currently studying Software Engineering, Computer Science, or a related...  ...rely on Mentorship from senior developers who'll invest in how... 
    Full time
    Work experience placement
    Remote work
    Flexible hours

    Flexspring

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior QA Automation Engineer. Be the first to apply!