Senior AI Test Engineer: Build Scalable AI & LLM Evaluations
TWG Global AI
TWG Global AI in New York is seeking a Senior or Staff AI Software Engineer in Test to develop test automation frameworks for AI products. The ideal candidate will have significant experience in Python and Java, along with a strong software engineering background focused on test automation. This role is a great opportunity to work collaboratively with AI engineers and contribute directly to deploying high-quality AI solutions in a rewarding environment. #J-18808-Ljbffr TWG Global AI
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Senior AI Test Engineer: Build Scalable AI & LLM Evaluations in New York, NY vacancy
- A well-funded startup in New York is looking for a Senior Fullstack Engineer to lead the development of their AI-driven platform. You will work hands-on across the stack, designing and building scalable systems that process millions of web events daily. The ideal candidate...Senior
$229.9k - $262.4k
Senior Lead AI Engineer (AI Foundations, LLM Core and Agentic AI) Overview At Capital... ...to continuing to build world‑class applied... ...product experiences and scalable, high‑performance... ...Design, develop, test, deploy, and... ...guardrails, model evaluation, experimentation, governance...SeniorFull timePart timeLocal area- A leading tech firm in New York is seeking a Senior AI Engineer to join its Equities Technology AI group. This role entails managing the design... ...will have a strong background in Python, experience with LLM frameworks, and a PhD or Master's with substantial work experience...SeniorWork experience placement
$178k - $267k
AlphaSense is looking for a passionate software engineer to join their AI & Search mission in New York. In this role, you'll architect and implement scalable services for AI and search products, improving reliability and cost efficiency. You should have solid back-end engineering...Senior$197.3k - $225.1k
...Lead AI Engineer (AI Foundations, LLM Core and Agentic AI) Overview At Capital... ...to continuing to build world‑class applied... ...experiences and scalable, high‑performance AI... ...One. Design, develop, test, deploy, and support... ..., guardrails, model evaluation, experimentation, governance...SuggestedFull timeLocal areaCapital One National Association
New York, NY3 days ago- ...re looking for an Senior AI Engineer to join a new AI Engineering team building foundational AI... ...engineers with early LLM/agentic AI... ...quality improvements. Evaluation & Quality:... ...frameworks, automated tests, experiments, regression... ...CD workflows, or scalable production...Senior
Talkiatry
New York, NY17 hours ago $175k
...Assured Guaranty is seeking a Senior AI Software Engineer to build production-grade AI... ...translating business workflows into scalable, governed software that... ...Experience building production LLM-based applications (prompting, integration, evaluation) Exposure to agent-based...SeniorAssured Guaranty
New York, NY2 days ago$197.3k - $225.1k
...Lead AI Engineer (AI Foundations, LLM Customization and Finetuning) Overview... ...to continuing to build world-class applied... ...experiences and scalable, high-performance AI... ...Design, develop, test, deploy, and support... ..., guardrails, model evaluation, experimentation, governance...Full timePart timeLocal areaCapital One Financial Corp
New York, NY3 days ago- ...Senior AI Engineer Washington D.C. / New York... ...functionally to deliver scalable AI-powered... ...design, development, evaluation, and deployment of... ..., including back-testing, rejection inference... ...AI/ML, GenAI/LLM technologies, staying... ...experience building and deploying machine...SeniorFlexible hours
VantageScore®
New York, NY2 days ago $160k - $170k
...more. Role As a Senior AI Engineer focused on... ...state management Build and maintain MCP... ...Integrate with managed LLM services across... ...balancing quality, scalability, latency, cost,... ...implement automated evaluation frameworks to... ...around reliability, testing, and operational...SeniorOctus Inc
New York, NY1 day ago$144.5k - $230k
...Senior AI Engineer We are the better way to work in finance... ...want to be part of building something great. Come... ...and RAG pipelines to evaluation frameworks and production... ...findings into scalable, reliable tools that... ...development, or applied LLM engineering Hands-...SeniorWork at officeLocal areaRemote work2 days per weekAccordion USA
New York, NY6 days ago- A tech-driven company is seeking an AI Engineer to design, build, and operate machine learning systems in a remote role based in the United States... ...for developing end-to-end ML pipelines, implementing evaluation metrics, and optimizing production services. Strong Python...Remote job
$229.9k - $262.4k
...Senior Lead AI Engineer (FM Hosting) Overview: At... ...to continuing to build world-class applied... ...experiences and scalable, high-performance... ...Design, develop, test, deploy, and support... ..., model evaluation, experimentation,... ...state-of-the-art LLM optimization techniques...SeniorFull timePart timeLocal areaCapital One
New York, NY4 days ago$175k - $245k
...Senior Software Engineer II - Applied AI and Evaluations (Remote Eligible) -REMOTE, USA- For over 20 years... ...management to smart, scalable solutions, we’ve always... ...with flow. We’re building tools that empower teams... ...at the intersection of LLM evaluation, prompt and...SeniorFull timeTemporary workLocal areaImmediate startRemote workSmartsheet
New York, NY2 days ago- A leading technology research organization is seeking an AI Data Engineer to build scalable data pipelines and collaborate with data scientists on impactful AI initiatives. This is a fully remote role ideal for someone with at least 4 years of Data Engineering experience...Remote job
- Talkiatry is seeking a Senior AI Engineer to join their new AI Engineering team. The role involves... .... You'll work across the AI lifecycle, building backend APIs, frontend interfaces, and... ...This position is ideal for engineers with LLM experience and a desire to grow their...Senior
- ...Senior Applied AI Engineer Paramount Skydance Corp. is seeking... ...to architect, build, and... ...Vertex AI to deliver scalable, resilient, and impactful... ...releases, flaky test detection, intelligent... ..., hybrid LLM architectures.... ...Develop model evaluation frameworks ensuring...Senior
Paramount Global Services
New York, NY1 day ago $180k - $200k
...insurance. The Data and AI team at Starr builds first-of-a-kind... ...growth for Starr. The Senior AI/ML engineer is an important,... ...responsible for developing scalable solutions to provide... ...model fine-tuning, evaluation, and implementation,... ...working with LLM. ~ Expertise in Python...SeniorStarr Insurance Companies
New York, NY1 day ago$229.9k - $262.4k
...Senior Lead AI Engineer(MLX, Agentic AI, Gen AI platform... ...to continuing to build world-class applied... ...experiences and scalable, high-performance... ...Design, develop, test, deploy, and support... ..., model evaluation, experimentation,... ...state-of-the-art LLM optimization techniques...SeniorFull timePart timeLocal areaCapital One
New York, NY7 days ago- Job Title: Senior AI Engineer (5+ Years Experience) Location... ...department is responsible for building and deploying next-... ...layers. Integrate LLM-powered capabilities... ...outputs. Implement evaluation, monitoring, and... ...translate requirements into scalable AI solutions....SeniorRemote workWork from home
- Framework Ventures is looking for a Security Engineer to build AI-driven security infrastructure for LLM applications. The role involves designing multi-agent systems and integrating security methods into development pipelines. Candidates should have at least 3 years of...Senior
- A pioneering VBC technology firm is seeking a Full Stack AI Engineer for a remote opportunity. In this role, you will join a small,... ...organizations. Candidates should have extensive experience in building scalable systems, a passion for user experiences, and proficiency in...Remote job
- ...United States is looking for a Full Stack AI Engineer to develop innovative solutions in a remote environment. The role entails building customer-centric user experiences and... ...engineers and scientists to enhance product scalability. Ideal candidates will have over 5 years...Remote job
$142.32k - $213.48k
Senior AI Engineer for Banking Technology Apply (opens... ...to design, build, and deploy cutting... ...on implementing scalable, agentic AI frameworks... ...integrating LLM‑driven workflows... ...and improvement. Evaluation & Optimization: Define... ...including Git, CI/CD, testing, code reviews,...SeniorFull time- HartleyCo in New York is hiring a Senior Engineer to lead the development of production-level AI systems. This role involves designing workflows, improving retrieval pipelines, and ensuring reliability in a high-autonomy environment. You will work directly with the founders...Senior
$165k - $230k
BetterUp, Inc. is seeking an experienced engineer for an applied agent engineering role in New York. You will work on AI tools that enhance workflows across various functions, including HR, Sales, and Customer Success. A background of 7+ years in engineering and expertise...SeniorFlexible hours- 6AM City, LLC is seeking a skilled engineer in Georgia to enhance AI capabilities in customer and employee experiences. This role demands expertise in software engineering with a solid understanding of AI and machine learning, particularly in AWS environments. The ideal...Senior
- ...Job Description The Senior Vice President, AI Engineering is a senior... ...reliability, security, and scalability at enterprise scale... ..., including LLM-powered applications... ...management, observability, evaluation frameworks,... ...operational stability Build and mentor engineering...SeniorFlexible hours
- ...Senior AI Platform Engineer Locations: San Francisco, CA / Jacksonville,... ...around the globe. We are building a next-generation... ...infrastructure, and enabling scalable, intelligent, and... ...deployment of AI agents. Evaluate new agentic AI, LLM, and orchestration technologies...SeniorCasual workFlexible hours
Blue Prism Group
New York, NY1 day ago - Senior AI Engineer About Supernal Supernal helps small-... ...Senior AI Engineer to build and ship the first... ...systems, evaluation harnesses, and the... ...agent infrastructure: Scalable orchestration patterns... ...: curated test sets + scenario suites... ...building agentic or LLM‑powered systems...SeniorFull timeTemporary workRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Test Engineer: Build Scalable AI & LLM Evaluations. Be the first to apply!
Related searches
- ai engineer New York, NY
- machine learning ai engineer New York, NY
- ai research engineer New York, NY
- ai ml engineer New York, NY
- senior ai engineer New York, NY
- ai prompt engineer New York, NY
- ai developer New York, NY
- ai engineer remote New York, NY
- test engineer New York, NY
- performance test engineer New York, NY