Lead AI QA Engineer - Evaluation & Reliability
BMC Software Inc
BMC Software, Inc. is seeking a skilled AI evaluation engineer to build evaluation frameworks, golden datasets, and rubrics for generative AI products. You will stand up automated CI evaluation harnesses and drive down hallucinations and drift while ensuring release quality. You will own defined eval work from definition through delivery. You will collaborate with DS, engineering, and product teams, design robust testing strategies for multi-agent systems, and champion Responsible AI across the #J-18808-Ljbffr BMC Software
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Lead AI QA Engineer - Evaluation & Reliability in New York, NY vacancy
- Equiliem is seeking an AI Evaluation/Test Engineer to test AI-powered agent experiences across customer... ...on prompt and agent evaluation, QA testing, observability, and defect management... ...to identify quality gaps and improve reliability, accuracy, and user experience in AI...SuggestedRemote job
- BMC Software, Inc. in the United States is hiring a Principal Quality Automation Engineer to shape evaluation frameworks for Generative AI and agentic AI systems. You will own the end-to-end evaluation process and build robust datasets, rubrics, and CI-tested evaluation...SuggestedRemote job
- ...Job Title AI Evaluation Engineer Location Hybrid / Remote Employment Type... ...and product teams to improve AI system reliability and user experience. Key Responsibilities... ...scientists, product managers, and QA teams throughout the AI development...SuggestedFull timeRemote work
Ova Technologies
New York, NY5 days ago $229.9k - $262.4k
AI Engineer 5 (Gen AI SDK: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been... ...engineering teams to deliver our industry leading capabilities with breakthrough...SuggestedFull timePart timeLocal areaCapital One
New York, NY14 hours ago- ...next steps. Our partner is looking for an AI Evaluation Engineer based in United States. As a... ...ensures production-grade AI agents perform reliably, safely, and efficiently. You will develop... ...pipelines that go beyond traditional QA and manual prompt testing. You will establish...SuggestedHourly payFor contractorsRemote workFlexible hours
$200k - $250k
Applied AI Engineer, Agents & Evaluation Compensation: $200k-$250k base + equity Location: New York City — fully onsite, 5 days per week Employment... ...these agents behave and how the team proves they are working reliably in production. What You’ll Do Design, build, and iterate...Full timeFor contractors$229.9k - $262.4k
Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital... ...teams to deliver our industry leading capabilities with breakthrough...Full timePart timeLocal areaCapital One
New York, NY14 hours ago- People in AI is hiring an Applied AI Engineer for Agents & Evaluation in New York City, onsite. You will design, build, and iterate on production AI agents, with... ...orchestration, prompts, and guardrails to ensure reliability. You’ll own the evaluation infrastructure,...
- DataAnnotation seeks a Chemist to evaluate AI chemical reasoning, identify unsafe or incorrect conclusions, and provide trained-chemist style answers. You’ll craft prompts that probe mechanisms and analyses, review AI outputs for accuracy and safety, and deliver corrected...Remote jobFlexible hours
- Compunnel, Inc. seeks an Evaluation Engineer - AI Models in New York to evaluate and validate AI/ML and Generative AI models across business and technical use cases. This role combines extensive Quality Engineering and software testing experience with hands-on AI model...
- OpenTeams seeks a Senior AI/ML Test and Evaluation Engineer to build and operate benchmarking and evaluation capabilities at the core of our AI platform. You will design automated metrics and human judgments to compare models and agentic workflows, surface failure modes...Remote job
$152k - $241.5k
...believe open-weight models are foundational to American AI leadership and cybersecurity, and that trust in AI grows... ...and broad scientific scrutiny. Our AI Safety & Security Engineering team builds and evaluates AI-powered tooling that helps find, validate, and patch software...Full timeRemote workNvidia
New York, NY4 days ago$200k - $300k
...Trading (HRT) is seeking a Software Engineer focused on GPU reliability to join our Systems Development team.... ...to know you.Please be advised: Use of AI tools during interviews or assessments... ...upon. We employ various methods to evaluate the authenticity of candidate responses...Work at officeLocal areaImmediate startHudson River Trading
New York, NY1 day ago$200k - $250k
...Description Director of AI Engineering \n Location: New York, NY... ...pipelines, model gateways, and evaluation/observability tooling for... ...emerging AI techniques, lead complex cross-cutting initiatives... ...tradeoffs among safety, reliability, scale, and cost \n Lead,...Local areaRelocationVisa sponsorshipOpen Systems Technologies
New York, NY3 days ago$180k - $250k
Our client, an early-stage AI startup, are hiring a Founding Applied AI Engineer to join their team in New York. The successful candidate will design and build... ...and run A/B tests, bandit systems and offline evaluation frameworks. Define what is stored, inferred and exposed...$100 - $120 per hour
...Snowflake AI Engineer – Cortex Location: New York City - Hybrid 2 days per week Contract... .... Turn business use cases into reliable, production-ready AI applications.... ...including APIs and MCP-based tooling. Evaluation & Quality ~ Set up testing and evaluation...Contract work2 days per weekCloudHive
New York, NY3 days ago$180k - $200k
...Forward Deployed AI Engineer Location: New York, NY Compensation: $180-200K + Bonus... ...software teams to scale prototypes into reliable, production-ready systems. Leverage... ...LlamaIndex. Hands-on experience with AI evaluation metrics, observability tools, and...Local areaImmediate startVisa sponsorshipOpen Systems Technologies
New York, NY4 days ago$160k - $200k
...AI Engineer [Senior Associate / AVP] Founded in 1992, Cerberus is... ...emerging AI capabilities into reliable products that users actually... ...use, structured outputs, evaluation and monitoring considerations... ...experience designing AI systems or leading small AI-native engineering...Cerberus Capital Management
New York, NY14 hours ago- ...Responsibilities Design and build agentic AI applications that can reason,... ...structured outputs. Develop evaluation frameworks to measure agent accuracy, reliability, latency, and task completion.... ...environments. Partner with engineering, data, security, product, and...Full time
The Phoenix Group
New York, NY1 day ago - ...AI Engineer (GenAI Workflows for OTC Derivatives) Location: New York City (SoHo) — In-Person... .... You’ll own key parts of the evaluation and context engineering infrastructure... ...ensuring model outputs are high-quality, reliable, and explainable. You’ll work closely...Full time
Nxt Level
New York, NY3 days ago $115k - $130k
...technology to deliver industry-leading digital solutions.... ..., Data, and Software Engineering, servicing an array of... ...a hands-on Senior AI Engineer to design,... ...stakeholders to deliver reliable, scalable, and... ...including model versioning, evaluation, observability, drift...Full timeTemporary workFlexible hoursSynechron
New York, NY14 hours ago- ...AI / Forward Deployed Engineer Must have 4+ years of professional experience after... ...Engineering searches with leading investment firms and have... ...security, scalability, and reliability. What we’re looking for... ...engineering, or LLM evaluation. Strong problem-solving...
Long Ridge Partners
New York, NY1 day ago - ...Agentic AI Engineer Location: NYC, NY (Hybrid) (Only Local Candidates) Fulltime Need... ...or equivalent technologies. · Evaluate, test, and improve agent performance, including... ...quality, response accuracy, latency, reliability, guardrails, security, and cost...Full timeLocal area
HireOn Tech
New York, NY2 days ago - ...The Senior AI Engineer reports to the Lead AI Engineer and serves as their primary technical partner in... ...and product teams to move AI products reliably into production. Key... ...Lead AI Engineer. Build and apply evaluation frameworks for quality, groundedness...Full time
Atlantic Partners Corporation
New York, NY2 days ago - ...early-stage, research-driven AI startup building the infrastructure... ...of AI research and engineering, developing systems that enable... ...support the development and evaluation of advanced AI agents. Translate... ...systems. Develop reliable, scalable platforms and services...Full time
Tenth Revolution Group
New York, NY3 days ago - ...Job Title: Agentic AI Engineer Experience: 8–10 Years Employment Type: Long term Contract... ...Platform (GCP) . Implement LLM evaluation, observability, monitoring, guardrails,... ...applications for performance, scalability, reliability, latency, and cost. Collaborate with...Long term contract
Acunor
New York, NY2 days ago - ...Role:- Applied AI Engineer – GenAI / LLMOps Location: New York, NY 10019 Duration: Contract... ...selection, orchestration patterns, and evaluation strategies. Establish and enhance... ..., regression testing, and production reliability. Design and implement controls for entitlements...Contract work
PTR Global
New York, NY4 days ago $200k - $225k
...Senior AI Engineer - Agentic Systems Remote (NYC HQ) $200,000-225,000 Base Salary + Equity... ...and marketing processes into reliable, agent-driven solutions. Develop the... ...deploying trustworthy AI systems, including evaluation frameworks, observability, guardrails,...Work at officeLocal areaRemote workHarnham
New York, NY2 days ago- Mercor is hiring experienced music producers and audio engineers to evaluate generative music AI models. You will assess AI-generated music across genres and rate it against detailed quality standards, working in Hindi and English. Responsibilities include head-to-head...Remote work
$195k - $225k
...SummaryWe are pioneering a new AI development team dedicated to... ...and deeply technical AI Engineer to be a foundational member of... ...large language models, ensuring reliability, performance, and... ...the AI ecosystem, continuously evaluating emerging models, tools, and frameworksLead...Full timeHorizon Media
New York, NY3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead AI QA Engineer - Evaluation & Reliability. Be the first to apply!
Related searches
- lead engineer New York, NY
- lead operating engineer New York, NY
- lead security engineer New York, NY
- lead app. developer New York, NY
- lead infrastructure engineer New York, NY
- lead product engineer New York, NY
- lead network engineer New York, NY
- lead quality engineer New York, NY
- lead system engineer New York, NY
- lead algorithm engineer New York, NY