Lead AI QA Engineer - Evaluation & Reliability
BMC Software Inc
BMC Software, Inc. is seeking a skilled AI evaluation engineer to build evaluation frameworks, golden datasets, and rubrics for generative AI products. You will stand up automated CI evaluation harnesses and drive down hallucinations and drift while ensuring release quality. You will own defined eval work from definition through delivery. You will collaborate with DS, engineering, and product teams, design robust testing strategies for multi-agent systems, and champion Responsible AI across the #J-18808-Ljbffr
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Lead AI QA Engineer - Evaluation & Reliability in New York, NY vacancy
- BMC Software, Inc. in the United States is hiring a Principal Quality Automation Engineer to shape evaluation frameworks for Generative AI and agentic AI systems. You will own the end-to-end evaluation process and build robust datasets, rubrics, and CI-tested evaluation...SuggestedRemote job
$229.9k - $262.4k
...AI Engineer 5 (Gen AI SDK: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been... ...engineering teams to deliver our industry leading capabilities with breakthrough...SuggestedFull timePart timeLocal areaCapital One
New York, NY2 days ago$197.3k - $225.1k
AI Engineer 4 (AI Foundations: Benchmarking, Evaluation, and Explainability)At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has... ...teams to deliver our industry leading capabilities with breakthrough product...SuggestedFull timePart timeLocal areaCapital One Group
New York, NY2 days ago- ...SaidGig seeks a skilled software engineer to evaluate frontier AI models by reviewing AI-generated code against real-world tasks and shaping evaluation... .... You will diagnose correctness, performance, and reliability, enabling high-quality benchmarks and grading rubrics. Ideal...SuggestedRemote work
SaidGig
New York, NY2 days ago - ...Founding AI Engineer, Agents & Evaluation New York City Full time On-site Engineering We're building the intelligence layer for all home-based... ...and the reactive-trap diagnosis. You walk us through a reliability or evaluation system you've built for a production ML/...SuggestedPermanent employmentFull timeWork at officeRemote workNight shift
Zingage
New York, NY4 days ago $229.9k - $262.4k
...AI Engineer 5 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluations) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One... ...teams to deliver our industry leading capabilities with breakthrough...Full timePart timeLocal areaCapital One
New York, NY2 days ago$229.9k - $262.4k
...AI Engineer 5 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has... ...teams to deliver our industry leading capabilities with breakthrough product...Full timePart timeLocal areaCapital One
New York, NY4 days ago$229.9k - $262.4k
...Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years,... ...engineering teams to deliver our industry leading capabilities with breakthrough...Full timePart timeLocal areaITSMF
New York, NY4 days ago- ...People in AI is hiring an Applied AI Engineer for Agents & Evaluation in New York City, onsite. You will design, build, and iterate on production AI agents,... ...orchestration, prompts, and guardrails to ensure reliability. You’ll own the evaluation infrastructure, partner...
People In AI
New York, NY4 days ago - ...Taxo.ai, Inc. is seeking a Software Engineer focused on AI Quality & Reliability in New York City. You will build automated evaluation and regression systems for voice AI agents, ensuring reliable releases and measurable quality across conversations, tool calls, and integrations...Work at office
Taxo.ai, Inc.
New York, NY4 days ago - YO AI Labs is seeking a detail-oriented QA Engineer for a remote contractor role to support AI training projects. You will validate AI outputs, review data annotations, and ensure data quality through rigorous testing and documentation. The position emphasizes analytical...Remote jobFor contractors
$115k - $260k
..., and Great Careers. Sr. Staff AI Engineer, AI Agent Platform (Agent & Evaluation Harness) Why Join GEICO? GEICO... ...that make agents on the platform reliably good at their jobs. You'll work alongside... ...practice across the company. Lead Set technical direction for...Hourly payWork experience placementLocal areaGovernment Employees Insurance Company
New York, NY2 days ago- ...Lead Clinical Automation & AI Systems Engineer New York, NY Lead Clinical Automation & AI Systems Engineer... ...Systems Interoperability: Build reliable API-level integrations connecting... ...safety guardrails, and automated evaluation suites to continuously measure...Remote workDay shift
Generation Next Fertility
New York, NY3 hours ago - ...Zingage (Kuzushi Inc.) is building the intelligence layer for home-based care in New York City. We seek a founding AI engineer to own evaluation, simulation, retrieval, and alignment infrastructure for autonomous agents in a high-impact healthcare setting. You will ship...
Zingage
New York, NY4 days ago - ...Compunnel, Inc. seeks an Evaluation Engineer - AI Models in New York to evaluate and validate AI/ML and Generative AI models across business and technical use cases. This role combines extensive Quality Engineering and software testing experience with hands-on AI model...
Compunnel
New York, NY18 hours ago - ...AI Reliability EngineerWe are seeking an AI Reliability Engineer to ensure the reliability, availability, scalability, and operational excellence of AI and machine learning systems in production. This role combines Site Reliability Engineering (SRE), MLOps, and cloud...Remote work
Ova Technologies
New York, NY4 days ago - ...Mercor is partnering with a leading AI research organization to recruit experienced sales engineers who will define evaluation criteria for real-world technical sales work. You will score samples, provide written justifications, and help ensure reproducible, defensible...
Mercor Inc
New York, NY2 days ago - ...build and implement quality systems for Reliability & Maintenance Engineering (RME). You will guide field teams in... ...-making tools for site managers, evaluate Fulfillment Center RME performance,... ...reliability metrics year over year. You will lead network-wide Continuous Improvement...
Amazon
New York, NY2 days ago - ...Architect (Senior Manager)Build an automated evaluation harness covering correctness, prompt... ...automated evaluation frameworks to validate AI agent correctness, stability,... ...have good communication skills and strong technical skills to support and lead a team....
Virtusa
New York, NY4 days ago - ...Turing is hiring a Principal AI Engineer to lead enterprise-scale Agentic AI implementations for Fortune 500 clients. The role focuses on... ...designing and deploying autonomous, tool-calling AI systems with reliability at scale. The candidate will provide technical leadership,...
SIDRAM TECHNOLOGIES
New York, NY3 days ago $152k - $241.5k
...believe open-weight models are foundational to American AI leadership and cybersecurity, and that trust in AI grows... ...and broad scientific scrutiny. Our AI Safety & Security Engineering team builds and evaluates AI-powered tooling that helps find, validate, and patch software...Full timeRemote workNvidia
New York, NY6 days ago- ...Job description: The Frontier Engineer needs to have a multi-disciplined skill... ...designing and governing agentic systems, leading domain-level AI architecture, setting technical... ...auditability and observability - Ensure reliability of the overall AI skills ecosystem...
TechDigital Group
New York, NY2 days ago $200k - $300k
...Trading (HRT) is seeking a Software Engineer focused on GPU reliability to join our Systems Development team.... ...to know you.Please be advised: Use of AI tools during interviews or assessments... ...upon. We employ various methods to evaluate the authenticity of candidate responses...Work at officeLocal areaImmediate startHudson River Trading
New York, NY8 days ago- ...AI / Forward Deployed Engineer Must have 4+ years of professional experience after... ...Engineering searches with leading investment firms and have... ...security, scalability, and reliability. What we’re looking for... ...engineering, or LLM evaluation. Strong problem-solving...
Long Ridge Partners
New York, NY3 days ago - ...AI Engineer (GenAI Workflows for OTC Derivatives) Location: New York City (SoHo) — In-Person... .... You’ll own key parts of the evaluation and context engineering infrastructure... ...ensuring model outputs are high-quality, reliable, and explainable. You’ll work closely...Full time
Nxt Level
New York, NY1 day ago $168.15k - $269.22k
...serves the world’s leading brands through a global... ...Architect - AI Systems and Experience Engineering Digitas is seeking... ..., performance, reliability, security, auditability... ...agentic patterns, evaluation approaches, and emerging... ..., and QA teams. ~ Track...Temporary workWork experience placementFreelanceFlexible hoursPublicis Groupe
New York, NY3 days ago- We are looking for a Lead Python AI Engineer to build robust, AI-powered API services and agentic... ...distributed system. If you take pride in API reliability, graceful failure handling, and... ...engineering foundation-baking in evaluation, comprehensive observability, robust...
- ...management, design, and engineering together with the... ...You are an exceptional AI-native software engineer... ..., establish tests and evaluations, and review generated... ...architecture, security, reliability, and quality of the... ...AccentureAccenture is a leading global professional services...Permanent employmentFull timeWork experience placementLive inWork at officeLocal area
Accenture
New York, NY6 days ago $123k - $215.25k
...bonus + benefitsJob Function: Engineering & ArchitectureSchedule: Full... ...ExpressDescriptionAt American Express, AI is reshaping the future of... ...security, explainability, reliability, and compliance required in... ...frameworks, and evaluation and monitoring tooling, that...Visa sponsorshipAmerican Express
New York, NY5 days ago$122k - $240.5k
...Summary Agentic AI is moving from... ...'re growing a team of engineers who want to work at the... ...relationships Ability to lead projects or workstreamsAbility... ..., state management, evaluation, guardrails, and human... ...the performance, reliability, and usability of AI solutions...Work at officeLocal areaVisa sponsorshipShift workDeloitte
New York, NY4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead AI QA Engineer - Evaluation & Reliability. Be the first to apply!
Related searches
- lead solutions engineer New York, NY
- lead infrastructure engineer New York, NY
- lead operating engineer New York, NY
- lead automation engineer New York, NY
- lead algorithm engineer New York, NY
- lead quality engineer New York, NY
- lead web developer New York, NY
- lead security engineer New York, NY
- lead product engineer New York, NY
- lead network engineer New York, NY