Senior AI Quality Engineer — Agent Evaluation & Testing
$176k - $253kHarper
Harper is seeking a Senior Member of Technical Staff, AI Quality, in San Francisco. Your main goal will be to turn agent quality into quantifiable metrics, ensuring high standards through robust evaluation processes. You'll build capability regression evaluation suites, design grading systems, and work directly with engineers to ensure our AI systems excel. Ideal candidates have 3–6 years of software experience, particularly in LLM and agent evaluations. Competitive compensation includes a base salary of $176,000–$253,000, with equity options and benefits like meals and a gym membership. #J-18808-Ljbffr Harper
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Senior AI Quality Engineer — Agent Evaluation & Testing in San Francisco, CA vacancy
$135.2k - $306.4k
...Job Description The Senior Principal AI Agent / ML Software Engineer is a Senior Staff-level,... ..., memory, retrieval, evaluation, guardrails, and cloud services... ...eval suites, regression testing, experimentation, safety... ...to contribute high-quality production code, reviews...SeniorTemporary workFlexible hoursOracle
San Francisco, CA15 hours ago$96.8k - $306.4k
Job Description The Senior Principal AI Agent / ML Software Engineer is a Senior Staff-level, hands... ..., memory, retrieval, evaluation, guardrails, and cloud services... ...eval suites, regression testing, experimentation, safety... ...to contribute high‑quality production code, reviews...SeniorTemporary workFlexible hours- A leading tech company in San Francisco is seeking a Quality Engineer to develop high-performance testing infrastructure and ensure the quality of software products. The ideal candidate has strong CS fundamentals and over 5 years of experience in building testing software...Senior
- Sail is the foundation of useful, agentic AI. We are here to take a big swing at the most ambitious engineering challenge of our careers. Everyone working at Sail will... ...is just one piece of an effective background agent. Let's design and build the rest of the system,...SeniorWork at officeImmediate start
- Adobe Inc. in San Francisco is seeking a Senior Software Quality Engineer to establish standards for a new AI-first web product. You will build fast, reliable test systems for evolving AI workflows and ensure real-time performance, accessibility and cross-browser coverage...Senior
- ...company seeks language agnostic engineers who are able and... ...building a simulation and evaluation platform for AI agents, and 50% working on the framework... ...used to rigorously test, benchmark, and improve AI... ...performance, reliability, decision quality, and failure modes. Build...Senior
- Braven is agentic infrastructure for reinsurance. A Senior engineer on the Platform team you own the agent runtime and the evaluation system around it. Paid in USD. Remote, with in person hackathons and offsites. You will own the agent runtime and evaluation system, building...SeniorRemote job
- Role Title: Principal AI Engineer (LLM Agents & Orchestration) Focus: Building Autonomous "Super Agents" Who We Are... ...memory and context awareness across sessions. Evaluation & Observability: Establish a rigorous testing framework for non‑deterministic model outputs to...
- Adobe is seeking a Senior Software Quality Engineer to own the quality strategy for a new AI-first web application. You... ...and maintain modern web test automation, spanning the prompt/agent layer, in-browser experience... .... You will stand up evaluation harnesses, ensure...Senior
- Techire AI is seeking a Senior Software Engineer to design and build simulation environments for frontier AI. You will own end-to-end RL environments... .... You will collaborate with researchers to push evaluation quality, reliability, and safety of AI systems, shaping the...Senior
$240k - $360k
## AI Agent Abuse Prevention EngineerPostulerremote type... ...**Zendesk is hiring a Senior Staff-level technical... ...company-wide and engineering changes to prevent AI... ...be used to screen or evaluate applications for this... ...complete any pre-employment testing, or otherwise participate...Remote work$140k - $300k
This role owns the full Agent development lifecycle -... ...including orchestration engines, tool-use pipelines, memory... ..., monitoring) and evaluation pipelines covering personalization quality, memory accuracy, and multi... ...canary deployment, A/B testing, and production feedback...$240k - $280k
Sentry, located in San Francisco, is searching for a Senior Software Engineer to enhance its AI/ML team. In this role, you'll build the evaluation infrastructure for AI systems, ensuring they perform accurately and reliably. Responsibilities include designing datasets,...Senior$200k - $250k
Fieldguide, a remote-first company based in San Francisco, is seeking a Senior AI Engineer to lead the evaluation infrastructure for AI agents. This role focuses on enhancing the reliability and performance of AI systems to support top accounting and consulting firms....SeniorRemote jobFlexible hours$260k
...Title : Senior AI Product Engineer Location : San Francisco, CA... ...decisions about how LLMs, agents, and RAG pipelines... ...signal on AI output quality and feed it back to the... ...-stack integration testing Establish reusable... ...AI systems are built, evaluated, and deployed in high...SeniorHarnham
San Francisco, CA1 day ago- ...build core simulation engines, evaluation systems, and... ...improving conversational agents. You’ll work at... ...real-time voice, AI evaluation, and... ...shaping how agents are tested and improved. You’... ...with senior engineers on latency... ...barge-in, and audio quality challenges, while...
$240k - $280k
A leading software monitoring company is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure for measuring the performance of AI systems. This role involves designing datasets, creating benchmarks, and ensuring AI features behave reliably...Senior- Responsibilities Build the AI Runtime Design and... ...and verify. Build evaluation frameworks that... ...improve quality. Implement permission... ...Product, Design, Sales Engineering, and Customer... ...Experiment with new agent architectures... ...APIs, concurrency, testing, and system design...Senior
$160k - $207k
...gets smarter as you build, with AI that learns your context to... ...Gartner in Application Security Testing and is trusted by leading... ...dev. About the role As an AI engineer, you’ll apply LLM technologies... ...powered solutions and rigorously evaluate the efficacy of different prompts...SeniorCurrently hiringLocal areaRemote workWeekend work3 days per week$124k - $280k
...expertise, and network to deliver quality results. You motivate and... ...through innovative, AI-driven solutions. As a Senior Manager, you will lead... ...strategy, transformation and engineering projects and teams Design... ...closely with team members. We evaluate these factors thoughtfully...SeniorFull timeH1bPwC
San Francisco, CA3 hours ago$80 per hour
Prolific is seeking an AI & Machine Learning Engineer in San Francisco to evaluate and refine AI models. The role involves auditing ML code, providing human feedback on AI frameworks, and analyzing model reasoning. Candidates should have a relevant degree and experience...SeniorRemote jobHourly payWork from homeFlexible hours$250k - $375k
Senior Software Engineer, Agent Orchestration Decagon is a leading conversational AI platform that empowers every brand to... ...coordinating model reasoning, evaluating agent behavior, and... ..., reliability, and quality. In this role, you... ...through better testing, observability, and...SeniorFull timeWork at office$113.4k - $221.75k
...a new‑generation, AI‑first web application... ...from scratch, and quality engineering is a core founding... ...focus. We seek a Senior Software Quality... ...creating fast, reliable test systems for... ...spanning the prompt and agent layer, the in‑... .... Stand up evaluation harnesses for automated...SeniorShift workAdobe
San Francisco, CA3 hours ago$200k - $400k
...conversational AI platform empowering... ...to deploy AI agents that power... ...frontier‑style engineering. The team continuously... ...offline evaluation and online experimentation... ...to improve quality, reliability,... ...the Role As a Senior Software... ...experimentation (A/B testing) and contribute...SeniorFull timeWork at officeLocal area$204k - $259k
...environments crucial for testing and training the... ...behavior of diverse agents (vehicles, pedestrians... ...we create metrics and evaluation methodologies to measure... ...assessing simulator quality, identifying critical... ...and reports to a Senior Engineering Manager. You will:...Senior$200k - $300k
...Full time Department AI Compensation $200K -... ...is seeking an energetic engineer to join our highly driven Comet Agents engineering team. The Comet... ...a high craft and quality bar, in both AI agent performance... ..., code quality, AI evaluation, testing, and maintenance across...Full timeFlexible hours$170k - $200k
...Opportunity We're hiring a Senior Software Engineer to drive the... ...Nightshift , a conversational agent that helps... ...Experience with LLM evaluation (LangSmith/Langfuse),... ...and logging for the AI system to monitor agentic... ...Establish best practices for testing and deploying AI...SeniorWork at officeWork from homeHome officeFlexible hoursNight shift$180k - $215k
Staff/Senior Agentic AI Engineers (Multiple roles) Heartflow is a medical technology... ..., non‑invasive cardiac test supported by the ACC/AHA Chest... ...autonomous, intelligent agents to execute multi‑step medical... ...Implement advanced guardrails, evaluation frameworks, and reasoning...SeniorLocal areaWorldwideRelocation$80 per hour
Overview AI & Machine Learning Engineer - AI Training at Prolific. Prolific is building the biggest pool of quality human data in the world. Over 35,000 AI... ...skills. Responsibilities Evaluate LLM architecture logic:... ...performance: conduct comparative testing between different model...SeniorHourly payWork from homeFlexible hours- Skyrocket Ventures is looking for engineers who are proficient in Python and willing to learn new technologies. The role focuses on building and improving simulation and evaluation platforms for AI agents in San Francisco. Successful candidates will have 5-10 years of...Senior
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Quality Engineer — Agent Evaluation & Testing. Be the first to apply!
Related searches
- ai engineer San Francisco, CA
- ai ml engineer San Francisco, CA
- ai research engineer San Francisco, CA
- senior ai engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- ai developer San Francisco, CA
- ai prompt engineer San Francisco, CA
- ai engineer remote San Francisco, CA
- electronics quality engineer San Francisco, CA
- quality assurance quality control engineer San Francisco, CA