Senior AI Quality Engineer — Agent Evaluation & Testing
$176k - $253kHarper
Harper is seeking a Senior Member of Technical Staff, AI Quality, in San Francisco. Your main goal will be to turn agent quality into quantifiable metrics, ensuring high standards through robust evaluation processes. You'll build capability regression evaluation suites, design grading systems, and work directly with engineers to ensure our AI systems excel. Ideal candidates have 3–6 years of software experience, particularly in LLM and agent evaluations. Competitive compensation includes a base salary of $176,000–$253,000, with equity options and benefits like meals and a gym membership. #J-18808-Ljbffr Harper
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Senior AI Quality Engineer — Agent Evaluation & Testing in San Francisco, CA vacancy
- ...support of caring nurses while AI agents handle the tedious backend... ...FOR We're looking for an AI engineer to build the LLM-powered... ...work autonomously, check the quality of what got done, and assist... ...work from start to finish, the evaluation infrastructure that tells us...SeniorFull time
Mira Mace
San Francisco, CA22 hours ago $166.9k - $225.9k
...automation by integrating intelligent AI capabilities into its trust platform. We are seeking a Senior AI Engineer to help design, build, and... ...scalable LLM + retrieval + agent systems in production... ...latency, cost, reliability, and evaluation in real-world enterprise workloads...SeniorFull timeWork at officeImmediate startWorldwideMonday to FridayFlexible hoursDrata
San Francisco, CA22 hours ago- ...company seeks language agnostic engineers who are able and... ...building a simulation and evaluation platform for AI agents, and 50% working on the framework... ...used to rigorously test, benchmark, and improve AI... ...performance, reliability, decision quality, and failure modes. Build...Senior
$163k - $246.5k
...gets smarter as you build, with AI that learns your context to... ...Gartner in Application Security Testing and is trusted by leading... ...About the role As an AI engineer, you’ll apply LLM technologies... ...powered solutions and rigorously evaluate the efficacy of different prompts...SeniorFull timeCurrently hiringLocal areaRemote workWeekend work3 days per weekSemgrep, Inc.
San Francisco, CA22 hours ago- ...SoFi’s Staff AI Engineer is a hands-on AI... ...This is a critical, senior role responsible for... ...scale. Distributed Agent Memory & State:... ...advanced tracing/testing frameworks that guarantee... ...production quality, compliance, and reliability... ..., LLMs, and NLP, evaluating and experimenting...SeniorFull time
Sofi
San Francisco, CA22 hours ago - ...at the intersection of AI, biology, chemistry, and large-scale engineering. Our goal is to translate... .... The Role As a Senior AI/ML Engineer, you will... ...Do Design, train, and evaluate large-scale models,... ...fundamentals: clean code, testing, reproducibility, and debugging...SeniorFull timeRemote workFlexible hours
Absentia Labs
San Francisco, CA22 hours ago $55k - $151.47k
...data and analytics engineering focus on... ...and to deliver quality. Embracing increased... ...the People Tech & AI team you will develop, test, and validate Generative AI agents and data pipelines... ...standards. As a Senior Associate you... ...- Executing LLM evaluation frameworks using...SeniorFull timeWork experience placementH1bRemote work$180k - $215k
...flagship product—an AI-driven, non-invasive cardiac test supported by the... ...roles across both the Senior and Staff levels.... ...greenfield product engineering. You won't just be... ...autonomous, intelligent agents to execute multi-... ...guardrails, evaluation frameworks, and reasoning...SeniorLocal areaWorldwideRelocation$229k - $281k
MissionAs a AI Accelerated Engineering Lead, you will guide... ...code, pressure-testing ideas with real... ...teams ship high quality software using clear... ...where AI agents and engineers work... ...practices such as evaluation harnesses, code-... ...San Francisco* Senior Principal: $229,...SeniorTemporary workLocal areaShift work$293.5k
...Information Job Title Expert Senior Manager, AI Engineering Job ID 104335... ...systems, including multi-agent workflows and production-... ...model experimentation, and evaluation design to production... ...GenAIOps processes for automated testing, regression evaluation,...SeniorPermanent employmentFull timeApprenticeshipWork at officeLocal areaWork from homeHome office3 days per week$175k - $215k
...equipping human experts with AI instruments to enable... ...seeking a hands-on Senior AI Engineer who can bridge the... ...experimentation and testing, all the way to production... ...Building automated evaluation systems for LLM-dependent... ...regression datasets, quality metrics, failure...SeniorFull timeWork at officeRemote workWeave
San Francisco, CA1 day ago$240k - $280k
A leading software monitoring company is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure for measuring the performance of AI systems. This role involves designing datasets, creating benchmarks, and ensuring AI features behave reliably...SeniorSentry
San Francisco, CA1 day ago- ...re building an agentic AI caregiver advocate... ...coordinates across multiple sub-agents to get things done. It... ...over time. The AI engineering challenge: build an... ...-use frameworks, and evaluation infrastructure that... ...good enough. Develop testing infrastructure for multi...SeniorImmediate startRemote workFlexible hours
Citizen Health
San Francisco, CA18 hours ago - ...Senior AI Engineer 1 Your Journey at Crowe Starts Here:... ...responsible for producing high-quality, well-architected... ...training and model evaluation workflows that... ...Implement advanced automated testing, including stress... ...consumption and building agents with multi-step...SeniorRemote workFlexible hours
Crowe
San Francisco, CA1 day ago $155k
...About the Team The Quality Engineering team builds the shared testing infrastructure and... ...are looking for a Senior Software Engineer,... ...in implementing how AI reshapes quality... ...layers that enable AI agents to validate the... ...Experience using or evaluating AI-powered engineering...SeniorFull timeContract workLocal areaHome officeFlexible hoursShift workEarly shiftScribd, Inc.
San Francisco, CA22 hours ago- ...Perplexity is seeking energetic engineers to join our highly driven Agents engineering team. The... ..., full-stack, and AI/ML engineers who collaborate... ...Ensure a high craft and quality bar, in both AI agent... ...reliability, code quality, AI evaluation, testing, and maintenance across...Full timeFlexible hours
Perplexity
San Francisco, CA1 day ago $124k - $171k
...comes next.Job Title:Senior AI Automation... ...Senior AI Automation Engineer accelerates the delivery... ...applications, agents, copilots, task assistants... ...agent workflows, evaluation, guardrails, human... ...implementing testing, evaluation,... ...improve efficiency, quality, and speed. Exercises...SeniorFull timeContract workFor contractors- ...has two products: - Kapwing AI: A prompt-first AI editing... ...based video editors. Design, test, and implement LLM powered... ...technologies and generative models to evaluate quality, revise prompts, and improve... ...of production-level software engineering experience in a modern web...Senior
Kapwing
San Francisco, CA23 days ago - ...is pharma-native, AI-driven... ...Role As a Senior AI/ML Engineer at Condor, you will... ...operation of the AI agents and machine learning... ...retrieval systems, evaluation harnesses, and serving... ..., designers, quality engineers, and... ...implement evaluation, testing, and guardrail...SeniorLive inImmediate start
Condor Software
San Francisco, CA21 hours ago - CoreWeave is hiring software engineers to build and scale Weave, the observability and evaluation platform for AI agents. In a small, growing team, you will design core functionality, ship features, and engage directly with users to improve the product experience. You will...Senior
- Notion is seeking a data-driven AI quality engineer to own the quality bar for Notion AI products. You’ll collaborate with product and engineering... ...the AI engineering team, focusing on context engineering, evaluation systems, and data analysis. You’ll partner with frontier...
- ...About Us At FurtherAI, we’re building the next generation of AI agents for the insurance industry - a trillion-dollar market ready for... ...experience. Now, we’re looking for an exceptional AI Engineer to join our agent team and help shape both our product....SeniorFull timeWork at office
Further AI
San Francisco, CA22 hours ago - ...Cooperidge Consulting Firm is seeking an AI Agent Software Engineer on behalf of a top AI-driven technology client building next-generation... ...Interest in tools that improve agent transparency through testing, visualization, and simulation ~ Full-time roles...Hourly payFull timeFlexible hoursWeekend work
Cooperidge Consulting Firm
San Francisco, CA22 hours ago - Responsibilities Build the AI Runtime Design and... ...and verify. Build evaluation frameworks that... ...improve quality. Implement permission... ...Product, Design, Sales Engineering, and Customer... ...Experiment with new agent architectures... ...APIs, concurrency, testing, and system design...Senior
$166.9k - $225.9k
...Drata is seeking a Senior Applied Research Engineer to drive the quality and effectiveness of our AI systems through rigorous... ...experimentation, evaluation, and applied research... ...across RAG, agents, and classic ML: chunking... ...dataset creation, A/B testing, statistical significance...SeniorFlexible hours- ...Engineering Hiring Sprint We're growing our engineering... ...API Engineers AI Agents Engineers Engineering... ..., routing, retrieval, evaluation, evidence generation,... ...continuously improve quality. Implement permission... ...including APIs, concurrency, testing, and system design....SeniorWork at officeLocal areaFlexible hours
Airbyte
San Francisco, CA3 hours ago $185k - $230k
...industry by building an AI platform for wealth... ...'re excited to hire a Senior QA Engineer to join our growing Engineering... ...role, you will own quality across our user-facing... ...exploratory manual testing and automation efforts... ...review cycles. By evaluating compensation on a regular...SeniorFull timeWork at officeImmediate startAltruist
San Francisco, CA1 day ago$115k - $188k
...AreHP IQ is HP’s new AI innovation lab.... ...world-class team—engineers, designers, researchers... ...with exceptional quality and developer... ...teams to strengthen test strategies, automation... ...updates for senior leadership.Communicate... ..., or model evaluation workflows.Understanding...SeniorFull timeTemporary workLocal areaFlexible hours$105.4k - $207.8k
...next generation of AI-enabled cyber... ...for a hands-on Data Engineer to build and operate... ...Work you'll doAs a Senior Consultant,... ...acceptance, control testing outcomes, audit evidence... ...Implementing data quality checks, lineage,... ...function calling, evaluation or monitoring, prompt...SeniorLocal areaVisa sponsorship- ...AI at Zapier At Zapier , we build and use automation every... ...Platform team as a Sr. Applied AI Engineer ! As a key member of this... ...are accessed, monitored, evaluated, deployed, governed, and operated... ...the full lifecycle of building, testing, deploying, and scaling ML/...SeniorRemote jobFull time
Zapier
San Francisco, CA22 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Quality Engineer — Agent Evaluation & Testing. Be the first to apply!
Related searches
- ai engineer San Francisco, CA
- ai research engineer San Francisco, CA
- machine learning ai engineer San Francisco, CA
- senior ai engineer San Francisco, CA
- ai ml engineer San Francisco, CA
- ai prompt engineer San Francisco, CA
- ai engineer remote San Francisco, CA
- ai developer San Francisco, CA
- program quality engineer San Francisco, CA
- senior quality systems engineer San Francisco, CA