Applied AI Engineer, Evals
Full-time
Sable
Responsibilities
- Design evaluations for voice, vision, and browser use to guide improvements in the agent, context, and models.
- Build and maintain internal benchmarks and simulations.
- Develop systems to score customer-specific scenarios and identify gaps in current capabilities.
- Use evaluation findings to optimize the agent, context, and models.
Requirements
- Engineering experience with statistics and/or applied machine learning.
- Comfort working in a production codebase.
- Prior experience with evaluations, voice, realtime systems, or browser/computer-use agents is a bonus.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Applied AI Engineer, Evals in San Francisco, CA vacancy
- ...CTO & co-founder at Magic Patterns! Our engineering team has grown a lot over the last few months... ...directly to the product. If you’re an AI engineer, our roadmap is probably a... ...thousands of users use religiously everyday Evals with a self-improving agent loop...SuggestedFull time
Magic Patterns
San Francisco, CA1 day ago - ...About the Role Roger is an AI platform that frees home health clinicians from paperwork... .... We are now looking for a Senior Applied AI Engineer to build the intelligence layer at the... ...you fixed it. ~ Strong instincts for evals, observability, and the feedback loops that...SuggestedFull timeRemote workWork from home
Roger Healthcare
San Francisco, CA1 day ago - ...the real world. A modern, AI-native platform for physical product... ...goal of this job: you'll own applied AI across Doss: Dossbot (query... ...operational data model, evals, guardrails, and cost/latency... ...chain, finance, and ops data. Engineer AI-assisted tooling that uses...SuggestedFull timeWork at officeImmediate startRelocation packageFlexible hours
Doss
San Francisco, CA1 day ago $266k - $362k
About the teamThe Applied AI Engineering team is responsible for helping developers and enterprises safely and effectively deploy OpenAI technologies... ...AI workflows with retrieval, structured outputs, tool use, evals, guardrails, telemetry, sandboxing, and human-in-the-loop...SuggestedWork at officeLocal areaRelocation packageFlexible hours- ...behavior over time. About the Role We’re looking for applied AI engineers to help bring Codex agents from impressive demos to... ...workflows. Work closely with research to develop and run evals to measure agent performance, regressions, failure modes, and...SuggestedFull time
OpenAI
San Francisco, CA1 day ago - ...hiring an ambitious, self-initiated, product-minded Principal Applied AI Engineer to build the agentic workflows that power Redesign's most... ...agentic systems well beyond basic RAG, and have wrestled with the evals, context engineering, and failure modes that emerge when...
Redesign Health
San Francisco, CA1 day ago $155k - $300k
...In 2025, we started Handshake AI and built the fastest-growing... ...institutions Work together with engineers, scientists, operators, and... .... About the Role As an Applied AI Engineer at Handshake AI Enterprise... ...deploying agents, designing evals, and iterating until...Full timeWork at officeRemote workFlexible hoursHandshake
San Francisco, CA17 days ago- ...quickly. We're looking for product-minded engineers to build the AI experiences behind Mintlify. You'll... ...strong full-stack engineering with applied AI expertise to create agents that reliably... ...evaluation and iteration: Defining evals, build feedback loops, analyze...Immediate start
- ...the platform for evaluating how AI models perform in the real... ...their models, run and deliver evals, and act as their go-to person... ...'ll take on specific customer engineering asks and ship solutions that push... ...scale into our Data or Applied ML stack — but it starts with...Permanent employmentWork at office
- ...single record and served to humans and AI agents alike. This role sits on a small... ...there are billions of them. The Role As an Applied AI Engineer on this team, you build the multi-step... ...Continuous AI feedback loops : traces, evals, and observability are built in from day...Remote workShift work
- We're partnering with a fast-growing AI company building production-grade conversational... ...industry. They're hiring a Senior Applied AI Engineer to take ownership of the intelligence layer... ...and orchestration Experience building evals, running experiments, and improving models...
- ...Confidential client — a Pre-seed startup (1-10 employees) building AI coworkers for IT teams: a security-focused product where an agent... ...decides at runtime and what ships as code. You'll build and run evals against replicas of real customer environments rather than...Full timeContract workH1bVisa sponsorship
Thomas Talent Network
San Francisco, CA24 days ago - ...visa to move to the US. About Clera : Clera is a fast-growing AI start-up reimagining how top talent finds their dream jobs. We’... ...- and opportunity. We are looking for an exceptional founding engineer. You will work directly with our CTO & Co-Founder Daniel and...Full timeVisa sponsorship
Clera
San Francisco, CA1 day ago - ...About the Team GTM Growth Engineering builds AI-native products that help OpenAI's go-to-market and B2B marketing organizations scale with greater... ...speed, intelligence, and operational effectiveness. We apply OpenAI models to real business workflows and build the...Full time
OpenAI
San Francisco, CA1 day ago - ...Document Intelligence is the horizontal engine that turns that raw material — any file a... ...trustworthy maintenance knowledge, and into the AI-native entities and answers built on top... .... Minimum Requirements: Strong applied GenAI craft - prompt engineering, structured...
Maintainx
San Francisco, CA1 day ago - ...organizations to safely deploy, monitor, and scale autonomous AI agents with full visibility and governance. Our platform... ...dynamics. Role Overview We are looking for a Senior Applied AI Engineer to join our on-site Research & Intelligence team in San Francisco...Full time
Alterion, Inc.
San Francisco, CA1 day ago $120k - $160k
...Simbe is building the AI powered operating system for physical retail. Our autonomous robots and multimodal computer vision... ...team productivity. Simbe is looking for a Customer Facing Applied AI Engineer to help turn our computer vision and data platform into customer...Full timeSimbe Robotics Inc
San Francisco, CA1 day ago- ...The Role As an Applied AI Engineer, you'll bring the frontier of AI research and engineering to Rowspace, transforming how our platform understands and surfaces critical financial insights. In a typical week, you might build and productionize an agent in response to customer...Full timeWork at office
Rowspace
San Francisco, CA1 day ago - ...About Arcade Arcade is building the world's first AI physical product creation platform, where imagination becomes reality.... ...building fast. Overview of Role We're looking for a Staff Applied AI Engineer to architect the ML systems at the core of Arcade's product....Full time
Arcade
San Francisco, CA1 day ago - ...About Orum Orum ’s AI-powered suite frees salespeople to do what they do best: connect... ...AI and ML best practices across the engineering org Partner with Product and Engineering... ...Mentor engineers and raise the bar on applied AI engineering practices What You...Remote jobFull time
Orum
San Francisco, CA1 day ago $211k - $235k
About AlembicAlembic is an applied science company building GPU-resident distributed data systems that deliver 10-100x performance... ...impossible.The RoleWe're hiring a Senior Software Engineer onto our Applied AI team to build and extend the backend systems that power our...Work at office$125k - $186k
...future of logistics but building what comes next.Job Title:Senior AI Applied EngineerCompany:PrologisA day in the lifeIn this role, you... ..., observable, cost-aware architectures and strong software engineering practices.• Communicate results through demos and concise readouts...Full time$251k - $278k
About the TeamOpenAI’s Applied AI Engineering team helps organizations turn frontier AI capabilities into safe, reliable, and high-impact production systems. We work with customer executives, product and engineeriIng teams, security leaders, and transformation teams to...Work at officeLocal areaRelocation packageFlexible hours$152k - $271k
...work.Document Intelligence is the horizontal engine that turns that raw material — any file a... ...maintenance knowledge, and into the AI-native entities and answers built on top of... ...Intelligence.Minimum Requirements:Strong applied GenAI craft - prompt engineering, structured...$215k - $260k
...intelligence. As the only vertically integrated AI infrastructure company built from the... ...models in use today.The work is applied, not academic. The optimizations you build... ...you will also work directly with customer engineering teams to tailor deployments to their needs...Temporary work$197k - $278k
About the teamThe Partner Applied AI Engineer team ensures the safe and effective deployment of Generative AI applications for developers and enterprises. We act as trusted advisors and technical partners to our customers, helping them build and execute their AI adoption...Work at officeLocal areaRelocation packageFlexible hours$302k - $335k
About the TeamThe Applied AI Engineering team partners closely with customers to help them move from experimentation to production with OpenAI’s technologies. We act as trusted technical advisors, working across customer strategy, architecture, deployment, and adoption...Work at officeLocal areaFlexible hours$302k - $335k
About the TeamThe Applied AI Engineering team is responsible for helping customers turn frontier AI capabilities into real products, workflows, and business impact. We act as trusted technical partners across solution design, architecture, implementation, evaluation, and...Work at officeLocal areaRelocation packageFlexible hours$10k
...About the Role The Financial Intelligence team, part of Ramp's Applied AI org, powers the agentic analysis, FP&A, and book-close... ...artifacts on top of. This is a frontend-leaning role for an engineer who obsesses over interaction design and data-dense UX, but is...Full timeWork at officeHome officeRelocation packageFlexible hoursParibus (Ramp)
San Francisco, CA4 days ago- ...Intelligence is building the future of vision AI: pairing a world-first sensor that sees... ...Model—the most powerful intelligence engine for the physical world. This system doesn'... ...Documentation Build with security first: apply proper secret management, access control,...Permanent employmentShift work
SupportFinity
San Francisco, CA2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Applied AI Engineer, Evals. Be the first to apply!
Related searches
- hydraulic application engineer San Francisco, CA
- application system engineer San Francisco, CA
- junior application support engineer San Francisco, CA
- application engineer San Francisco, CA
- application performance engineer San Francisco, CA
- network applications engineer San Francisco, CA
- project application engineer San Francisco, CA
- application support engineer San Francisco, CA
- application operations engineer San Francisco, CA
- senior application support engineer San Francisco, CA