Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior AI Agent Engineer - Open Models & Evaluation Systems

Sail Research

Sail is the foundation of useful, agentic AI. We are here to take a big swing at the most ambitious engineering challenge of our careers. Everyone working at Sail will become an expert; nothing less will do in our immensely competitive market. Inference is just one piece of an effective background agent. Let's design and build the rest of the system, that turns billions of tokens into the best possible answers. What you’ll do Design custom evals for multi-turn, massively parallel agents. Build agent harnesses to improve open model (Deepseek, Qwen, Llama) performance. Claude Code is all about agent/harness codesign; let's do the same for open source! Automate prompt optimization techniques like DSPy. What we’re looking for Experience building AI agents. Familiarity with open source models. Interview process Meet the CEO. This is the first step because we respect your time. Ask any question and get a definitive answer immediately. Meet the CTO, who will ask about your experience, and share as much technical detail about Sail as you want to hear. Come in to Sail's SF office for an interview day. Meet the whole team, then you'll have 3-4 hours to work on a problem that closely simulates the work we do daily. It's an objectively scored task, so you'll have immediate feedback on how well your code is working - just like we do in production! AI assistance is highly encouraged, and we'll provide a laptop with all the best tools set up. Finish with a short presentation describing your process, learnings, and results. Offer. Once the team decides we want to work with you, we make a strong offer quickly and will be quite persistent over email/text/calls :) Life at Sail We work out of a beautiful, sunny office in downtown San Francisco. All meals are on us (and actually great; SF is a food paradise and it would be a shame to eat only bowl slop). Everyone gets a Studio Display at their desk. We are serious about investing in anything that saves us time or energy. There are six different ways to make coffee or tea in the office. A friendly (hypoallergenic) black cat named Coco visits occasionally. #J-18808-Ljbffr Sail Research

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior AI Agent Engineer - Open Models & Evaluation Systems in San Francisco, CA vacancy
  • B Capital seeks a talented individual for an AI Evaluation role in San Francisco. This position involves conducting...  ...comparative analysis, refining evaluation systems, and collaborating with various teams to enhance model capabilities. The ideal candidate will have strong... 
    Suggested

    B Capital

    San Francisco, CA
    10 hours ago
  • Cline is building open-source AI agents for developers and power users. We seek an experienced engineer to design and implement the agent systems behind the platform—runtime, orchestration, tools,...  ...across teams, and contribute to a model-agnostic, open-weight capable agent... 
    Senior

    Cline

    San Francisco, CA
    4 days ago
  •  ...caring nurses while AI agents handle the tedious...  ...the healthcare system on your behalf, and...  ...looking for an AI engineer to build the LLM-powered...  ...to finish, the evaluation infrastructure...  ...prompts, retrieval, and models. Make quality a...  ...Passionate and open. You exhibit enthusiasm... 
    Senior
    Full time

    Mira Mace

    San Francisco, CA
    19 hours ago
  •  ...and optimizing features for an AI runtime and SaaS platform. The...  ...years of experience in backend systems, proficiency in Python and C++,...  ...teams. You will contribute to open-source initiatives and help shape...  ...position offers a hybrid working model with a hands-on approach to AI... 
    Senior

    Valid8 Financial, Inc.

    San Francisco, CA
    1 day ago
  • Sail is advancing AI with a focus on agentic systems and scalable evaluation. This role centers on designing custom evals for multi-turn, massively parallel agents and building harnesses to improve open models. You’ll also help automate prompt optimization to push performance... 
    Suggested

    SAIL

    San Francisco, CA
    2 days ago
  • AI Systems Engineer - Codex Core Agents Location San Francisco Employment Type Full time Department...  ...Codex agents interpret model outputs, use tools,...  ...development environments. Develop evaluation, experimentation, and...  ...for other teams and open-source users to build on.... 
    Full time
    Work at office
    Local area
    Relocation package
    Flexible hours

    Slope

    San Francisco, CA
    1 day ago
  • $166.9k - $225.9k

     ...thoughtful hybrid model because we believe...  ...intelligent AI capabilities into...  ...We are seeking a Senior AI Engineer to help design, build...  ...robust, high-impact AI systems that improve...  ...LLM + retrieval + agent systems in production...  ...reliability, and evaluation in real-world enterprise... 
    Senior
    Full time
    Work at office
    Immediate start
    Worldwide
    Monday to Friday
    Flexible hours

    Drata

    San Francisco, CA
    19 hours ago
  • $200k - $300k

     ...startup building AI “coworkers” for IT...  ...product in which an agent operates as a...  ..., and can operate systems it was never handed...  ...one of the earliest engineers alongside a small...  ...layer that turns raw model capability into...  ...experience across model evaluation, fine-tuning, and/... 
    Senior
    Full time
    Contract work
    H1b
    Work at office
    Visa sponsorship
    Monday to Friday

    David Joseph & Company

    San Francisco, CA
    3 days ago
  • $139k - $257.55k

     ..., reusable design systems, and collaboration...  ...next generation of AI-native creative...  ...both users and AI agents. Our mission is to...  ...forward-thinking engineers who are excited to...  ...experience.Develop evaluation and quality frameworks...  ...this role is open to hiring in Colorado... 
    Senior
    Full time
    Temporary work
    Local area
    Worldwide

    Adobe Systems

    San Francisco, CA
    19 hours ago
  • Arcadia is looking for a Senior Software Engineer (AI Agents) in San Francisco. In this pivotal role, you will build core systems for an intelligent automation platform used in operations-heavy industries. Your responsibilities will include designing production-grade LLM... 
    Senior

    Arcadia

    San Francisco, CA
    1 day ago
  • Cacheflow is seeking a Senior Applied Research Engineer to enhance the effectiveness of our AI systems through focused research and experimentation. This role involves designing information retrieval strategies and collaborating with engineers to turn validated approaches... 
    Senior
    Flexible hours

    Cacheflow

    San Francisco, CA
    10 hours ago
  • $240k - $280k

     ...software monitoring company is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure for measuring the performance of AI systems. This role involves designing...  ...The position offers a hybrid work model and a salary range of $240,000 to $... 
    Senior

    Sentry

    San Francisco, CA
    19 hours ago
  •  ...building the next generation of AI agents for the insurance industry -...  ...for an exceptional AI Engineer to join our agent team and help...  ...techniques (e.g. multi-agent systems, reasoning-from-feedback, RFT...  ...Note that this is not a model-training role - you’ll be building... 
    Senior
    Full time
    Work at office

    Further AI

    San Francisco, CA
    19 hours ago
  • $200k - $400k

     ...Job Title: Founding AI Engineer Location: San Francisco, CA (On-site...  ...design, build, and launch a multi-agent AI system that will redefine the future of talent evaluation. You’ll join at a pivotal...  ...DNA — hackathon wins, notable open-source contributions, or personal... 
    Senior
    Full time
    Immediate start
    Relocation
    Visa sponsorship
    Relocation package
    Flexible hours

    Heaven and Hire Recruiting

    San Francisco, CA
    more than 2 months ago
  • $190.2k - $360.5k

     ...looking for a Principal AI Systems Engineer with deep C++...  ...integration layers, evaluation systems, and reliability...  ...-powered services, agents, and model-backed workflows. Define...  ...teams. Mentor senior engineers and raise...  ...NoticeIf this role is open to hiring in Colorado... 
    Full time
    Temporary work
    Local area
    Remote work
    Worldwide

    Adobe Systems

    San Francisco, CA
    2 days ago
  • $163k - $246.5k

     ...as you build, with AI that learns your context...  ...role As an AI engineer, you’ll apply LLM...  ...and rigorously evaluate the efficacy of different prompts and models through experimentation...  ...provided you are open to learning them quickly...  ...employee with a system that equally... 
    Senior
    Full time
    Currently hiring
    Local area
    Remote work
    Weekend work
    3 days per week

    Semgrep, Inc.

    San Francisco, CA
    19 hours ago
  • $314.8k - $359.3k

    {"description": "Senior Distinguished AI Engineer At Capital One...  ...responsible and reliable AI systems, changing banking...  ...customers. Our AI models and platforms...  ..., guardrails, model evaluation, experimentation, governance...  ...a broad stack of Open Source and SaaS AI... 
    Senior
    Full time
    Part time
    Local area

    Capital One Financial Corporation

    San Francisco, CA
    19 hours ago
  •  ...scale autonomous AI agents with full...  ...looking for a Senior Applied AI Engineer to join our on-...  ...improving production systems . You will work...  ...including agent evaluation, observability,...  ...maintain pipelines for model evaluation,...  ...LLMs (APIs or open-source models) in... 
    Senior
    Full time

    Alterion, Inc.

    San Francisco, CA
    19 hours ago
  •  ...part of Schwab’s AI Strategy &...  ...together product, engineering, strategy, and...  ...We’re seeking a Senior AI Engineer who...  ...ensure that the systems you build are...  ...leveraging AI models for measurable...  ...proprietary or open-source LLMs (e....  ...writing and running evaluations for quality and... 
    Senior
    Full time
    Work at office

    The Charles Schwab Corporation

    San Francisco, CA
    2 days ago
  • $190.9k - $334.1k

     ...DescriptionIt all started when engineer Fred Luddy wrote...  ...ServiceNow is the AI control tower for...  .... Develop novel AI Agent frameworks,...  ...customer enterprise systems including ServiceNow...  ...engineering, and data modeling. Curiosity &...  ...ConsultingExperience level: Mid-Senior LevelIndustry:... 
    Senior
    Work at office
    Immediate start
    Remote work
    Flexible hours
    Day shift

    ServiceNow

    San Francisco, CA
    1 day ago
  • $2,000 per month

     ...is the leading AI analytics...  ...With powerful AI Agents embedded across...  ...looking for a Senior AI Engineer to help advance...  ..., build the systems that deliver it...  ...orchestration, evaluation, retrieval,...  ...AI ecosystem: model APIs, agent frameworks...  ...empathy and openness, seeking to... 
    Senior
    Home office

    Amplitude

    San Francisco, CA
    3 days ago
  • $128k - $252.5k

     ...covering valuation modeling, cost...  ...scientists to AI strategists, machine...  ...specialists, and data engineers. SFL Scientific...  ...to add a Senior AI Engineer to...  ...cloud/enterprise/open-source solutions...  ..., autonomous systems and edge AI, and...  ...understand and evaluate new and essential... 
    Senior
    Local area
    Visa sponsorship

    Deloitte

    San Francisco, CA
    4 days ago
  •  ...building intelligent systems that sit at the...  ...of AI, biology, chemistry...  ...and large-scale engineering. Our goal is to...  ...The Role As a Senior AI/ML Engineer,...  ...learning models that form the core...  ...Design, train, and evaluate large-scale...  ...Contributions to open-source ML frameworks... 
    Senior
    Full time
    Remote work
    Flexible hours

    Absentia Labs

    San Francisco, CA
    19 hours ago
  • Senior AI Architect - Multi-Agent Systems & Platform Infrastructure Senior AI Architect - Multi...  ...Orchestration / Head of Engineering Seniority: Senior-Level (...  ...and refine test plans, evaluation pipelines, and debug tools...  ...LLMs • Contributions to open-source AI orchestration or... 
    Senior
    Full time
    Work at office
    Remote work

    Nivalto

    San Francisco, CA
    4 days ago
  • $200k - $250k

     ...building the AI Search Infrastructure...  ...modern AI systems. Our goal is...  ...layer that agents, applications...  ...language models, and large-scale...  ...includes engineers, researchers,...  ...vertical feeds, evaluation tooling, and...  ...credibility with senior engineers and...  ...footprint — open-source,... 
    Senior
    Full time
    Immediate start
    Remote work
    Work from home
    Flexible hours
    Early shift

    You.com

    San Francisco, CA
    19 hours ago
  • $180k - $215k

     ...flagship product—an AI-driven, non-...  ...-coded, 3D model of a patient’...  ...both the Senior and Staff levels...  ...product engineering. You won't just...  ...intelligent agents to execute multi...  ...agentic systems from scratch,...  ...guardrails, evaluation frameworks, and...  ...solving complex, open-ended... 
    Senior
    Local area
    Worldwide
    Relocation

    HeartFlow

    San Francisco, CA
    14 hours ago
  • $229k - $281k

    MissionAs a AI Accelerated Engineering Lead, you will guide...  ...where AI agents and engineers...  ...practices such as evaluation harnesses,...  ...leadership in system or platform architecture...  ...databases, or model routing...  ...to open source or thought...  ...San Francisco* Senior Principal: $22... 
    Senior
    Temporary work
    Local area
    Shift work

    Slalom

    San Francisco, CA
    2 days ago
  • $124k - $280k

     ...in data and analytics engineering focus on leveraging advanced...  ...implementing advanced AI and ML solutions to...  ...algorithms, models, and systems to enable intelligent...  ...staying relevant.Initiate open and honest coaching conversations...  ...with team members. We evaluate these factors... 
    Senior
    Full time
    H1b

    PwC

    San Francisco, CA
    19 hours ago
  • $293.5k

     ...Title Expert Senior Manager, AI Engineering Job ID 1043...  ...of an integrated system that combines technology...  ..., operating model changes, and...  ...including multi-agent workflows and production...  ..., and evaluation design to production...  ...job postings for open roles. The estimated... 
    Senior
    Permanent employment
    Full time
    Apprenticeship
    Work at office
    Local area
    Work from home
    Home office
    3 days per week

    Bain & Company

    San Francisco, CA
    2 days ago
  •  ...AI Systems EngineerTransluce is a fast-moving research...  ...industry standards for evaluation. We are a non-profit with...  ...AI systems engineer to lead the design and...  ...organisational reach (open-source tools the entire...  ...enough to allow complex model introspection and intervention... 
    Flexible hours

    Transluce

    San Francisco, CA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior AI Agent Engineer - Open Models & Evaluation Systems. Be the first to apply!