Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior AI Agent Engineer - Open Models & Evaluation Systems

Sail Research

Sail is the foundation of useful, agentic AI. We are here to take a big swing at the most ambitious engineering challenge of our careers. Everyone working at Sail will become an expert; nothing less will do in our immensely competitive market. Inference is just one piece of an effective background agent. Let's design and build the rest of the system, that turns billions of tokens into the best possible answers. What you’ll do Design custom evals for multi-turn, massively parallel agents. Build agent harnesses to improve open model (Deepseek, Qwen, Llama) performance. Claude Code is all about agent/harness codesign; let's do the same for open source! Automate prompt optimization techniques like DSPy. What we’re looking for Experience building AI agents. Familiarity with open source models. Interview process Meet the CEO. This is the first step because we respect your time. Ask any question and get a definitive answer immediately. Meet the CTO, who will ask about your experience, and share as much technical detail about Sail as you want to hear. Come in to Sail's SF office for an interview day. Meet the whole team, then you'll have 3-4 hours to work on a problem that closely simulates the work we do daily. It's an objectively scored task, so you'll have immediate feedback on how well your code is working - just like we do in production! AI assistance is highly encouraged, and we'll provide a laptop with all the best tools set up. Finish with a short presentation describing your process, learnings, and results. Offer. Once the team decides we want to work with you, we make a strong offer quickly and will be quite persistent over email/text/calls :) Life at Sail We work out of a beautiful, sunny office in downtown San Francisco. All meals are on us (and actually great; SF is a food paradise and it would be a shame to eat only bowl slop). Everyone gets a Studio Display at their desk. We are serious about investing in anything that saves us time or energy. There are six different ways to make coffee or tea in the office. A friendly (hypoallergenic) black cat named Coco visits occasionally. #J-18808-Ljbffr Sail Research

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Senior AI Agent Engineer - Open Models & Evaluation Systems in San Francisco, CA vacancy
  • B Capital seeks a talented individual for an AI Evaluation role in San Francisco. This position involves conducting...  ...comparative analysis, refining evaluation systems, and collaborating with various teams to enhance model capabilities. The ideal candidate will have strong... 
    Suggested

    B Capital

    San Francisco, CA
    3 days ago
  •  ...caring nurses while AI agents handle the tedious...  ...the healthcare system on your behalf, and...  ...looking for an AI engineer to build the LLM-powered...  ...to finish, the evaluation infrastructure...  ...prompts, retrieval, and models. Make quality a...  ...Passionate and open. You exhibit enthusiasm... 
    Senior
    Full time

    Mira Mace

    San Francisco, CA
    23 hours ago
  •  ...and optimizing features for an AI runtime and SaaS platform. The...  ...years of experience in backend systems, proficiency in Python and C++,...  ...teams. You will contribute to open-source initiatives and help shape...  ...position offers a hybrid working model with a hands-on approach to AI... 
    Senior

    Valid8 Financial, Inc.

    San Francisco, CA
    4 days ago
  • Sail is advancing AI with a focus on agentic systems and scalable evaluation. This role centers on designing custom evals for multi-turn, massively parallel agents and building harnesses to improve open models. You’ll also help automate prompt optimization to push performance... 
    Suggested

    SAIL

    San Francisco, CA
    23 hours ago
  • $180k - $220k

     ...backed healthcare AI company...  ...advocate while AI agents handle the tedious...  ...the agentic systems that automate healthcare...  ...Design evaluation infrastructure...  ...train and improve models over time...  ...experience in AI/ML engineering, including systems...  ...sponsorship: open to visa... 
    Senior
    Full time
    H1b
    Remote work
    Relocation
    Visa sponsorship

    David Joseph & Company

    San Francisco, CA
    12 days ago
  • AI Systems Engineer - Codex Core Agents Location San Francisco Employment Type Full time Department...  ...Codex agents interpret model outputs, use tools,...  ...development environments. Develop evaluation, experimentation, and...  ...for other teams and open-source users to build on.... 
    Full time
    Work at office
    Local area
    Relocation package
    Flexible hours

    Slope

    San Francisco, CA
    4 days ago
  • $230k - $385k

    AI Systems Engineer - Codex Core Agents About The Team The Codex Core Agents team builds the agent harness that turns model capability into real-world action. We own...  ...quality. The harness is open source and...  ...models are trained and evaluated, making this one of the... 

    OpenAI

    San Francisco, CA
    23 hours ago
  • $102.5k - $213.4k

     ...our product-driven, AI-centric approach,...  ...designers, and software engineers enable our clients...  ...As a Senior Application Designer...  ...Work on distributed systems problems ranging from...  ...and extensibility models for platform use cases...  ...Generative AI and building Agents Ability to... 
    Senior
    Full time
    Summer holiday
    Flexible hours

    EY

    San Francisco, CA
    2 days ago
  • $250.8k - $286.2k

    Senior Lead AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) Overview: At Capital One, we are creating...  ...and reliable AI systems, changing banking for...  ...of customers. Our AI models and platforms...  ...Leverage a broad stack of Open Source and SaaS AI technologies... 
    Senior
    Full time
    Part time
    Local area

    Capital One

    San Francisco, CA
    5 days ago
  • $229.9k - $286.2k

    Senior Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) Overview : At Capital One, we are creating...  ...and reliable AI systems, changing banking...  ...of customers. Our AI models and platforms empower...  ...across a broad stack of Open Source and SaaS AI... 
    Senior
    Full time
    Part time
    Local area

    Capital One

    San Francisco, CA
    5 days ago
  • $139k - $257.55k

     ..., reusable design systems, and collaboration...  ...next generation of AI-native creative...  ...both users and AI agents. Our mission is to...  ...forward-thinking engineers who are excited to...  ...experience.Develop evaluation and quality frameworks...  ...this role is open to hiring in Colorado... 
    Senior
    Full time
    Temporary work
    Local area
    Worldwide

    Adobe Systems

    San Francisco, CA
    3 days ago
  • $197.3k - $225.1k

    Lead AI Engineer (Gen AI Platform Services: Agentic AI, Agent Guardrails, Agent Evaluation, Agent Memory) Overview At Capital...  ...responsible and reliable AI systems, changing banking...  ...customers. Our AI models and platforms...  ...Leverage a broad stack of Open Source and SaaS AI... 
    Full time
    Part time
    Local area

    Socket.dev

    San Francisco, CA
    3 days ago
  •  ...is partnering with an AI research initiative to...  ...support a Frontier Code Agents project in San Francisco. You will evaluate frontier AI coding models by performing realistic data engineering tasks, reviewing ETL pipelines...  ..., and distributed systems. Join a sprint-based... 

    Obsidian

    San Francisco, CA
    1 day ago
  • Innovaccer is seeking a Senior AI Engineer to join our AI team and build production-grade AI systems at scale. You will collaborate with product, engineering, and data teams...  ..., including LLM-based solutions, AI agents, and retrieval-augmented generation workflows.... 
    Senior

    Innovaccer

    San Francisco, CA
    23 hours ago
  • Arcadia is looking for a Senior Software Engineer (AI Agents) in San Francisco. In this pivotal role, you will build core systems for an intelligent automation platform used in operations-heavy industries. Your responsibilities will include designing production-grade LLM... 
    Senior

    Arcadia

    San Francisco, CA
    4 days ago
  • $240k - $280k

     ...software monitoring company is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure for measuring the performance of AI systems. This role involves designing...  ...The position offers a hybrid work model and a salary range of $240,000 to $... 
    Senior

    Sentry

    San Francisco, CA
    3 days ago
  •  ...is partnering with a leading AI research lab to support a Frontier Code Agents project. You will help evaluate frontier AI coding models by performing infrastructure engineering tasks and reviewing model-...  ...concepts to production-scale systems. #J-18808-Ljbffr Mercor Inc

    Mercor Inc

    San Francisco, CA
    3 days ago
  • Perplexity is seeking energetic engineers to join our highly driven Agents engineering team. The...  ...backend, full-stack, and AI/ML systems powering delightful...  ...and engineering to solve open problems in AI, including...  ...designing AI agents, training models, and delivering secure,... 

    Neura Market

    San Francisco, CA
    4 days ago
  •  ...building the next generation of AI agents for the insurance industry -...  ...for an exceptional AI Engineer to join our agent team and help...  ...techniques (e.g. multi-agent systems, reasoning-from-feedback, RFT...  ...Note that this is not a model-training role - you’ll be building... 
    Senior
    Full time
    Work at office

    Further AI

    San Francisco, CA
    23 hours ago
  • $175k - $240k

    Member of Technical Staff, Senior Applied AI Engineer, Agents Aug 25, 2026 $175,000 - $2...  ..., tool integrations, and evaluation frameworks. Develop reusable...  ...experience shipping systems that real users depend on....  ...applications. Contributions to open-source scientific or AI... 
    Senior
    Full time
    Work at office

    Edison Scientific

    San Francisco, CA
    1 day ago
  • $190.2k - $360.5k

     ...looking for a Principal AI Systems Engineer with deep C++...  ...integration layers, evaluation systems, and reliability...  ...-powered services, agents, and model-backed workflows. Define...  ...teams. Mentor senior engineers and raise...  ...NoticeIf this role is open to hiring in Colorado... 
    Full time
    Temporary work
    Local area
    Remote work
    Worldwide

    Adobe Systems

    San Francisco, CA
    23 hours ago
  •  ...scale autonomous AI agents with full...  ...looking for a Senior Applied AI Engineer to join our on-...  ...improving production systems . You will work...  ...including agent evaluation, observability,...  ...maintain pipelines for model evaluation,...  ...LLMs (APIs or open-source models) in... 
    Senior
    Full time

    Alterion, Inc.

    San Francisco, CA
    23 hours ago
  •  ...Roger is an AI platform that...  ...requires LLM systems that extract structure...  ...looking for a Senior Applied AI Engineer to build the...  ...fine-tuning models on our data, building...  ...and fine-tune open source models,...  ...-tuning, or evaluating LLMs and open...  ...where one agent's output drives... 
    Senior
    Full time
    Remote work
    Work from home

    Roger Healthcare

    San Francisco, CA
    23 hours ago
  • $204k - $259k

     ...U.S. states. The Planner Evaluation team works on one of the key...  ...experienced data-minded software engineers and data scientists to help...  ...to navigate complex systems and pursue open-ended problems to completion...  ...data analysis tools for rapid modeling and prototyping... 
    Senior
    Full time
    Remote work

    Waymo

    San Francisco, CA
    23 hours ago
  • $200k - $400k

    Overview Job Title: Founding AI Engineer Location: San Francisco, CA (On-site...  ..., build, and launch a multi-agent AI system that will redefine the future of talent evaluation. You’ll join at a pivotal growth...  ...DNA — hackathon wins, notable open-source contributions, or personal... 
    Senior
    Full time
    Immediate start
    Relocation
    Visa sponsorship
    Relocation package
    Flexible hours

    Heaven and Hire Recruiting

    San Francisco, CA
    4 days ago
  • $2,000 per month

     ...is the leading AI analytics...  ...With powerful AI Agents embedded across...  ...looking for a Senior AI Engineer to help advance...  ..., build the systems that deliver it...  ...orchestration, evaluation, retrieval,...  ...AI ecosystem: model APIs, agent frameworks...  ...empathy and openness, seeking to... 
    Senior
    Home office

    Amplitude

    San Francisco, CA
    1 day ago
  •  ...building intelligent systems that sit at the...  ...of AI, biology, chemistry...  ...and large-scale engineering. Our goal is to...  ...The Role As a Senior AI/ML Engineer,...  ...learning models that form the core...  ...Design, train, and evaluate large-scale...  ...Contributions to open-source ML frameworks... 
    Senior
    Full time
    Remote work
    Flexible hours

    Absentia Labs

    San Francisco, CA
    23 hours ago
  • $200k - $250k

     ...building the AI Search Infrastructure...  ...modern AI systems. Our goal is...  ...layer that agents, applications...  ...language models, and large-scale...  ...includes engineers, researchers,...  ...vertical feeds, evaluation tooling, and...  ...credibility with senior engineers and...  ...footprint — open-source,... 
    Senior
    Full time
    Immediate start
    Remote work
    Work from home
    Flexible hours
    Early shift

    You.com

    San Francisco, CA
    23 hours ago
  • $190.9k - $334.1k

     ...DescriptionIt all started when engineer Fred Luddy wrote...  ...ServiceNow is the AI control tower for...  .... Develop novel AI Agent frameworks,...  ...customer enterprise systems including ServiceNow...  ...engineering, and data modeling. Curiosity &...  ...ConsultingExperience level: Mid-Senior LevelIndustry:... 
    Senior
    Work at office
    Immediate start
    Remote work
    Flexible hours
    Day shift

    ServiceNow

    San Francisco, CA
    4 days ago
  • EY seeks a Lead AI Software Architect in the United States...  ...architects designing AI Agent Control systems. You’ll shape...  ...Infrastructure teams, drive open research, and mentor engineers through an AI-native design...  ...process. EY offers a hybrid model and strong rewards. #J-1... 
    Senior

    EY

    San Francisco, CA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior AI Agent Engineer - Open Models & Evaluation Systems. Be the first to apply!