Senior AI Agent Engineer - Open Models & Evaluation Systems
Sail Research
Sail is the foundation of useful, agentic AI. We are here to take a big swing at the most ambitious engineering challenge of our careers. Everyone working at Sail will become an expert; nothing less will do in our immensely competitive market. Inference is just one piece of an effective background agent. Let's design and build the rest of the system, that turns billions of tokens into the best possible answers. What you’ll do Design custom evals for multi-turn, massively parallel agents. Build agent harnesses to improve open model (Deepseek, Qwen, Llama) performance. Claude Code is all about agent/harness codesign; let's do the same for open source! Automate prompt optimization techniques like DSPy. What we’re looking for Experience building AI agents. Familiarity with open source models. Interview process Meet the CEO. This is the first step because we respect your time. Ask any question and get a definitive answer immediately. Meet the CTO, who will ask about your experience, and share as much technical detail about Sail as you want to hear. Come in to Sail's SF office for an interview day. Meet the whole team, then you'll have 3-4 hours to work on a problem that closely simulates the work we do daily. It's an objectively scored task, so you'll have immediate feedback on how well your code is working - just like we do in production! AI assistance is highly encouraged, and we'll provide a laptop with all the best tools set up. Finish with a short presentation describing your process, learnings, and results. Offer. Once the team decides we want to work with you, we make a strong offer quickly and will be quite persistent over email/text/calls :) Life at Sail We work out of a beautiful, sunny office in downtown San Francisco. All meals are on us (and actually great; SF is a food paradise and it would be a shame to eat only bowl slop). Everyone gets a Studio Display at their desk. We are serious about investing in anything that saves us time or energy. There are six different ways to make coffee or tea in the office. A friendly (hypoallergenic) black cat named Coco visits occasionally. #J-18808-Ljbffr Sail Research
- B Capital seeks a talented individual for an AI Evaluation role in San Francisco. This position involves conducting... ...comparative analysis, refining evaluation systems, and collaborating with various teams to enhance model capabilities. The ideal candidate will have strong...Suggested
- ...caring nurses while AI agents handle the tedious... ...the healthcare system on your behalf, and... ...looking for an AI engineer to build the LLM-powered... ...to finish, the evaluation infrastructure... ...prompts, retrieval, and models. Make quality a... ...Passionate and open. You exhibit enthusiasm...SeniorFull time
- ...and optimizing features for an AI runtime and SaaS platform. The... ...years of experience in backend systems, proficiency in Python and C++,... ...teams. You will contribute to open-source initiatives and help shape... ...position offers a hybrid working model with a hands-on approach to AI...Senior
- Sail is advancing AI with a focus on agentic systems and scalable evaluation. This role centers on designing custom evals for multi-turn, massively parallel agents and building harnesses to improve open models. You’ll also help automate prompt optimization to push performance...Suggested
$180k - $220k
...backed healthcare AI company... ...advocate while AI agents handle the tedious... ...the agentic systems that automate healthcare... ...Design evaluation infrastructure... ...train and improve models over time... ...experience in AI/ML engineering, including systems... ...sponsorship: open to visa...SeniorFull timeH1bRemote workRelocationVisa sponsorship- AI Systems Engineer - Codex Core Agents Location San Francisco Employment Type Full time Department... ...Codex agents interpret model outputs, use tools,... ...development environments. Develop evaluation, experimentation, and... ...for other teams and open-source users to build on....Full timeWork at officeLocal areaRelocation packageFlexible hours
$230k - $385k
AI Systems Engineer - Codex Core Agents About The Team The Codex Core Agents team builds the agent harness that turns model capability into real-world action. We own... ...quality. The harness is open source and... ...models are trained and evaluated, making this one of the...$102.5k - $213.4k
...our product-driven, AI-centric approach,... ...designers, and software engineers enable our clients... ...As a Senior Application Designer... ...Work on distributed systems problems ranging from... ...and extensibility models for platform use cases... ...Generative AI and building Agents Ability to...SeniorFull timeSummer holidayFlexible hours$250.8k - $286.2k
Senior Lead AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) Overview: At Capital One, we are creating... ...and reliable AI systems, changing banking for... ...of customers. Our AI models and platforms... ...Leverage a broad stack of Open Source and SaaS AI technologies...SeniorFull timePart timeLocal area$229.9k - $286.2k
Senior Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) Overview : At Capital One, we are creating... ...and reliable AI systems, changing banking... ...of customers. Our AI models and platforms empower... ...across a broad stack of Open Source and SaaS AI...SeniorFull timePart timeLocal area$139k - $257.55k
..., reusable design systems, and collaboration... ...next generation of AI-native creative... ...both users and AI agents. Our mission is to... ...forward-thinking engineers who are excited to... ...experience.Develop evaluation and quality frameworks... ...this role is open to hiring in Colorado...SeniorFull timeTemporary workLocal areaWorldwide$197.3k - $225.1k
Lead AI Engineer (Gen AI Platform Services: Agentic AI, Agent Guardrails, Agent Evaluation, Agent Memory) Overview At Capital... ...responsible and reliable AI systems, changing banking... ...customers. Our AI models and platforms... ...Leverage a broad stack of Open Source and SaaS AI...Full timePart timeLocal area- ...is partnering with an AI research initiative to... ...support a Frontier Code Agents project in San Francisco. You will evaluate frontier AI coding models by performing realistic data engineering tasks, reviewing ETL pipelines... ..., and distributed systems. Join a sprint-based...
- Innovaccer is seeking a Senior AI Engineer to join our AI team and build production-grade AI systems at scale. You will collaborate with product, engineering, and data teams... ..., including LLM-based solutions, AI agents, and retrieval-augmented generation workflows....Senior
- Arcadia is looking for a Senior Software Engineer (AI Agents) in San Francisco. In this pivotal role, you will build core systems for an intelligent automation platform used in operations-heavy industries. Your responsibilities will include designing production-grade LLM...Senior
$240k - $280k
...software monitoring company is seeking a Senior Software Engineer on its AI/ML team to build evaluation infrastructure for measuring the performance of AI systems. This role involves designing... ...The position offers a hybrid work model and a salary range of $240,000 to $...Senior- ...is partnering with a leading AI research lab to support a Frontier Code Agents project. You will help evaluate frontier AI coding models by performing infrastructure engineering tasks and reviewing model-... ...concepts to production-scale systems. #J-18808-Ljbffr Mercor Inc
- Perplexity is seeking energetic engineers to join our highly driven Agents engineering team. The... ...backend, full-stack, and AI/ML systems powering delightful... ...and engineering to solve open problems in AI, including... ...designing AI agents, training models, and delivering secure,...
- ...building the next generation of AI agents for the insurance industry -... ...for an exceptional AI Engineer to join our agent team and help... ...techniques (e.g. multi-agent systems, reasoning-from-feedback, RFT... ...Note that this is not a model-training role - you’ll be building...SeniorFull timeWork at office
$175k - $240k
Member of Technical Staff, Senior Applied AI Engineer, Agents Aug 25, 2026 $175,000 - $2... ..., tool integrations, and evaluation frameworks. Develop reusable... ...experience shipping systems that real users depend on.... ...applications. Contributions to open-source scientific or AI...SeniorFull timeWork at office$190.2k - $360.5k
...looking for a Principal AI Systems Engineer with deep C++... ...integration layers, evaluation systems, and reliability... ...-powered services, agents, and model-backed workflows. Define... ...teams. Mentor senior engineers and raise... ...NoticeIf this role is open to hiring in Colorado...Full timeTemporary workLocal areaRemote workWorldwide- ...scale autonomous AI agents with full... ...looking for a Senior Applied AI Engineer to join our on-... ...improving production systems . You will work... ...including agent evaluation, observability,... ...maintain pipelines for model evaluation,... ...LLMs (APIs or open-source models) in...SeniorFull time
- ...Roger is an AI platform that... ...requires LLM systems that extract structure... ...looking for a Senior Applied AI Engineer to build the... ...fine-tuning models on our data, building... ...and fine-tune open source models,... ...-tuning, or evaluating LLMs and open... ...where one agent's output drives...SeniorFull timeRemote workWork from home
$204k - $259k
...U.S. states. The Planner Evaluation team works on one of the key... ...experienced data-minded software engineers and data scientists to help... ...to navigate complex systems and pursue open-ended problems to completion... ...data analysis tools for rapid modeling and prototyping...SeniorFull timeRemote work$200k - $400k
Overview Job Title: Founding AI Engineer Location: San Francisco, CA (On-site... ..., build, and launch a multi-agent AI system that will redefine the future of talent evaluation. You’ll join at a pivotal growth... ...DNA — hackathon wins, notable open-source contributions, or personal...SeniorFull timeImmediate startRelocationVisa sponsorshipRelocation packageFlexible hours$2,000 per month
...is the leading AI analytics... ...With powerful AI Agents embedded across... ...looking for a Senior AI Engineer to help advance... ..., build the systems that deliver it... ...orchestration, evaluation, retrieval,... ...AI ecosystem: model APIs, agent frameworks... ...empathy and openness, seeking to...SeniorHome office- ...building intelligent systems that sit at the... ...of AI, biology, chemistry... ...and large-scale engineering. Our goal is to... ...The Role As a Senior AI/ML Engineer,... ...learning models that form the core... ...Design, train, and evaluate large-scale... ...Contributions to open-source ML frameworks...SeniorFull timeRemote workFlexible hours
$200k - $250k
...building the AI Search Infrastructure... ...modern AI systems. Our goal is... ...layer that agents, applications... ...language models, and large-scale... ...includes engineers, researchers,... ...vertical feeds, evaluation tooling, and... ...credibility with senior engineers and... ...footprint — open-source,...SeniorFull timeImmediate startRemote workWork from homeFlexible hoursEarly shift$190.9k - $334.1k
...DescriptionIt all started when engineer Fred Luddy wrote... ...ServiceNow is the AI control tower for... .... Develop novel AI Agent frameworks,... ...customer enterprise systems including ServiceNow... ...engineering, and data modeling. Curiosity &... ...ConsultingExperience level: Mid-Senior LevelIndustry:...SeniorWork at officeImmediate startRemote workFlexible hoursDay shift- EY seeks a Lead AI Software Architect in the United States... ...architects designing AI Agent Control systems. You’ll shape... ...Infrastructure teams, drive open research, and mentor engineers through an AI-native design... ...process. EY offers a hybrid model and strong rewards. #J-1...Senior
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Agent Engineer - Open Models & Evaluation Systems. Be the first to apply!
- state farm agent San Francisco, CA
- tsa agent San Francisco, CA
- right of way agent San Francisco, CA
- operations agent San Francisco, CA
- import export agent San Francisco, CA
- commissioning agent San Francisco, CA
- remote chat agent San Francisco, CA
- sourcing agent San Francisco, CA
- agent San Francisco, CA
- showing agent San Francisco, CA


