Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Staff Technical Product Manager, Evals

$200k - $350k

Hercules

What are we looking for? Hercules is looking for a senior or staff level product manager to own evals for our AI coding agent What will you do? Your #1 and only goal is to make Hercules' AI coding agent cheaper, faster, and more accurate (our pareto frontier) You will own Hercules Bench, our offline eval set You will own Hercules online evals You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to guide your eval creation You will anonymize data to protect our customer's privacy What skills do we expect you to have? We expect candidates to be incredibly strong across all of the following areas: Software fundamentals : You are building an AI coding agent. We expect you to be technical and able to code. We expect you to have a decent understanding of system design, architecture, data modeling, and API design. In saying this, you will work with an engineering counterpart to work on complex implementation Evals . You have explicitly created and/or managed technical evals before. You should be able to work across a variety of different eval methodologies and deeply understand optimal reward modeling, Product sense : strong intuition for the goals customers are trying to achieve and novel solutions to help them achieve them Design sense : Track record of building simple, delightful UX/UI that makes it easy for customers to achieve their goals AI context and harness engineering : strong experience with the most up to date context and harness engineering best practices like summarization/compression, prompting, prompt caching, subagents, memory, context management, tools, async tools, deferred tools, skills, etc Yes, we know this industry is evolving rapidly and the \"best practices\" change every week, but yes, we are still actively looking for someone who has wrestled with these challenges at scale and has strong opinions about what works and what doesn't Our core stack : TypeScript, React, (some sort of agent framework like AI SDK), (some sort of eval platform like LangSmith) AI : able to truly 10x your workflow by being on frontier of AI coding What’s Hercules’ stack? Technologies: Typescript, React, AI SDK, Postgres, Redis, Clickhouse, Tailwind, IaC (CDK, Terraform), Docker, Otel, Hono, Drizzle ORM, Temporal Clouds: Cloudflare, AWS AI Models: OpenAI, Anthropic, Google What are Hercules’ operating principles? Hercules’ operating principles guide how we make decisions every day. We take them very seriously. Most of the principles can be learned on the job, however, we do specifically filter for the following principles in our hiring process: We ship . We ship fast, iterate, and correct mistakes quickly. Our bias for action speeds up our learning and delights our customers. We strive for excellence. Doing things well is in our DNA. We have good taste. We get the details right. Our outputs are exceptionally simple and self-serve. We’re proud of our work. We are hungry . We are willing to work hard and break through walls to achieve the outcomes we want. We are willing to operate at any level to get things done. Is Hercules in-office or remote? Hercules founding team works in-office in San Francisco (Kearny and Bush). We have a very strong preference for in office. We are open to remote for exceptional candidates who are willing to 1) work in US time zones and 2) travel to the office regularly. What are Hercules’ benefits? Unlimited AI token budget Take what you need vacation policy Healthcare, vision, and dental Daily lunches, dinners and snacks in the office We guarantee you will do the best work of your career here What’s the compensation? $200k-$350k + equity (variable based on seniority, equity, and location) #J-18808-Ljbffr Hercules

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Staff Technical Product Manager, Evals in San Francisco, CA vacancy
  •  ...Job Description Job Description Job Description Senior Technical Product Manager - AI Agents, Evals & Reliability, Artificial Intelligence (AI) Required, Work From Home   We are looking for a Senior Technical Product Manager.  This is a deeply technical,... 
    Suggested
    Remote work
    Work from home

    Parallel Partners

    San Francisco, CA
    7 days ago
  • Software Guidance & Assistance, Inc., (SGA), is searching for an Technical AI Product Manager for a Remote Full Time assignment with one of our premier Technology clients. Responsibilities : Define and document requirements for AI-powered products and systems... 
    Suggested
    Full time
    Remote work

    Software Guidance & Assistance

    San Francisco, CA
    3 days ago
  •  ...Product ManagerWe're hiring a Product Manager to help decide what Pump builds next as we grow from cloud savings into AI cost optimization. You'll own...  ...fast-growing B2B SaaS startupTrack record of shipping technical products that customers adopted and paid forTechnical... 
    Suggested

    Pump

    San Francisco, CA
    3 days ago
  •  ...running on software built 20-40 years ago. The product sits between legacy ERPs and modern...  ...across ERP Sync, AP Automation, Expense Management, and Vendor Management. About the Role We’re hiring a hands-on, technical product leader to build new products on top... 
    Suggested
    Full time
    For contractors
    Internship
    Immediate start
    Remote work

    Ersilia

    San Francisco, CA
    3 days ago
  •  ...Join to apply for the Technical Product Manager role at Novel Technology Services . Before continuing to the job description, please note a few important details: The client requires candidates to possess five years of experience as a Product Manager/Engineer. Internships... 
    Suggested
    Internship

    Novel Technology Services

    San Francisco, CA
    2 days ago
  •  ...enabling them to build the future without being constrained by infrastructure complexity. About the Role Smarsh is seeking a Technical AI Product Manager to lead the strategy, development, and commercialization of AI-powered solutions within our compliance and data archival... 

    Remote Services Inc.

    San Francisco, CA
    4 days ago
  •  ...not solved before. Other work expands a product that an existing customer already uses....  ...product that shipped 3+ years in product management, including helping take a product from 0...  ...in conversation Hands-on experience with technical products. You can write and review SQL,... 
    Home office

    HealthLeap

    San Francisco, CA
    4 days ago
  •  ...Technical Product ManagerSan Francisco | 5 Days OnsiteLocation: Onsite in San FranciscoCompensation: Competitive Salary + EquityUniversalAGI...  ...About the RoleUniversalAGI is hiring our first Technical Product Manager to build and own the execution of our core platform from 0 to... 
    For contractors
    Work at office
    Immediate start
    Flexible hours
    1 day per week

    UniversalAGI

    San Francisco, CA
    2 days ago
  • $143k - $175k

     ...their use, and combat underage usage of our products. We have the opportunity to address one...  ..., scientists, designers, product managers, operations experts, and customer service...  ...and requirements to audiences of varying technical backgrounds, including business stakeholders... 

    Juul

    San Francisco, CA
    5 days ago
  •  ...Job title: Senior Technical Product Manager Number of Positions: 2 Domain: Order-to-Cash (O2C) We are seeking Senior Technical Product Managers / Product Architects who possess a strong blend of business, functional, and technical expertise within the Order-to-Cash (O2... 

    CriticalRiver Inc.

    San Francisco, CA
    2 days ago
  •  ...Senior Technical Product ManagerLocation: San Francisco, CA Company Stage of Funding: Series B AI Enterprise Software (91M Raised) Office Type...  ..., and workflow automation. As a Senior Technical Product Manager, you'll own core product areas that blend AI, data, and enterprise... 
    Work at office
    Relocation package

    Recruiting from Scratch

    San Francisco, CA
    2 days ago
  • $160k - $180k

     ...Bobyard Construction Product RoleBobyard Construction is a multi-trillion industry running on PDFs, rulers, and 1990s software. The biggest choke point is the cost estimate. Takeoffs are manual, slow, and one mistake blows millions. We use computer vision and NLP to read... 

    Bobyard

    San Francisco, CA
    3 days ago
  •  ...David AI Technical Product ManagerDavid AI is the first audio data research company. We bring an R&D approach to data–developing datasets with...  ...to push the frontier of audio AI.Our Technical Product Manager team builds the Data Factory at David AI. We set the strategy... 
    Work experience placement
    Work at office

    David AI

    San Francisco, CA
    4 days ago
  •  ...teams to prioritize patient care and access to healthcare. Technical Product Manager You own product vision, roadmap, and momentum for our agentic...  ...too early. You work with health care executives and ground staff all the same. You sit at the intersection of customers,... 
    Immediate start

    CareForce

    San Francisco, CA
    5 days ago
  •  ...looking for bold, ambitious people to help redefine how businesses manage and experience travel. Working here, you can expect...  ...ordinary and build the future of travel with us. Your Mission As a Technical Product Manager at Engine, you'll own key parts of a platform at the... 

    Remote Services Inc.

    San Francisco, CA
    3 days ago
  •  ...Technical Product ManagerSensor Tower's digital intelligence platform provides unparalleled insight into market trends, advertising, competitor...  ...industry leaders have found success. As a technical product manager focused on our mobile intelligence offering, you will help... 
    Contract work
    Worldwide

    Sensor Tower

    San Francisco, CA
    5 days ago
  • $104.9k - $199.07k

     ...important healthcare questions. Through our products, education, and services, MedInsight is...  ...solutions for healthcare cost and care management.MedInsight is a subsidiary of Milliman;...  ...Summary: MedInsight is seeking a Technical Product Manager with a platform engineering... 
    Full time
    Work experience placement
    Remote work
    Flexible hours

    Milliman

    San Francisco, CA
    1 day ago
  • $190.8k - $267.1k

    Senior Technical Product Manager, Ads ML Platform Reddit Senior Technical Product Manager, Ads ML Platform Team Description: The Ads Marketplace team is a strategic growth engine for Reddit Ads. Our mission is to democratize high-performance advertising by enabling brands... 
    For contractors
    Work experience placement
    Work at office
    Remote work
    Home office
    Flexible hours

    Big Internet Seller Services Inc

    San Francisco, CA
    1 day ago
  • $180k - $220k

     ...challenging work, with the prime directive of "Keeping Our Heroes Connected"? THE OPPORTUNITY Silvus is seeking a highly motivated Technical Product Manager (TPM) to drive requirements, product backlog, market adoption, and revenue growth for the Electronic Warfare (EW) &... 
    Permanent employment
    For contractors
    Work at office
    Remote work
    Relocation

    Motorola Solutions Inc

    San Francisco, CA
    1 day ago
  • Motorola Solutions’ Silvus Technologies, Irvine, CA, seeks a Technical Product Manager (Spectrum Dominance) to translate DoD requirements into structured software features and own the end-to-end backlog. The role bridges R&D and go-to-market for advanced spectrum dominance... 
    Remote job

    Motorola Solutions Inc

    San Francisco, CA
    1 day ago
  • $100k - $150k

     ...of our customers delivering life-changing medicines and treatments to patients more efficient and productive. We are looking for an experienced technical product manager to design and deliver high-value AI Agent solutions for Vault Medical. In this role, you will be the... 
    Work at office
    Local area
    Work from home

    Veeva Systems, Inc.

    San Francisco, CA
    11 hours ago
  • $101k - $152k

     ...any employer, at the date of hire. This position is ineligible for employment Visa sponsorship. Overall Purpose The Technical Product Manager is a key role on our Data & AI team charged with defining and delivering the next generation of enterprise data platform capabilities... 
    Hourly pay
    Temporary work
    Work at office
    Immediate start
    Visa sponsorship
    Work visa
    Flexible hours

    Early Warning Services, LLC

    San Francisco, CA
    1 day ago
  •  ...Technical Product ManagerSan Francisco, CAAbout UsMetriport is an open-source data intelligence platform that helps healthcare organizations...  ...product specifications for the engineering team.Execution: Managing the product lifecycle from ideation through development,... 
    Work at office
    Work from home
    Relocation

    Metriport Inc

    San Francisco, CA
    1 day ago
  • Anthropic Limited is seeking a Technical Program Manager for Safeguards Infrastructure and Evals in San Francisco. You will own operational health, drive reliability, incident response, and post-mortem processes while coordinating platform investments and cross-team dependencies... 

    Anthropic Limited

    San Francisco, CA
    1 day ago
  • $178k - $258k

     ...experience in corporate/commercial strategy, management consulting, business architecture, or Go-To-Market strategy. Experience in product or program management. Preferred...  ...responsible for all support and media and technical services for customers big and small across... 
    Worldwide

    Google LLC

    San Francisco, CA
    1 day ago
  • $225k

     ...scheduling, and dedicated local property management, so owners get the joy of a true...  ...to purchase and make our growth product-led. We're hiring a principal-level technical product manager to own that...  ...how to prove they're ready. Evals and guardrails go in before launch... 
    Full time
    Local area
    Remote work
    Home office

    Pacaso

    San Francisco, CA
    3 days ago
  • $220k - $240k

     ...AI and high-performance computing. Our Technical Sourcing team sits at the center of that...  ...IREN is looking for a Technical Sourcing Manager, Server & Storage Platforms to take full...  ...’s speed and margins. Track OEM and ODM product roadmaps closely and shape sourcing strategy... 
    Daily paid
    Contract work
    Temporary work
    Live in
    Live out
    Relocation package

    IREN

    San Francisco, CA
    1 day ago
  •  ...information, this is the place for you.   Evals at Exa   The Evals Team sits at the...  .... That’s where you come in.   As a Product Manager on the Evals team, you will partner...  ...Strong numerical literacy and technical background. CS/EE degree, production engineering... 
    Full time
    H1b

    exa

    San Francisco, CA
    5 days ago
  • $228k - $314k

     ...NBCUniversal, and Square—build better products and digital experiences. With...  .... As Director of Product Management for AI & Growth, you’ll lead...  ...” looks like for AI. Set up evals, quality benchmarks,...  ...adoption and revenue.Enough technical depth to engage credibly with... 
    Home office

    Amplitude

    San Francisco, CA
    3 days ago
  • $385k

     ...to build beneficial AI systems. About the role As a Product Manager, Technical, you will own and be responsible for building a world class...  ...Location-based hybrid policy: Currently, we expect all staff to be in one of our offices at least 25% of the time. However... 
    Work at office
    Local area
    Visa sponsorship
    Flexible hours

    Anthropic

    San Francisco, CA
    9 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Staff Technical Product Manager, Evals. Be the first to apply!