Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior AI Product Manager, Evals & Annotations (Remote)

$200k - $250k

Arize

About Arize AI is rapidly transforming the world. As generative AI reshapes industries, teams need powerful ways to monitor, troubleshoot, and optimize their AI systems. That’s where we come in. Arize AI is the leading AI & Agent Engineering observability and evaluation platform, empowering AI engineers to ship high-performing, reliable agents and applications. From first prototype to production scale, Arize unifies build, test, and run in a single workspace, so teams can ship faster with confidence. Arize serves a rapidly growing customer base of 150+ leading enterprises and Fortune 500 companies. Customers like Booking.com, Uber, Siemens, and PepsiCo leverage Arize to deliver AI that works. The Opportunity Evaluations are central to how AI developers understand, measure, and improve their applications. As AI systems become increasingly complex and agentic, developers need cohesive workflows to define what good looks like, evaluate behavior, incorporate human feedback, and continuously improve their applications. We're looking for a highly technical, product-minded PM to own Evals & Annotations at Arize. You'll own the end-to-end evaluation lifecycle—from eval creation and execution to human review, analysis, and iteration. This is an opportunity to bring together powerful capabilities into a cohesive, best-in-class developer experience while helping define the future of evaluations in a world of increasingly autonomous and managed agents, including through our integration with Dynatrace. This role is ideal for someone with strong product and UX opinions who doesn't just define requirements but actively shapes the experience. You'll independently explore workflows, build prototypes, and collaborate closely with Design and Engineering to turn ideas into polished, technically sound products. You should be as comfortable working through an interaction design as you are discussing technical tradeoffs with engineers. While Arize has been acquired, we continue to operate with the urgency, resourcefulness, and hands‑on mentality of an early‑stage startup. Product, Engineering, and Design collaborate closely, and our founders remain actively involved in product thinking and design. You'll have real ownership while working directly with our cofounder/CPO in an environment that values initiative, constructive debate, and a willingness to jump in wherever needed. What You'll Do Own Arize's Evals & Annotations product area end‑to‑end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle. Set the product strategy and roadmap, balancing customer needs, foundational platform investments, UX improvements, and emerging opportunities in AI and agent development. Drive a best‑in‑class evaluation experience. Develop strong opinions about how workflows should work, independently prototype solutions, and partner with Design to turn those ideas into intuitive, polished experiences. Shape the future of evaluations as AI development evolves toward autonomous and managed agents, including opportunities within the broader Dynatrace ecosystem. Work closely with Engineering on technically complex problems spanning evaluation execution, backend systems, orchestration, and data infrastructure. Contribute to technical discussions, implementation approaches, and tradeoffs. Spend significant time with customers—typically multiple conversations each week—to understand how they evaluate and improve AI systems, uncover underlying problems, and validate product direction. Collaborate directly with our cofounders and cross-functional teams on product direction, design exploration, and prioritization, bringing independent judgment and turning collaborative thinking into execution. Make clear prioritization decisions across competing demands and drive products from problem definition through launch, adoption, and iteration. What We're Looking For Significant product management experience independently owning technically complex developer‑facing products. We're flexible on exact years of experience for candidates with exceptional technical depth, product judgment, and ownership. Strong understanding of LLM evaluations and AI development workflows, including automated evals, human feedback, experimentation, and how developers iterate on AI applications and agents. Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products, with a solid understanding of backend systems and software architecture. Strong technical depth and hands‑on engineering ability. We don't expect you to ship large features as an IC engineer, but we do expect you to: Debug issues independently and navigate code when needed. Put up small fixes or PRs. Build prototypes and technical proofs of concept. Propose technical implementations and reason through tradeoffs. Hold credible technical discussions with engineers. Excellent product and UX instincts. You can independently define and prototype core product experiences, not just write requirements, while partnering effectively with designers and design engineers to refine and ship them. Strong ownership, initiative, and bias toward action. You're comfortable navigating ambiguity, making tradeoffs, and getting hands‑on without waiting for a fully defined roadmap or direction. A collaborative, low‑ego approach to working with founders, engineers, and designers. You bring strong opinions, engage in constructive debate, and thrive in an environment where responsibilities overlap. Experience working closely with technical customers, with strong communication skills and the ability to move comfortably between strategy and execution. Bonus Points, But Not Required Direct experience with LLM evaluation platforms, annotation systems, human‑in‑the‑loop workflows, or agent orchestration. Strong backend or platform engineering background, particularly with distributed systems, workflow orchestration, or data infrastructure. Hands‑on programming experience and comfort working across backend and frontend technologies. Experience with LLM-as-a-judge, custom evaluators, experiment tracking, datasets, or feedback‑driven improvement workflows. Previous engineering experience or substantial hands‑on software development experience. Early‑stage or high‑growth startup experience, particularly in founder‑led environments. The estimated annual salary for this role is between $200,000-$250,000, plus a competitive equity package. Actual compensation is determined based upon a variety of job related factors that may include: transferable work experience, skill sets, and qualifications. Total compensation also includes a comprehensive benefit package, including: medical, dental, vision, 401(k) plan, unlimited paid time off, generous parental leave plan, and others for mental and wellness support. While we are a remote‑first company, we have opened offices in New York City and the San Francisco Bay Area, as an option for those in those cities who wish to work in‑person. For all other employees, there is a WFH monthly stipend to pay for co‑working spaces. More About Arize Arize’s mission is to make the world’s AI work—and work for people. Our founders came together through a shared frustration: while investments in AI are growing rapidly across every industry, organizations face a critical challenge—understanding whether AI is performing and how to improve it at scale. Learn more about what we're doing here: Diversity & Inclusion @ Arize Our company's mission is to make AI work and make AI work for the people, we hope to make an impact in bias industry-wide and that's a big motivator for people who work here. We actively hope that individuals contribute to a good culture Regularly have chats with industry experts, researchers, and ethicists across the ecosystem to advance the use of responsible AI Culturally conscious events such as LGBTQ trivia during pride month We have an active Lady Arizers subgroup #J-18808-Ljbffr Arize

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior AI Product Manager, Evals & Annotations (Remote) in San Francisco, CA vacancy
  • $200k - $250k

     ...About Arize AI is rapidly transforming the world...  ...first prototype to production scale, Arize unifies build...  ...-minded PM to own Evals & Annotations at Arize. You'll own...  ...increasingly autonomous and managed agents, including...  .... While we are a remote-first company, we have... 
    Remote work
    Senior
    Full time
    Work experience placement
    Work from home
    Flexible hours

    Arize AI

    Remote
    8 days ago
  • Arize, a leader in AI observability, is seeking a highly technical, product-minded PM to own Evals & Annotations. You’ll shape evaluation workflows end-to-end—from creation to human...  ...founders and cross-functional peers in a remote‑first environment, with optional offices... 
    Remote work
    Senior

    Arize

    San Francisco, CA
    1 day ago
  •  ...Description Job Description Job Description Senior Technical Product Manager - AI Agents, Evals & Reliability, Artificial Intelligence (AI) Required...  ...required for this position.  This position is 100% Remote.   MUST BE WILLING TO TAKE A 60 MINUTE CODING... 
    Remote work
    Senior
    Work from home

    Parallel Partners

    San Francisco, CA
    7 days ago
  • $153.94k - $213k

     ...Engine Engine is the AI-native platform for...  ...redefine how businesses manage and experience travel....  ...It’s showing up in our product, our pace of innovation...  ...where they fail, and how evals and human review fit into...  ...offices or fully remote, we'll make sure you have... 
    Remote work
    Senior
    Full time

    Engine

    Remote
    2 days ago
  • $170k - $250k

     ...About the job Senior Technical Product Manager Senior Technical Product Manager $170K–$250K Remote US. This is a pure IC role. No direct reports...  ...Able to prototype using AI coding tools. Experience...  ...of LLMs, prompting, evals, retrieval, and fine-tuning.... 
    Remote work
    Senior

    Prime Talent Recruiting

    Sacramento, CA
    1 day ago
  • $180k

     ...their suppliers at scale. Our AI research and monitoring agents...  ...Seattle and Warsaw. We support remote and hybrid work, with team...  ...base. Companies, individuals and product networks hide things: ownership...  ...LLM-backed product and run evals on it. This is the bar. ~ You... 
    Remote work
    Senior
    Full time

    Craft Machine Inc

    Remote
    2 days ago
  • $400k

     ...with intuitive spend management, bill pay, and...  ...effortlessly. Brex’s AI-native automation and...  ...grow your career.Product at BrexThe Product...  ...prompts, designing the evals that prove it works...  ..., and Thursday. Remote work is allowed up...  ...and judgment this senior role calls for.You'... 
    Remote work
    Senior
    Work at office
    Shift work
    3 days per week

    Brex

    San Francisco, CA
    1 day ago
  • $184k - $230k

     ...with intuitive spend management, bill pay, and...  ...effortlessly. Brex’s AI-native automation and...  ...behind the bet. Product at Brex The Product...  ...space — the right evals, the right feedback...  ...Wednesday, and Thursday. Remote work is allowed up...  ...and judgment this senior role calls for.... 
    Remote work
    Senior
    Work at office
    Shift work
    3 days per week

    Brex

    San Francisco, CA
    1 day ago
  • $200.7k - $229.1k

     ...One Shopping - the AI Enablement team...  ...builds and owns the AI products and platforms the...  ...runs on. Product Management at Capital One is...  ...to-SQL, tool use, evals, and where these systems...  ...to present to senior executives and...  ...open to hiring a Remote Employee for this... 
    Remote work
    Senior
    Full time
    Part time
    Work experience placement
    Local area

    Capital One

    New York, NY
    13 days ago
  •  ...Owning the capabilities and builder experience, the full-time Senior AI Product Manager will drive the adoption and go-to-market strategy for an...  ...a seamless experience for developers while working remotely. Key responsibilities Define and manage the adoption lifecycle... 
    Remote work
    Senior
    Full time

    Virtual Vocations Inc

    United States
    5 days ago
  • $190.9k - $334.1k

     ...Description Moveworksis the Agentic AI Assistant platform that...  .... Job Description As a product manager for Tool Studio,you’llbe on the...  ...deconstructing complex problems, annotating & understanding patterns, and...  .... Work personas (flexible, remote, or required in office) are categories... 
    Remote work
    Senior
    Work at office
    Flexible hours

    ServiceNow

    Mountain View, CA
    1 day ago
  •  ...Owning the product direction for the AI & Query Platform team, the full-time Senior Product Manager, AI will manage product strategy, oversee the evolution of query capabilities, and drive AI-powered experiences in a remote work environment. Key responsibilities Develop... 
    Remote work
    Senior
    Full time

    Virtual Vocations Inc

    United States
    5 days ago
  • Siemens AG in the United States is seeking a Senior Principal Product Manager to define the strategy and architecture for a key AI-native product domain, working across engineering, ML, design, and hardware to create novel interfaces and platforms. This role demands navigating... 
    Remote job
    Senior

    Siemens AG

    Fremont, CA
    1 day ago
  • Fieldguide, a remote-first growth-stage startup in California, seeks a Principal/Staff Product Manager for Advisory. You’ll shape product strategy and roadmaps that scale our AI-native agents from individual workflows to full engagement lifecycle. You’ll work with top accounting... 
    Remote work
    Senior

    Fieldguide

    San Francisco, CA
    1 day ago
  • Chainguard seeks a Senior Product Manager for AI CI/CD to own the product vision, strategy, and roadmap. You will study how developers and agents...  ...role emphasizes 0→1 product work, deep collaboration with engineering, and the ability to translate #J-18808-Ljbffr Remote
    Remote job
    Senior

    Remote

    San Francisco, CA
    1 day ago
  • Capital One Shopping is seeking a Senior Product Manager for AI Enablement with a focus on analytics. You’ll own an agentic analytics platform, work across data governance layers, and prototype AI-driven solutions using modern data stack tools. The role emphasizes building... 
    Remote job
    Senior

    Capital One Group

    Boston, MA
    1 day ago
  • Roku is hiring a Senior Product Manager, Content Discovery to build experiences that help viewers discover, browse, and engage with content across...  ..., and entertain viewers. As content catalogs grow and AI opens new possibilities, discovery is a high-impact space. This... 
    Remote job
    Senior

    Roku

    New York, NY
    1 day ago
  • ServiceNow is seeking a Senior Staff Inbound Product Manager for Conversational AI to own the roadmap, vision, and requirements for AI features. You’ll translate customer feedback into strategic priorities and guide product strategy toward scaling revenue, collaborating... 
    Remote job
    Senior

    ServiceNow

    San Diego, CA
    1 day ago
  • Workiva Inc. is seeking a Principal BT Product Manager for Quote-to-Cash to lead strategy, delivery, and health of the revenue-lifecycle architecture...  ...into recognized revenue, leveraging CPQ, Billing, ERP and AI tooling. This role requires strong leadership of BSAs,... 
    Remote job
    Senior

    Workiva Inc.

    Ames, IA
    1 day ago
  •  ...Health Partners India is seeking a seasoned Product Manager to lead product initiatives in...  ...offshore engineering teams, and leverage AI-enabled tools to validate ideas and accelerate...  ...fostering growth and innovation in a remote India setting. #J-18808-Ljbffr Ensemble... 
    Remote work
    Senior

    Ensemble Health Partners

    New York, NY
    15 hours ago
  • OneSignal is seeking a Senior Product Manager to own the roadmap for a product area and collaborate with engineering, design, and marketing to deliver high-impact features that scale for B2C app customers. You will engage with customers, translate problems into opportunities... 
    Remote job
    Senior

    OneSignal

    New York, NY
    1 day ago
  •  ...organizations and serve their members. As a Senior Product Manager on our association software portfolio,...  ...use and value what you ship. Use AI in your daily work for research,...  ...Employer-Paid Short-Term Disability Remote Work Flexibility Momentive Software... 
    Remote work
    Senior
    Full time
    Temporary work
    Local area
    Flexible hours

    Momentive

    Remote
    2 days ago
  • ServiceNow is seeking a Principal Inbound Product Manager for the Now Engage platform. You will...  ...end-to-end product strategy, roadmap, and AI-enabled capabilities, partnering with...  ...execution risk and strategic priorities in a remote-friendly, globally #J-18808-Ljbffr... 
    Remote job
    Senior

    ServiceNow

    San Diego, CA
    1 day ago
  • Veeva Systems, Inc. is seeking an experienced Senior Product Manager to design and deliver high-value AI Agent solutions for Vault Medical. You’ll leverage AI to...  ...engineering, and ROI-driven metrics. This role supports remote/hybrid work in the EST region. #J-18808-Ljbffr... 
    Remote job
    Senior

    Veeva Systems, Inc.

    Boston, MA
    1 day ago
  • Scopely’s AI-native incubation team is seeking a Director or Senior Director of Product. This role can be remote in the US or hybrid in Culver City, LA. You will lead rapid experimentation, make decisions with incomplete information, and bring new ideas to life across... 
    Remote job
    Senior

    Scopely

    Sacramento, CA
    1 day ago
  • Capital One Shopping is seeking a Sr Director of Product Management for Growth, with a strong emphasis on AI-driven product development and data-backed decision making. You will lead cross-functional teams to craft incentive-driven experiences for merchants and consumers... 
    Remote job
    Senior

    Capital One Group

    Frankfort, KY
    1 day ago
  • $85k

    Carrington Mortgage Services, LLC is seeking a Sr LS Product Integration Analyst to lead cross-functional initiatives, evaluating, implementing...  ...value and alignment with strategic goals. The role covers AI, workflow automation, system modernization, and third-party... 
    Remote job
    Senior

    Carrington Mortgage Services, LLC

    New York, NY
    1 day ago
  • Carrington Mortgage Services is seeking a Sr LS Product Integration Analyst to lead enterprise...  ...on high-profile initiatives, including AI and automation, to deliver measurable...  ...executive reporting, with a focus on adoption and change management. #J-18808-Ljbffr Carrington
    Remote job
    Senior

    Carrington

    Frankfort, KY
    4 days ago
  •  ...reporting and integrations, all on one system. We're hiring a remote Senior Technical Product Manager to own it: decide what we build, in what order, and what...  ...Establish consistent testing standards and help improve AI-assisted regression testing, working with Operations to... 
    Remote work
    Senior

    Proven Software, Inc.

    Annapolis, MD
    4 days ago
  • Carrington Mortgage Services is seeking a Sr LS Product Integration Analyst to drive cross-functional...  ...Based in the United States with the option to work remotely, the position requires leading strategic initiatives, managing complex enterprise #J-18808-Ljbffr Carrington
    Remote job
    Senior

    Carrington

    Dover, DE
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior AI Product Manager, Evals & Annotations (Remote). Be the first to apply!