Senior Technical Product Manager - AI Agents, Evals & Reliability

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Technical Product Manager - AI Agents, Evals & Reliability based in United States.

This is a deeply technical product leadership role focused on turning advanced AI capabilities into reliable, user-facing experiences.
You will shape systems designed to support long-running workflows, persistent context, multi-step reasoning, and real-world task completion.
Working closely with ML, backend, mobile, and product teams, you will translate technical possibilities into clear product requirements and system decisions.
A major focus will be building robust evaluation frameworks that measure quality, reliability, and user impact across AI systems.
You’ll make high-leverage trade-offs involving quality, latency, cost, reliability, and user experience in a rapidly evolving environment.
The role offers significant ownership within a high-talent-density, hands-on team where decisions move quickly and execution matters.
Your work will directly influence how AI products behave, earn user trust, and deliver practical value at global scale.

Accountabilities:

• Define end-to-end requirements for AI-powered systems, connecting model capabilities and technical constraints to user needs and measurable product outcomes.
• Translate model performance, data limitations, evaluation results, and user feedback into clear product and system decisions.
• Partner closely with ML, backend, mobile, and other engineering teams on architecture, system design, evaluation, iteration, and product delivery.
• Establish and continuously improve evaluation frameworks spanning offline metrics, online experiments, human feedback, and real-world user outcomes.
• Make informed trade-offs across product quality, reliability, latency, cost, speed of execution, and overall user experience.
• Drive execution through clear product specifications, disciplined prioritization, strong technical judgment, and effective cross-functional alignment.
• Own product quality end-to-end, with particular emphasis on correctness, predictability, reliability, failure handling, and user trust.
• Identify weaknesses in AI behavior, including hallucinations, bias, model limitations, drift, and other sources of unreliable performance, and drive solutions to address them.
• Establish strong feedback loops that enable teams to learn quickly from production behavior and continuously improve AI-powered workflows.
• Ensure new AI capabilities are shipped efficiently and responsibly while maintaining high standards for quality and reliability.
• Help align product strategy, roadmaps, and priorities with user needs, AI capabilities, and broader business objectives.
• Bring structure to ambiguous problems and independently drive complex initiatives from definition through execution and measurable outcomes.

Requirements:

• Strong foundation in computer science fundamentals, including algorithms, data structures, and system design, with the ability to engage deeply in technical discussions.
• Solid understanding of machine learning fundamentals and how modern AI systems behave in production environments.
• Significant experience owning complex, technically sophisticated products from strategy through execution.
• Hands-on experience with AI-powered products, particularly LLM-based systems, model evaluation, prompt or pipeline iteration, and AI feedback loops.
• Strong intuition for AI limitations and failure modes, including hallucinations, bias, model drift, and non-deterministic behavior.
• Ability to read, review, and discuss technical design documents and confidently collaborate with senior engineers and ML specialists.
• Demonstrated ability to make sound product and technical decisions in ambiguous, rapidly changing environments.
• Strong judgment and prioritization skills, with the ability to balance ambitious product goals against technical, operational, and commercial realities.
• Excellent communication and collaboration skills, with the ability to create alignment across highly technical, cross-functional teams.
• A hands-on, independent working style and the ability to bring structure to complex problems while maintaining a high pace of execution.
• Experience working in high-talent-density or small, fast-moving teams where individuals are expected to take significant ownership.
• Experience shipping AI-heavy consumer products is strongly preferred.
• An engineering background or experience as a highly technical product manager is a plus.
• Experience defining evaluation metrics for machine learning systems, developing AI UX patterns, or working in zero-to-one product environments is highly valued.

Benefits:

• Opportunity to shape AI products designed to deliver practical value to billions of potential users.
• High-ownership role within a small, high-talent-density, hands-on team.
• Significant influence over AI product strategy, system behavior, evaluation, and reliability.
• Close collaboration with experienced ML, backend, mobile, and product engineering professionals.
• Fast-paced environment emphasizing rapid iteration, thoughtful experimentation, and high-quality execution.
• Opportunity to work on challenging problems involving AI agents, long-running workflows, evaluation, reliability, and real-world task completion.
• Direct impact on product quality, user trust, and the practical application of advanced AI technologies.
• Efficient interview process consisting of approximately 3–4 interviews for candidates who advance, conducted virtually and/or onsite.
• Prompt decision-making and a collaborative environment focused on meaningful outcomes.

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.