Senior AI Product Engineer

Role Description

We're looking for a Senior AI Product Engineer to own the continuous improvement of our customer-facing AI products, making them more accurate, reliable, cost-efficient, and genuinely useful to the people who rely on them every day. This isn't a prompt-writing job, it's about building the structured loop of evaluation, experimentation, and analytics that turns AI from a feature into a measurable product advantage. You'll work at the intersection of Product, Growth, and Engineering, with real influence over how AI behavior gets measured, improved, and shipped.

What You’ll Do

• Own the end-to-end quality of our AI-powered product experiences, improving response usefulness, consistency, and reliability in ways users actually notice.
• Build evaluation frameworks and repeatable test sets so we can measure AI quality objectively, not anecdotally.
• Design and run experiments on prompts, context strategies, and model settings, connecting every change to activation, conversion, retention, or satisfaction metrics.
• Monitor and reduce AI-related costs (tokens, latency, model choice, retrieval) without compromising the user experience.
• Translate AI failure patterns, hallucinations, inconsistent outputs, weak reasoning, into clear, prioritized fixes for Product and Engineering.
• Help Product, Growth, and Engineering teams understand where AI changes create real user and business impact.

Qualifications

• 5+ years in software engineering, AI product engineering, ML engineering, or a related technical role, plus 2+ years hands-on with LLMs, prompt engineering, or applied AI systems.
• Proven experience improving AI features post-launch through analytics, experimentation, and structured evaluation, not just initial prompt design.
• Strong grasp of product analytics, funnel metrics, and how to connect AI behavior to business outcomes.
• Ability to diagnose whether an AI quality issue stems from prompts, context, model limitations, UX, or data, and to fix the right layer.
• Strong written communication and the ability to work autonomously in a remote, async environment.
• Comfort working closely with Product, Growth, Design, Engineering, QA, and Analytics.
• Comfort with ownership, transparency, and a results-driven mindset.

Requirements

• Experience with A/B testing frameworks, AI observability tools, or custom evaluation pipelines.
• Hands-on work with OpenAI, Anthropic, Gemini, AWS Bedrock, or similar LLM providers.
• Familiarity with RAG, embeddings, semantic search, or tool-calling architectures.
• Background in chatbot, assistant, content generation, or recommendation products.

Benefits

• Remote-first: work from anywhere in the world, no relocation required.
• Flexible hours: we measure outcomes, not time at a desk.
• Unlimited PTO: take the time you need to rest, recharge, and do your best work.
• Learning & development budget: courses, books, conferences, certifications, whatever helps you grow.
• Performance-based compensation: your impact directly shapes your pay and career trajectory.
• A small, senior, collaborative team where your work is visible and your decisions matter.
• The chance to build products used by millions of people across 160+ countries.

Equal Opportunity

Progress Partners is an equal opportunity employer, and we’re serious about building a diverse, inclusive team. We welcome applications from people of all genders, gender identities and expressions, sexual orientations, ages, ethnicities, nationalities, religions, disabilities, neurotypes, family situations, socioeconomic backgrounds, and life experiences.

We hire on skills, experience, potential, and values fit, never on protected characteristics. If you need accommodations during the application or interview process, just let us know.