AI Operations Analyst

About Calltree We're a YC W25 company with multiple enterprise customers, including public retail brands you'd recognize. We're building AI that actually works for complex, real-world use cases. For example, our AI handles 30+ minute technical troubleshooting calls—the kind everyone else avoids. The Role You'll be the human supervisor behind our AI agents, working with live customer interactions in real-time. You'll monitor how AI performs, ensure quality, and when needed, jump in to rescue conversations and ensure great customer outcomes. You'll also work directly with engineering to make the AI smarter every day. What You'll Do Monitor live AI-customer conversations and ensure quality standards Make real-time judgment calls on when to intervene Step in as the escalation point when customers need a human, de-escalating frustrated customers and resolving complex issues Identify patterns in AI performance and document what's working and what's not QA new AI experiences before release—interact with them as customers would and flag issues Work with engineering to improve the AI based on what you observe Build the playbooks and processes for this function as we scale What Success Looks Like Measurable improvement in AI performance metrics AND customer satisfaction Good judgment on when to intervene vs. let the AI continue Catching issues before they hit production Building institutional knowledge about what makes conversations succeed or fail What We're Looking For Recent college graduate (any field) Comfortable being on the phone with customers, including frustrated ones Someone who's excited about AI technology AND genuinely enjoys helping people Strong judgment and decision-making skills Ability to work one weekend day Honest Expectations This role is a balance of two things: operational monitoring work (watching, reading, documenting) and live customer interaction. Some days will feel repetitive. But you're also accountable for real customer outcomes—when you jump in, you're the person who saves the experience. If you love technology but never want to talk to customers, this isn't the role. If you want both—mastering AI systems AND helping real people—you'll thrive here. Career Growth This isn't just a job—it's a launchpad. Top performers have clear paths to become AI Team Leads, move into Product roles, or specialize in AI training and development. You'll gain hands-on experience with cutting-edge AI technology that will set you apart in your career. #J-18808-Ljbffr