Senior Evaluation Specialist, AI Operations

What we’re building and why we’re building it Fetch helps people live rewarded every day, with a vision to become the rewards destination for everyone. We turn everyday activities into meaningful rewards, whether it’s grocery shopping, grabbing a quick meal, or playing a favorite mobile game. To date, we’ve awarded more than $1 billion in Fetch Points to our users. Each day, more than 13 million receipts are submitted on Fetch, providing visibility into over $212 billion in gross merchandise value. This creates the largest retail‑agnostic, SKU‑level view of household spending, powering Fetch as an outcomes‑based advertising platform that helps brands acquire and retain lifelong consumers. The Fetch app is available on the App Store and Google Play, with more than 6 million five‑star reviews from a highly engaged and loyal user base. It’s not just our users who believe in Fetch: with investments from Softbank, ICONIQ, DST, Greycroft, and partnerships ranging from challenger brands to Fortune 500 companies, Fetch is reshaping how brands and consumers connect in the marketplace. When you work at Fetch, you play a vital role in a platform that drives brand loyalty and creates lifelong consumers with the power of Fetch points. User and partner success are at the heart of everything we do, and we extend that same commitment to our employees. At Fetch, we value curiosity, adaptability, and the confidence to explore new tools, especially AI, to drive smarter, faster work. You don’t need to be an expert, but you should be ready to learn quickly and think critically. We welcome learners who move fast, challenge the status quo, and shape what’s next, with us. Ranked as one of America’s Best Startup Employers by Forbes for two years in a row, Fetch fosters a people‑first culture rooted in trust, accountability, and innovation. We encourage our employees to challenge ideas, think bigger, and always bring the fun to Fetch. About The Role At Fetch, we’re building AI and automation systems that make our work smarter, faster, and more scalable. The AI Operations team ensures our models and workflows perform with quality, reliability, and measurable impact. As a Senior Automation Specialist, you’ll own evaluation and dataset workstreams that improve AI system performance. You’ll define what “good” looks like, design how we measure it, and build the datasets, evaluations, and automations that turn insights into action. This role is ideal for someone who is self‑directed, biases toward action and rapid iteration, and takes initiative to turn ambiguous problems into structured, scalable solutions. This is a full‑time role that can be held from one of our US offices or remotely in the United States. Role Responsibilities Own evaluation & datasets: Define evaluation approaches, design gold datasets (GDS), and ensure coverage of real‑world scenarios and edge cases Build evaluation systems: Develop manual and automated evals, including LLM‑as‑judge patterns, to measure model quality and performance Translate ambiguity into structure: Turn open‑ended questions into clear evaluation frameworks and execution plans Build automations: Create automations that improve workflows, including dataset creation, evaluation pipelines, and lightweight operational processes Measure and iterate: Define and track performance metrics; refine datasets, evaluations, and workflows based on results Drive execution forward: Operate with urgency and ownership; identify next steps, unblock progress, and move work forward with minimal oversight Collaborate cross‑functionally: Partner with other Automation Specialists, engineering, cross‑functional stakeholders, and project leads to ensure high‑quality, timely project deliverables Improve systems: Identify gaps and implement scalable improvements to evaluation and data workflows Minimum Requirements 3+ years of experience designing or working with evaluation frameworks, datasets, or quality measurement systems Experience building or managing datasets (labeling, QA, iteration) Ability to independently drive tasks from problem definition to execution Hands‑on experience with AI tools, LLM workflows, or automation platforms Experience in defining and tracking model performance Basic scripting or data skills (SQL, Python, etc.) Preferred Requirements Experience with LLM‑as‑judge or model evaluation techniques Familiarity with prompt evaluation or benchmarking approaches Experience productionizing evaluation workflows with engineering teams Compensation At Fetch, we offer competitive compensation packages including base, equity, and benefits to the exceptional folks we hire. The base salary range for this position is [$75,000 - $85,000]. Discover our benefits and how our employees live rewarded at Benefits Equity: We offer full‑time employees equity in Fetch, so that everyone can benefit from Fetch’s growth. 401k Match: Dollar‑for‑dollar match up to 4%. Benefits for humans and pets: We offer comprehensive medical, dental and vision plans for everyone including your pets. Continuing Education: Fetch provides ten thousand per year in education reimbursement. Employee Resource Groups: Take part in employee‑led groups that are centered around fostering a diverse and inclusive workplace through events, dialogue and advocacy. The ERGs participate in our Inclusion Council with members of executive leadership. Paid Time Off: On top of our flexible PTO, Fetch observes 9 paid holidays, as well as our year‑end week‑long break. Robust Leave Policies: 20 weeks of paid parental leave for primary caregivers, 14 weeks for secondary caregivers, and a flexible return to work schedule. Calvin Care Cash: Employees who are welcoming new family members will also receive a one time $2,000 incentive to assist employees with covering the cost of childcare, clothing, diapers and much more! Flexible Work Environment: Collaborate with your team in one of our stunning offices, or you can work fully remotely from anywhere in the US. We’ll ensure you are equally equipped with the hardware and software you need to get your job done in the comfort of your home. (applicable for most roles) Equal Opportunity Employer Fetch is an equal opportunity employer that embraces diversity, inclusion, and respect for all individuals. We do not discriminate on the basis of race, color, religion, gender, gender identity or expression, sexual orientation, age, national origin, marital status, veteran status, disability, or any other characteristic protected by applicable law. Our commitment to inclusivity ensures that everyone is treated with dignity and has the opportunity to succeed based on their talent, skills, and potential. Fetch also provides reasonable accommodations to qualified individuals with disabilities or those with sincerely held religious beliefs, as required by law. If you need assistance with the application process or require an accommodation, please contact us at View email address on click.appcast.io. #J-18808-Ljbffr Fetch