AI Operations Specialist

AI Operations Specialist Remote

12 months

A typical day

• Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
• Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
• Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it's live.
• Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
• Support product teams in agreeing on what "good enough to launch" means for an AI feature, and bring the evidence for the go/no-go call.
• Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
• Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
• Run labeling at scale with vendor agents, and check their work against agreement metrics and your own spot checks.
• Make what you build reusable and document what you learn, so the next team gets to production faster.
• Keep our methods current as models, tools and vendor agents change, and retire what stops working.
• Own the workstreams you're assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.

Skills

• 5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
• You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
• You're comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven't used them yet, you'll pick them up quickly with AI assistance.
• You're curious about how work actually gets done, including the awkward exceptions, and you've taken a messy process and made it measurable.
• You've worked on quality in some form, such as review guidelines, QA or audits.
• You write clearly and can work with business leads, PMs and engineers in the same week.
• You're fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn't required.
• You don't need to have designed evaluations or calibrated LLM judges before. We'll teach you, and you'll work with a team that has done it many times.

Preferred Experience

• Hands-on SQL, Python or API experience.
• Trust and safety, content moderation, annotation operations or other human-in-the-loop work at scale.
• Fine-tuning models or calibrating LLM judges.
• Background in a two-sided marketplace, travel, or another operations-heavy business.
• Experience setting up intake and prioritization for a new team.

We offer competitive compensation and benefits including medical, dental, vision, and retirement.

The pay range for this role is $39 - $75 per hour based on qualifications and experience.