Strategic Projects Lead

About Pareto Humanity is in a virtuous cycle: human insight improves AI, and better AI expands what people can do. Sustaining it depends on the one input that can’t be automated: expert human judgement. At Pareto, we build the platform that turns that judgement into the data, evals, and RL environments frontier models learn from. We work with leading frontier labs like Anthropic and GDM, and we give skilled people everywhere a way to shape the future of AI and share in what it creates. This RL environment and human‑data infrastructure is already in production. Our job now is to scale it. About This Role Pareto builds human training data pipelines for frontier AI labs. As a Strategic Projects Lead, you sit at the centre of that work — owning the architecture, execution, and continuous improvement of complex data collection and evaluation workflows from first scoping call to final delivery. This is a technical operations role, not a project management role. You’ll be expected to read code, deep dive data, reason about LLM internals, design evaluation frameworks, and — increasingly — deploy and iterate on AI agents to automate the work your pipelines do today. You’re actively building toward a model where agentic systems handle quality gates, expert routing, and output review, and SPLs are the people designing and operating those systems. You’ll work directly with AI researchers and technical program managers at our client organisations, own delivery against model performance benchmarks, and lead a team of project managers who handle day‑to‑day execution tracking. What You’ll Do Pipeline architecture — Design end‑to‑end data collection and evaluation pipelines for RLVR, RLHF, SFT, red‑teaming, and model evaluation workflows. Includes expert sampling strategy, annotation schema, rubric structure, inter‑rater calibration, and QA system design. Prototype novel workflows quickly, identify architectural risks before launch, and make trade‑off decisions with confidence. Understand how agents interact with tools to solve expert‑driven tasks, and communicate with engineering to ensure the environment is built accordingly to enable such tasks. Agentic system deployment — Build, test, and iterate on AI agents that automate pipeline tasks (quality gate review, expert matching, output flagging, throughput anomaly detection). Work closely with engineering to scope agent capabilities, write prompts and evaluation logic, and monitor their performance in production. Comfort with agentic tooling (LangChain, DSPy, etc.) is a differentiator. Quality systems — Define data quality standards across annotation, evaluation, and expert output review. Design and run audits using inter‑rater reliability metrics, calibration sets, and statistical sampling. Build systems that prevent quality issues—automated checks, structured output validation, model‑assisted review layers. Spot‑check tasks and understand what makes a datapoint meaningful & high quality. Translate findings into clear feedback and expert guidelines. Client interface — Engage directly with AI researchers, TPMs, and PMs at client organisations. Translate research‑driven requirements into operational workflows. Communicate pipeline performance, raise technical risks early, contribute to project scoping and pricing decisions. Research integration — Stay current with developments in LLM post‑training, evaluation methodology, and data tooling. Evaluate new approaches—model‑assisted annotation, structured output formats, automated calibration methods—and integrate them into active pipelines where they improve quality or efficiency. What You’ll Need Proficiency in Python and SQL for data manipulation, pipeline monitoring, and quality analysis — comfortable writing scripts to parse formats, run statistical checks, and build lightweight tooling. Working knowledge of LLM internals: RLHF/SFT training loops, prompt structure impact, RL environment setup with tool use for agentic data collection / eval projects. Hands‑on experience with at least one agentic or LLM workflow framework (LangChain, DSPy, AutoGen, direct tool‑use via API, or equivalent). Demonstrated ownership of a data or ML pipeline from scoping through delivery, including quality design, not just throughput tracking. Strong written communication: write technical guidelines and rubrics for distributed expert workers, and brief senior researchers on pipeline performance. Comfort operating with ambiguity in a fast‑moving environment where model requirements shift and client priorities evolve. You’ll stand out if you have Direct experience with RL environment data pipelines, evaluation framework design, and red‑teaming workflows. Background in data engineering, ML research support or equivalent. Experience designing or operating agentic systems in a production or near‑production context. Familiarity with inter‑rater reliability methods, calibration set design, and annotation quality frameworks. Prior client‑facing or technical program management experience in an AI/ML‑adjacent context. Prior experience on scoping or driving projects with fuzzy upfront specs or evolving requirements. What We Value In Candidates We care less about credentials than about demonstrated ability to own complex technical work and build toward better systems. A strong background in software engineering, data science, or ML research is the most common path into this role, but we’ve also seen excellent SPLs come from ML operations, computational linguistics, and applied research support. What matters is that you can read a dataset, reason about what’s wrong with it, write code to fix it, and design a workflow that prevents the problem next time. #J-18808-Ljbffr SoGal Ventures