Tech Lead Machine Learning Engineer - LLM Evaluation & Agent Systems

TikTok USDS JV is looking for a Senior Machine Learning Engineer technical leader to help build an AI Risk & Compliance Intelligence Platform and an Intelligent Data Agent Ecosystem. This onsite role in Seattle, WA combines architecture and execution across LLM evaluation, synthetic data generation, and modular agent systems that support Data, Engineering, and SRE workflows. You will work within the TikTok U.S. Data Security (USDS) team, which focuses on the security, integrity, and compliance of TikTok's data in the United States. In parallel, collaboration with the Data Platform Team will support scalable, reliable, and cost-efficient infrastructure for self-service tools and scientific insights. Responsibilities Design, build, and scale robust LLM evaluation architecture , including automated offline datasets, metric suites, and benchmarking pipelines to assess prompt and model performance. Develop LLM-driven synthetic data generation approaches to address data scarcity bottlenecks and enable systematic evaluation of long-tail and edge-case risk scenarios. Move beyond criteria-based prompt engineering by extracting risk patterns from complex case data to create native, ML-driven decision models and reasoning abstractions. Architect, evaluate, and continuously optimize modular agent skills and tools tailored for Data Science, Analytics, Data Engineering, and SRE workflows, improving tool-use accuracy, execution efficiency, and task success rates. Apply advanced evaluation methodologies (including LLM-as-a-judge and multi-agent benchmarking) across both risk intelligence and agentic ecosystems. Provide technical leadership by defining core ML system design patterns, benchmarking standards, and engineering best practices, and by translating complex operational requirements into scalable ML solutions with cross-functional partners. Requirements 5+ years of software or ML engineering experience, including a strong track record designing, building, and deploying ML/LLM systems in production. Hands‑on expertise with LLM architectures , fine‑tuning, RAG, prompt tuning, and agentic frameworks. Experience building robust LLM evaluation frameworks (LLM-as-a-judge, automated benchmark pipelines) and working on data scarcity or imbalance problems. Strong coding skills with production experience supporting scalable data processing pipelines. Prior domain experience in Risk & Compliance Intelligence and Automated Decision Systems. Hands‑on experience with synthetic dataset generation or bootstrapping models for cold‑start scenarios. Experience building AI tools or agents for technical personas including Data Analysts, Data Scientists, Data Engineers, and Data SRE. Demonstrated ability to serve as a founding technical lead on ambiguous, high-impact initiative pods. Location and Compensation Location: Seattle, WA (onsite) Salary: USD 198,360 - 416,100 per year Benefits Day one access to medical, dental, and vision insurance 401(k) savings plan with company match Paid parental leave Short-term and long-term disability coverage Life insurance Wellbeing benefits, among others 10 paid holidays per year 10 paid sick days per year 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure) About USDS TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. On-site presence across teams is intended to improve speed, alignment, and agility, particularly for real-time decision-making, team development, and integrated execution. The company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week. USDS Reasonable Accommodation USDS is committed to providing reasonable accommodations in its recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs, or other reasons protected by applicable laws. #J-18808-Ljbffr TikTok USDS JV