Engineering Manager, LLM Evaluation

About Turing

Turing is a leading AI company accelerating the advancement and deployment of frontier AI systems. We work with the world’s top AI labs and enterprises to improve capabilities in reasoning, coding, agentic behavior, multimodality, and other advanced AI domains.

Role Overview

We are looking for an Engineering Manager to lead teams creating high-quality conversational data for training and evaluating advanced Large Language Models (LLMs). The role involves managing the design of realistic, multi-turn interactions between users and AI assistants, including tool-based workflows involving calendars, email, maps, cloud storage, and other applications.

The manager will oversee project execution, maintain quality standards, guide technical decision-making, and collaborate with engineering, operations, and review teams to deliver reliable training and evaluation datasets.

What You’ll Do

• Lead and manage teams responsible for creating and evaluating multi-turn LLM conversations.
• Define workflows, priorities, quality standards, and delivery goals for conversation data projects.
• Guide the design of realistic assistant interactions involving APIs, JSON, function calls, and external tools.
• Review complex scenarios involving decision-making, clarification, error handling, constraints, and task completion.
• Ensure conversations accurately reflect realistic user expectations and assistant capabilities.
• Establish quality-control processes and monitor team performance, consistency, and deliverable quality.
• Coach team members on technical reasoning, conversation design, tool usage, and adherence to project guidelines.
• Collaborate with engineering, product, operations, and quality teams to resolve issues and improve workflows.
• Analyze reviewer feedback and implement process improvements to increase realism, accuracy, and training value.
• Manage project risks, timelines, escalations, and stakeholder communication.

What We’re Looking For

• 5+ years of professional experience in software engineering, technical operations, data, AI, or a related technical field.
• Prior experience managing, mentoring, or leading technical teams.
• Strong understanding of APIs, JSON, data formats, logical reasoning, and structured workflows.
• Ability to guide teams in breaking down complex tasks into realistic conversational and tool-use flows.
• Strong understanding of assistant behavior, including tool selection, clarification, limitations, and error handling.
• Excellent written and verbal English communication skills.
• Strong attention to detail and ability to establish and enforce quality standards.
• Proven ability to manage multiple priorities, provide actionable feedback, and deliver projects within deadlines.
• Ability to collaborate effectively with cross-functional teams and external stakeholders.

Perks of Freelancing With Turing

• Work in a fully remote environment.
• Opportunity to work on cutting-edge AI projects with leading LLM companies.

Offer Details

• Commitments Required: At least 4 hours per day and minimum 40 hours per week with overlap of 4 hours with PST.
• Duration of Contract: 3 months (adjustable based on engagement)