Computational Linguist, AI Evaluation

Who We Are: Video is 90% of the world's data. Most of it is invisible to machines. TwelveLabs builds the intelligence layer to change that. Our multimodal AI models understand video the way humans do — across sight, sound, and motion — and power production-scale AI workloads across media, entertainment, sports, security, and government. We have raised more than $210 million from NEA, Radical Ventures, Amazon, NVIDIA, Snowflake, Databricks, Index Ventures, NAVER Ventures, Korea Investment Partners, Quadrille Capital, Red Bull Ventures, and AI pioneers including Fei-Fei Li, Silvio Savarese, and Alexandr Wang. We are a global company, headquartered in San Francisco with offices in Seoul, New York, and London, and employees around the world. We believe the differences in our cultural, educational, and life experiences make our products stronger. Building technology that understands the world in all its complexity requires people who see it from every angle. We are looking for individuals who are driven by hard problems and want their work to matter. Come build it with us! About the Role: You will be a vital member of our ML Data Operations Team – which leads the full spectrum of video-language data collection, labeling operations, and model quality measurement. This role comes with high ownership and includes responsibilities such as defining dataset and evaluation requirements in consultation with our research and product teams; designing and building data pipelines; and coordinating with our vendor partners that execute at scale. You will also be responsible for automating as much of the repetitive partnership and annotation-quality-evaluation work as possible. A desire to work cross functionally and to build relationships is critical for success in this position. Position is hybrid in San Francisco - onsite Tuesdays & Thursdays. In this role, you will: Evaluation Strategy & Design: Define and build evaluation protocols for video-language model quality, working with Research and Product teams to translate ambiguous quality questions into measurable, repeatable benchmarks or evaluation flows. Data-to-Insight Pipelines: Turn raw customer and usage data into structured signal, building pipelines and analyses that surface where our models are underperforming and working with XFN partners to translate these insights into concrete improvement plans. Labeling Operations: Design and execute video-language data collection and labeling projects, automating repetitive processes so the team can focus on higher-leverage work. Vendor & Partner Collaboration: Coordinate with vendor and outsourcing partners executing at scale, keeping quality high through clear guidelines and feedback loops. Cross-Functional Partnership: Work closely with Engineering and Research teams to align on top-priority data and evaluation needs, communicating findings through dashboards and reports that drive decisions. You may be a good fit if you have: Direct experience building or running model evaluation pipelines (benchmark design, human eval frameworks, automated scoring systems), and translating ambiguous quality questions into measurable criteria. 5+ years of experience working in an AI focused data operations organization. A proven track record designing and executing large scale data projects, including gathering, labeling, and post-processing data. The ability to analyze messy and complex data, identify overarching patterns, and distill your findings into crisp annotation guidelines or other accessible documentation. Proficiency with Python, agentic coding, or other popular industry tools for automation. Excellent communication and project management skills, and the ability to support several projects simultaneously. A foundational understanding of and interest in LLMs/VLMs and multimodal AI. Conviction that data is the key ingredient for the performance of AI models. Preferred Qualifications: Experience in data collection and labeling for multimodal language models. Experience working with research scientists and engineers. Experience planning and operating new data tools and labeling systems. Expertise or interest in video-centric domains, such as sports, advertising, and content creation. Tech Stack: Technical: Python, FFMPEG, AWS (S3, EC2) Project Management: Linear, Notion, Google Suite, Encord Benefits and Perks: An open and inclusive culture and work environment Work closely with a collaborative, mission-driven team on cutting-edge AI technology Full health, dental, and vision benefits Extremely flexible PTO and parental leave policy. Office closed the week of Christmas and New Years Monthly wellness stipend Annual Learning & Development stipend to invest in your growth Global offices in San Francisco and Seoul, and co-working office memberships for remote team members Transportation stipend (SF onsite/hybrid only) Daily lunch & dinner provided (SF onsite/hybrid only) #J-18808-Ljbffr Twelve-Labs