Senior Project Lead

Who we are and what we do Contra Labs is a human-centered research lab focused on understanding how creatives use AI, how AI tools can better align with creative needs, and how to preserve the diversity and originality that defines great creative work in an era of increasing homogenization. We connect a global network of top creative talent with frontier AI labs and product companies. By grounding our research in real creative workflows, we help define what "good" looks like across design, video, imagery, and beyond. Our work powers benchmarking, evaluation, and post‑training for leading AI models and applications. Built on top of Contra, the professional network for independent creatives, Contra Labs turns human judgment into the training and evaluation layer that helps AI tools better serve the people who use them. Creatives are at the center of everything we do. Why this role exists Contra Labs is scaling across client engagements including post‑training dataset creation, benchmarking programs, and ongoing evaluation partnerships. We need someone who can own the research and delivery side of this work end‑to‑end. Projects span AI labs, creative tool companies, and research teams, each with different requirements, timelines, and quality bars. This person scopes the methodology, designs the evaluation protocols, spins up the right evaluators, manages quality throughout, and makes sure we deliver rigorous, actionable results. What you’ll do Design and lead research studies including annotation protocols, rubrics, and evaluation methodologies for creative and multimodal outputs Conduct and oversee qualitative and quantitative analysis, synthesizing findings into actionable insights and recommendations for clients Interface with creative domain experts, manage evaluator teams, and calibrate for consistent quality through real‑time course‑correction Develop and standardize annotation, data quality, and inter‑rater reliability protocols across projects Communicate research findings effectively to stakeholders through reports, presentations, and workshops Collaborate with engineering on tooling needs and contribute to Contra Labs’ positioning through case studies and benchmark publications Translate client goals and priorities into well‑scoped research plans, managing multiple concurrent projects with different timelines and requirements About you You have deep experience in research methods, both qualitative and quantitative. You know how to design a study, run it, analyze the data, and translate findings into clear, actionable directions You have taste. You understand creative quality intuitively and can build evaluation frameworks that capture it You are more interested in people than in technology for its own sake. You understand how humans judge creative work and can design systems that capture that signal faithfully You can context‑switch across projects and manage priorities in a fast‑moving environment. Design study thinking comes naturally to you High ownership, low ego, bias toward action. You ship v1, iterate, and do whatever it takes to deliver Experience Master's degree or PhD in HCI, cognitive science, psychology, design research, or a related field (or equivalent depth of industry experience) 3+ years of hands‑on experience conducting research: designing studies, running evaluations, analyzing qualitative and quantitative data, and presenting findings Experience with statistical methods including inter‑rater reliability, sampling, regression analysis, and significance testing Proficiency with data analysis tools (Python, SQL) for working with evaluation data at scale Experience in data labeling, annotation, human‑in‑the‑loop systems, or AI/ML evaluation workflows Track record managing multiple concurrent projects, including annotation, evaluation, or QA workflows Strong understanding of creative domains and what "good" looks like Bonus Experience with RLHF, LLM evaluation, benchmark design, or human‑in‑the‑loop systems Background in ethnography, UX research, or design study methodologies Startup or 0‑to‑1 experience Requirements Based in NYC, USA In‑office 5 days/week (Williamsburg) Total Comp Salary: $150,000-$200,000 USD + Equity Medical, Dental, Vision Benefits 401k Matching We will provide you with a company laptop on your start date Interview Process Interview with Recruiting Team (20 minutes) Interview with CEO & Co‑Founder (30 minutes) Technical Interview with Research/Methods Focus (45 minutes) Interview with VP of Product (45 minutes) Paid Case Study and Presentation (60 minutes) Note: Contra communicates with applicants through @contra.com domains only. We never ask for money from potential employees. For the latest job postings, visit Contra Careers. #J-18808-Ljbffr