Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Research Scientist, Model Evaluation

Full-time

cohere

Who are we?Our mission is to scale intelligence to serve humanity. We’re training and deploying frontier models for developers and enterprises who are building AI systems to power magical experiences like content generation, semantic search, RAG, and agents. We believe that our work is instrumental to the widespread adoption of AI. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. We like to work hard and move fast to do what’s best for our customers. Cohere is a team of researchers, engineers, designers, and more, who are passionate about their craft. Each person is one of the best in the world at what they do. We believe that a diverse range of perspectives is a requirement for building great products. Join us on our mission and shape the future! Why this role?Evaluation is critical to making progress in scaling intelligence. As models continue to become superhuman in many real-world use cases, we must continue to develop new evaluation techniques that accurately reflect what models are already capable of, as well as set the agenda for what future models should be capable of. In this role, you are responsible for creating these next-generation evaluation methods and infrastructure to measure LLM progress. As a Senior Research Scientist, Model Evaluation, you will:

  • Create ambitious new evaluation benchmarks that push the limits of what our models can accomplish.
  • Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations.
  • Conduct research to advance the state-of-the-art in LLM evaluation methods, including training LLM judges; refining LLM-based data synthesis pipelines; and improving evaluation efficiency.
  • Build scalable and reusable tools for digging into model performance.
You may be a good fit if:
  • You enjoy rapidly building prototypes that demonstrate the boundaries of what LLMs are capable of, and you have developed resources to measure those capabilities.
  • You have spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality.
  • You are obsessive about rigorously measuring AI capabilities, and also about making sure your measurements actually align with the capabilities you care about.
  • You have strong software engineering skills.
If some of the above doesn’t line up perfectly with your experience, we still encourage you to apply! We value and celebrate diversity and strive to create an inclusive work environment for all. We welcome applicants from all backgrounds and are committed to providing equal opportunities. Should you require any accommodations during the recruitment process, please submit an [Accommodations Request Form]( and we will work together to meet your needs. Full-Time Employees at Cohere enjoy these Perks:
  • ? An open and inclusive culture and work environment
  • ?‍? Work closely with a team on the cutting edge of AI research
  • ? Weekly lunch stipend, in-office lunches & snacks
  • ? Full health and dental benefits, including a separate budget to take care of your mental health
  • ? 100% Parental Leave top-up for up to 6 months
  • ? Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • ? Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • ✈️ 6 weeks of vacation (30 working days!)
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Senior Research Scientist, Model Evaluation in San Francisco, CA vacancy
  • Contra Labs, based in San Francisco, seeks a researcher to own research engagements from scoping to delivery, turning ambiguous client...  ...and proposals. You will work with cross-disciplinary teams on evaluation, data methodology, and insight synthesis, producing client-ready... 
    Senior

    Contra

    San Francisco, CA
    2 days ago
  •  ...Responsibilities Design and build evaluation suites for manufacturing and...  .... Train and fine-tune models using evaluation results to...  ...Help establish the team’s research and evaluation practices....  ...of experience operating at a senior or staff level. Hands-on experience... 
    Senior
    Full time

    Clera

    San Francisco, CA
    1 day ago
  • TypeSafe AI is seeking a safety researcher to examine model behavior and help evaluate safety concerns within the team. You’ll build evals, document findings, and contribute to responsible research practices. During the six-week period, you’ll set up access, meet the team... 
    Suggested

    TypeSafe AI

    San Francisco, CA
    5 days ago
  • $77k - $202k

     ...Strategy& - Strategy Consulting Business Model Reinvention - Senior Associate, you will provide strategic...  ...competitive analysis and market research to inform strategic planning- Utilizing...  ...closely with team members. We evaluate these factors thoughtfully to establish... 
    Senior
    Full time
    H1b

    PwC

    San Francisco, CA
    3 days ago
  •  ...of all daily U.S. X-rays. Our research team brings experience from...  ...looking for an exceptional Research Scientist to push the frontier of vision-language models for medical imaging. You'll own...  ...-scale distributed training, evaluation, and handoff to post-training.... 
    Suggested
    Full time

    epsilon-health

    San Francisco, CA
    9 days ago
  •  ...accenture.com.We Are:Accenture's Enterprise Operating Model practice. We partner with Boards, CEOs, and other...  ...out the best in teams.The Work:Engage with senior clients on their biggest business challenges.Evaluate actionable solutions to these challenges, the value... 
    Senior
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    San Francisco, CA
    1 day ago
  •  ...accenture.com.We Are:Accenture's Enterprise Operating Model practice. We partner with Boards, CEOs, and other...  ...out the best in teams.The Work:Engage with senior clients on their biggest business challenges.Evaluate actionable solutions to these challenges, the value... 
    Senior
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    San Francisco, CA
    1 day ago
  • $200k - $320k

     ...Senior Research Scientist EliseAI is seeking world-class experts in Machine Learning and related...  ...machine learning and large language models, develop predictive models to enhance...  ...experiments, build prototypes, run large-scale evaluations, and drive the project through to... 
    Senior
    Work at office
    Relocation

    Eliseai

    San Francisco, CA
    4 days ago
  •  ...never been harder or more important. You'll research how to detect fraud and verify...  ...and manipulated images, understanding how models perform across populations, and finding...  ...images and emerging fraud attacks. Evaluate how reliably our models perform across different... 
    Senior
    Full time
    For contractors
    Internship

    Persona AI

    San Francisco, CA
    3 days ago
  •  ...developing next-generation multimodal AI models and a proprietary, high-efficiency...  ...role Sciforium is seeking an exceptional Senior Research Scientist specializing in advanced AI/ML to push...  ...architectures. Design, implement, and evaluate novel ML algorithms using frameworks... 
    Senior
    Flexible hours

    Sciforium

    San Francisco, CA
    2 days ago
  •  ...problem definition through experimentation, evaluation, deployment, and iteration. Translate...  ...clear ML objectives, data strategies, model approaches, and success criteria. Build...  ...shift, and workflow disruption. Communicate research findings and technical decisions clearly... 
    Senior
    Full time
    Remote work
    Flexible hours
    Shift work

    Rad AI

    San Francisco, CA
    2 days ago
  • Reflection, a research lab, seeks to advance open and accessible intelligence. You will conduct...  ...rigorous comparative analyses, build evaluation systems, and develop frameworks that...  ...applied teams to translate insights into model improvements. The role emphasizes high agency... 

    Reflection AI

    San Francisco, CA
    5 days ago
  • $134.5k - $265.1k

     ...an experienced SeniorQuantum Research Scientist -- Fault Tolerant Quantum...  ...quantum computing offering. As a Senior Research Scientist within...  ...software and research tools.Evaluate quantum approaches against relevant...  ...-body simulation, financial modelling and risk, or optimization.... 
    Senior
    Local area

    Deloitte

    San Francisco, CA
    8 hours ago
  • $136.44k - $166.75k

     ...AI platform for biotech R&D. Scientists use Benchling to design...  ...data, and run AI agents and models directly in their workflows....  ....You’ll build the datasets, evaluations, and systems that help close...  ...engineers, scientists, and external research partners.Desire to work in a... 
    Work at office
    Local area
    Monday to Friday
    Shift work

    Benchling

    San Francisco, CA
    23 hours ago
  • $250k - $300k

     ...other, come build with us at Crusoe.About This Role:The Senior Staff Software Engineer for the AI Model Lifecycle team will play a crucial role in building...  ..., and experiment management: versioning, lineage, evaluation, and reproducible fine-tuning at scale.What You’ll... 
    Senior
    Temporary work

    Crusoe

    San Francisco, CA
    1 day ago
  •  ...company at the forefront of advancing large language models (LLMs) and autonomous AI agents. The company develops sophisticated datasets, evaluation methodologies, and data infrastructure that help leading AI research laboratories improve model reasoning, behavior, and... 
    Immediate start

    Aperio Talent Solutions

    San Francisco, CA
    8 days ago
  • $165.6k - $207k

     ...provide high quality data and accelerate progress in GenAI research. We are looking for Research Scientists and Research Engineers with expertise in LLM post-training (SFT, RLHF, reward modeling) and evaluation. This role is on the evaluation pod within the GenAI Research... 
    Full time

    Scale AI

    San Francisco, CA
    1 day ago
  • $290.4k - $363k

     ...that power the most advanced models in the world. Our teams operate...  ...intersection of cutting-edge research, large-scale engineering, and...  ...the foundational research, evaluation methodologies, and agent/RL infrastructure...  ..., and deployed.As a Research Scientist Manager, you will lead a... 
    Senior
    Full time

    Scale AI

    San Francisco, CA
    8 hours ago
  • $150k - $200k

     ...creative preference datasets that power benchmarking, evaluation, and post-training for the world's leading AI models and applications. Our work helps define what "...  ...this role exists Contra Labs is expanding its research work with frontier AI labs across evaluation, post... 

    Contra

    San Francisco, CA
    5 days ago
  •  ...when those simulations can be trusted raises difficult questions about measurement, evidence, and generalization. As a Research Scientist in Evaluations, you will own an ambitious, open-ended research agenda with substantial uncertainty. The questions, methods, and path... 

    Tenera, Inc

    San Francisco, CA
    3 days ago
  • Tenera, Inc. seeks a Research Scientist in Evaluations to lead an ambitious, open-ended research program with substantial uncertainty. You will define questions, develop original approaches, and test ideas with scientific rigor, enjoying autonomy for independent work. You... 

    Tenera, Inc

    San Francisco, CA
    3 days ago
  •  ...We are hiring a commercially minded, operationally pragmatic Senior Manager to define how driverless trucking works as a scalable commercial...  ...own the design and evolution of Waabi’s driverless operating model across the full Life of a Load, with a particular focus on our... 
    Senior
    Full time
    Remote work

    waabi

    San Francisco, CA
    9 days ago
  • $162k - $175.7k

     ...Senior Scientist / Principal Scientist Induced Proximity Help shape Nurixs...  ...induced proximity research strategy. This individual will...  ...stabilizers and related mechanisms) Evaluate and integrate new and...  ...rational design and predictive modeling of ternary complex formation... 
    Senior
    Full time
    Local area

    Nurix

    Brisbane, CA
    7 days ago
  • $55 - $75 per hour

     ...About the opportunity Help evaluate AI-generated answers on cybersecurity tasks. You’ll read code, apply the assigned policy to each...  ...cybersecurity fundamentals, the ability to read code and spot mistakes in model output, and clear written reasoning. Experience with data... 
    Senior
    Hourly pay
    Full time
    For contractors
    Remote work

    Braintrust

    San Francisco, CA
    2 days ago
  •  ...efficacy, and safety data to make clear next-study and candidate decisions. Responsibilities Design, run, and interpret mouse studies evaluating therapeutic candidates across efficacy, pharmacology, and safety. Select appropriate dose, route, schedule, tissues, timepoints,... 
    Senior
    H1b
    Visa sponsorship

    Capable Labs

    San Francisco, CA
    5 days ago
  •  ...training and improving pre-trained models to be deployed into ChatGPT,...  ...behavior through human data, evaluations, reward models, and post-...  ...sits at the intersection of research, product, and model design. We...  ...As a Research Engineer / Scientist, you will research and develop... 
    Full time
    Work at office
    Relocation package

    OpenAI

    San Francisco, CA
    9 days ago
  • $77k - $202k

     ...Strategy& - Strategy Consulting Business Model Reinvention - Senior Associate The Opportunity As a Strategy& - Strategy Consulting Business...  ...objectives Conducting competitive analysis and market research to inform strategic planning Utilizing data analytics to... 
    Senior

    PwC (US)

    San Francisco, CA
    2 days ago
  • $204k - $216k

     ...from and reasons over. You design the data models, schemas, and representations that...  ...communities, without fragmenting. The Senior AI/ML Data Model Architect owns that foundation...  ...graph, applied AI, infrastructure, and research on shared representations. Align teams... 
    Senior
    Local area

    Sapience AI Corporation

    San Francisco, CA
    14 days ago
  • $192.2k - $260k

     ...Sr. Applied Scientist Amazon is seeking an exceptional Sr. Applied...  ...insufficient. You will lead research that translates cutting-edge...  ...modalities Develop deep learning models for object detection,...  ...calibration to model training, evaluation, and real-time deployment... 
    Senior
    Local area
    Flexible hours
    Night shift

    Amazon Technologies, Inc.

    San Francisco, CA
    5 days ago
  • Snorkel AI in New York, NY seeks a Staff or Senior Research Scientist to lead the development of frontier benchmarks and datasets. You will collaborate across research, data operations, product, and GTM to design, implement, and scale high-impact data solutions for real... 
    Senior

    Snorkel AI

    San Francisco, CA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Research Scientist, Model Evaluation. Be the first to apply!