Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Research Scientist, Model Evaluation

Full-time

cohere

Who are we?Our mission is to scale intelligence to serve humanity. We’re training and deploying frontier models for developers and enterprises who are building AI systems to power magical experiences like content generation, semantic search, RAG, and agents. We believe that our work is instrumental to the widespread adoption of AI. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. We like to work hard and move fast to do what’s best for our customers. Cohere is a team of researchers, engineers, designers, and more, who are passionate about their craft. Each person is one of the best in the world at what they do. We believe that a diverse range of perspectives is a requirement for building great products. Join us on our mission and shape the future! Why this role?Evaluation is critical to making progress in scaling intelligence. As models continue to become superhuman in many real-world use cases, we must continue to develop new evaluation techniques that accurately reflect what models are already capable of, as well as set the agenda for what future models should be capable of. In this role, you are responsible for creating these next-generation evaluation methods and infrastructure to measure LLM progress. As a Senior Research Scientist, Model Evaluation, you will:

  • Create ambitious new evaluation benchmarks that push the limits of what our models can accomplish.
  • Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations.
  • Conduct research to advance the state-of-the-art in LLM evaluation methods, including training LLM judges; refining LLM-based data synthesis pipelines; and improving evaluation efficiency.
  • Build scalable and reusable tools for digging into model performance.
You may be a good fit if:
  • You enjoy rapidly building prototypes that demonstrate the boundaries of what LLMs are capable of, and you have developed resources to measure those capabilities.
  • You have spent dozens of hours reviewing complex data and LLM outputs to ensure high data quality.
  • You are obsessive about rigorously measuring AI capabilities, and also about making sure your measurements actually align with the capabilities you care about.
  • You have strong software engineering skills.
If some of the above doesn’t line up perfectly with your experience, we still encourage you to apply! We value and celebrate diversity and strive to create an inclusive work environment for all. We welcome applicants from all backgrounds and are committed to providing equal opportunities. Should you require any accommodations during the recruitment process, please submit an [Accommodations Request Form]( and we will work together to meet your needs. Full-Time Employees at Cohere enjoy these Perks:
  • ? An open and inclusive culture and work environment
  • ?‍? Work closely with a team on the cutting edge of AI research
  • ? Weekly lunch stipend, in-office lunches & snacks
  • ? Full health and dental benefits, including a separate budget to take care of your mental health
  • ? 100% Parental Leave top-up for up to 6 months
  • ? Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement
  • ? Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend
  • ✈️ 6 weeks of vacation (30 working days!)
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Senior Research Scientist, Model Evaluation in New York, NY vacancy
  • $295k

     ...company. We build cutting-edge foundation AI models and end-to-end products that are...  ...for our customers. Cohere is a team of researchers, engineers, designers, and more, who are...  ...Berlin and Seoul. Join us!Role Overview:Evaluation is critical to making progress in scaling... 
    Senior
    Full time
    Work at office
    Local area
    Remote work
    Home office

    Cohere

    New York, NY
    10 hours ago
  • $77k - $202k

     ...Strategy& - Strategy Consulting Business Model Reinvention - Senior Associate, you will provide strategic...  ...competitive analysis and market research to inform strategic planning- Utilizing...  ...closely with team members. We evaluate these factors thoughtfully to establish... 
    Senior
    Full time
    H1b

    PwC

    New York, NY
    3 days ago
  •  ...accenture.com.We Are:Accenture's Enterprise Operating Model practice. We partner with Boards, CEOs, and other...  ...out the best in teams.The Work:Engage with senior clients on their biggest business challenges.Evaluate actionable solutions to these challenges, the value... 
    Senior
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    New York, NY
    1 day ago
  •  ...accenture.com.We Are:Accenture's Enterprise Operating Model practice. We partner with Boards, CEOs, and other...  ...out the best in teams.The Work:Engage with senior clients on their biggest business challenges.Evaluate actionable solutions to these challenges, the value... 
    Senior
    Work experience placement
    Live in
    Work at office
    Local area

    Accenture

    New York, NY
    1 day ago
  • $193.3k - $261.5k

    As a Senior Forward Deployed Engineer (Senior FDE) on the Custom Model and Optimization team, you are the technical lead embedded...  ...success metrics and the evaluation bar, and set technical roadmap...  ...leadership to a team of FDEs and scientists on the engagement, breaking ambiguous... 
    Senior
    Internship
    Local area
    Flexible hours

    AmazonWebServices

    New York, NY
    4 days ago
  • $190k - $245k

     ...See yourself at Dataminr As a Senior Research Scientist at Dataminr, you will tackle some of the...  ...state of the art in large language models, generative AI, and agentic systems while...  ...and adaptation, distillation, evaluation and benchmarking, agentic systems, retrieval... 
    Senior
    Work at office
    Flexible hours

    Dataminr

    New York, NY
    3 days ago
  • $65k - $120k

     ...thousands of physicians and scientists who have helped to...  ...education, scientific research, and direct patient...  ...to join our team as a Senior Research Scientist within...  ...-centered data for evaluation of interventions and characteristics...  ...to complex modeling to interactive data... 
    Senior
    Full time

    NYU Langone Health

    New York, NY
    2 days ago
  • $120k - $140k

     ... Research -- Basic Science/Technical New York • ID: _...  ...thousands of physicians and scientists who have helped to shape the...  ...opportunity to join our team as a Senior Research Scientist. The...  ...epidemiology, health services research, evaluation, biostatistics, and medical... 
    Senior
    Full time

    NYU Langone Health

    New York, NY
    3 days ago
  • $172.8k - $359k

     ...a Forward Deployed Engineer Senior Manager, you will be joining...  ...and interdisciplinary team of scientists and engineers who love to tackle...  ...into a prioritized, scalable Model Engineering roadmap that...  ...that enable implementation, evaluation, governance, monitoring, and... 
    Senior
    Immediate start
    Remote work

    EY (Ernst & Young)

    New York, NY
    4 days ago
  • $200k - $320k

     ...strengthen our expansive research team. We are looking...  ..., and computer scientists—who are passionate about...  ...learning and large language models, develop predictive...  ...data, and more. As a Senior Research Scientist you...  ...prototypes, run large‑scale evaluations, and drive the project... 
    Senior
    Work at office
    Local area
    Relocation

    EliseAI

    New York, NY
    2 days ago
  • $135k - $150k

     ...Position: Senior Research Scientist Department: NYU McSilver Institute for Poverty Policy and Research Supervisor: Dr. Fred Ssewamala...  ...highly collaborative research environment. Oversee performance evaluations, professional development, and day-to-day workloads of... 
    Senior
    Full time
    Work experience placement
    Work at office
    Immediate start
    3 days per week

    New York University

    New York, NY
    3 days ago
  •  ...Description: As a Senior AI Research Scientist, you will bridge cutting-edge AI research with production...  ...focus on applying Large Language Models (LLMs), AI Agents, and Multimodal AI...  ...through implementation, deployment, evaluation, and continuous iteration. Build AI... 
    Senior
    Full time

    CTW

    New York, NY
    2 days ago
  • $174k - $252k

    Author research papers to share and generate impact of research results...  ..., framework, design, and evaluation metrics for research solution...  ...AI accelerators (TPUs, GPUs), model architectures (encoders,...  ...other researchers. As a Research Scientist on the Graph Neural Network (... 
    Senior

    Google

    New York, NY
    4 days ago
  • $134.5k - $265.1k

     ...an experienced SeniorQuantum Research Scientist -- Fault Tolerant Quantum...  ...quantum computing offering. As a Senior Research Scientist within...  ...software and research tools.Evaluate quantum approaches against relevant...  ...-body simulation, financial modelling and risk, or optimization.... 
    Senior
    Local area

    Deloitte

    New York, NY
    10 hours ago
  •  ...Job Description Job Title: Language Model Evaluator Job Summary We are seeking a Language...  ...collaborating with AI Engineers, Data Scientists, Prompt Engineers, and Product teams to...  ...evaluation Contributions to AI evaluation research or open-source projects Key... 
    Full time
    Remote work

    Ova Technologies

    New York, NY
    26 days ago
  • Mecka AI in New York seeks an experienced Research Scientist to advance large-scale vision-language and multimodal foundation models. You will work on model training, post-training...  ...across data, training strategies, evaluation, and optimization, exploring emerging approaches... 
    Senior

    Mecka

    New York, NY
    10 hours ago
  • $40 - $125 per hour

     ...Drug Discovery Scientist, AI Trainer (Remote, Flexible Hours)100% Remote | Work When You Want | Part-Time or Full-Time | Freelance...  ...discovery scientists to work as remote AI trainers and evaluate how frontier AI models handle real discovery and preclinical work: SAR analysis,... 
    Hourly pay
    Full time
    Contract work
    Temporary work
    Part time
    Freelance
    Remote work
    Work from home
    10 hours per week
    Flexible hours
    Afternoon shift

    Crucial Recruiting

    New York, NY
    a month ago
  • $60 - $90 per hour

     ...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include...  ...and Jack Dorsey . Position: Machine Learning Engineer — Model Evaluation & Experimentation Type: Contract Compensation: $60... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    4 days ago
  • $127.4k - $236.6k

     ...Reuters Labs builds and ships custom models for professional workflows...  ...more on the latest model release.The research team trains and evaluates the Thomson models to perform at a...  ...evaluation efforts.We are hiring scientists at Senior and Lead level across three areas... 
    Senior
    Full time
    Local area
    Flexible hours

    Thomson Reuters

    New York, NY
    2 days ago
  • $165.6k - $207k

     ...provide high quality data and accelerate progress in GenAI research. We are looking for Research Scientists and Research Engineers with expertise in LLM post-training (SFT, RLHF, reward modeling) and evaluation. This role is on the evaluation pod within the GenAI Research... 
    Full time

    Scale AI

    New York, NY
    1 day ago
  • $290.4k - $363k

     ...that power the most advanced models in the world. Our teams operate...  ...intersection of cutting-edge research, large-scale engineering, and...  ...the foundational research, evaluation methodologies, and agent/RL infrastructure...  ..., and deployed.As a Research Scientist Manager, you will lead a... 
    Senior
    Full time

    Scale AI

    New York, NY
    10 hours ago
  • $150k - $200k

     ...creative preference datasets that power benchmarking, evaluation, and post-training for the world's leading AI models and applications. Our work helps define what "...  ...this role exists Contra Labs is expanding its research work with frontier AI labs across evaluation, post... 
    Work at office

    Contra

    New York, NY
    4 days ago
  • $55 - $75 per hour

     ...About the opportunity Help evaluate AI-generated answers on cybersecurity tasks. You’ll read code, apply the assigned policy to each...  ...cybersecurity fundamentals, the ability to read code and spot mistakes in model output, and clear written reasoning. Experience with data... 
    Senior
    Hourly pay
    Full time
    For contractors
    Remote work

    Braintrust

    New York, NY
    2 days ago
  • $262.5k - $299.6k

    Applied Researcher 5 (AI Foundations: Evaluation and Agentic Workflows) At Capital One, we are creating trustworthy...  ...a cross-functional team of data scientists, software engineers, machine...  ...textual data. Build AI foundation models through all phases of development,... 
    Full time
    Part time
    Work at office
    Local area
    Flexible hours

    Capital One

    New York, NY
    2 days ago
  •  ...growth marketers, is seeking a Senior Marketing Scientist to drive the rigor,...  ...frameworks—spanning Econometric Modeling (MMM), Multi-Touch...  ...curves. Design, execute, and evaluate incrementality frameworks,...  ...forefront of marketing science research, continually testing new open... 
    Senior

    Mile Marker

    New York, NY
    18 days ago
  •  ...The Research & Intelligence team builds the machine learning and...  ..., temporal and causal modeling, data valuation systems, and...  ...data assets. We are seeking a Senior Applied Scientist to lead and expand key research...  ...standards for model evaluation, monitoring, and lifecycle... 
    Senior
    Work at office
    Local area
    Remote work

    Carbon Arc

    New York, NY
    3 days ago
  • $167.4k - $310.8k

     ...development. Roche’s Research and Early Development...  ...and novel computational models to drive impact. Seamless...  ...(AI) to assist our scientists in both pRED and gRED...  ...techniques. We are seeking a Senior or Principal Machine...  ...strategies, and evaluation methodologies.Model Capability... 
    Senior
    Full time
    Local area
    Worldwide
    Relocation package

    Genentech

    New York, NY
    10 hours ago
  •  ...Senior Applied ScientistAI assistants are only as...  ...found. Modern language models run throughout, and we...  ...for a Senior Applied Scientist to help drive this work...  ...data through modeling, evaluation and deployment, while...  ...product, ranking and research teams to take new techniques... 
    Senior

    Microsoft Corporation

    New York, NY
    5 days ago
  • $167.44k - $226.07k

     ...Senior Applied Scientist Siemens builds the systems the physical world runs...  ...intersection of machine learning research, real world data, and...  ...experimentation, and into models that run in production. You...  ...series, control, planning, or evaluation Take problems from... 
    Senior
    Local area

    Siemens

    New York, NY
    4 days ago
  • $183.8k - $248.7k

     ...Applied Scientist We are seeking an Applied Scientist to lead the development of evaluation frameworks and data collection protocols for robotic...  ...datasets aligned with model development needs Build and...  ...degree and 6+ years of applied research experience Experience in... 
    Senior
    Flexible hours

    Amazon Technologies, Inc.

    New York, NY
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Research Scientist, Model Evaluation. Be the first to apply!