Researchers · Interviewed at NeurIPS 2025

Shivalika Singh

Cohere

NeurIPS 2025 in San Diego. The Leaderboard Illusion: How LLM Rankings Are Gamed

NeurIPS 2025 · December 20, 2025 · 17 min

NeurIPS 2025 in San Diego. The Leaderboard Illusion: How LLM Rankings Are Gamed

#NeurIPS has become the place where the future of AI evaluation is debated -- and this paper sparked one of the strongest discussions this year.

In this talk, Shivalika Singh (Research Engineer @CohereAI ) presents "The Leaderboard Illusion", a #NeurIPS2025 paper that takes a systematic look at Chatbot Arena, the widely used platform for ranking #LLM. The study reveals how leaderboard positions can be influenced by private variant testing, uneven sampling rates, and selective deprecations, showing that top spots aren't always just about model strength.

More from NeurIPS 2025

All event interviews →