Researchers · Interviewed at NeurIPS 2025
Cohere

NeurIPS 2025 · December 20, 2025 · 17 min
#NeurIPS has become the place where the future of AI evaluation is debated -- and this paper sparked one of the strongest discussions this year.
In this talk, Shivalika Singh (Research Engineer @CohereAI ) presents "The Leaderboard Illusion", a #NeurIPS2025 paper that takes a systematic look at Chatbot Arena, the widely used platform for ranking #LLM. The study reveals how leaderboard positions can be influenced by private variant testing, uneven sampling rates, and selective deprecations, showing that top spots aren't always just about model strength.