Episode 24
Is an Image Worth a Thousand Words? Hidden Failures of Multimodal Metrics, with Dr. Elisa Kreiss
August 19, 2026 · 1 h 18 min · WiAIR Podcast • Season 2
An image is not worth a thousand words - it's worth an indefinite number of them. So why do the metrics we use to evaluate AI-generated image descriptions still assume there's one correct answer? In this episode of Women in AI Research, I talk with Elisa Kreiss (Assistant Professor of Communication at UCLA, director of the Coalas Lab) about what happens when you actually test the metrics the field relies on, and why CLIPScore, one of the most widely used measures for scoring image descriptions, stops correlating with human judgment the moment you introduce context. We also get into why longer descriptions aren't necessarily more informative, what happens when you just ask a model to "be concise," and whether AI models trip over charts and graphs the same way humans do. Elisa's research sits at the intersection of linguistics, accessibility, and multimodal AI, and this conversation covers the full arc of her work, from the theoretical question of why humans never describe images the same way twice, to the practical question of what that means for building systems that actually work for blind and low-vision users. In this episode: - Why "context matters" is more radical than it sounds for image description evaluation - The hidden reason CLIPScore breaks down once context enters the picture - Why length is a bad proxy for information density - and what to use instead - What happens when you prompt a model to just "be concise" - Why charts and photos need completely different evaluation approaches - Whether AI models make the same mistakes as humans when reading data visualizations - What NeurIPS's Top Reviewer Award taught her about writing a genuinely useful peer review Chapters: 0:00 - "An image is NOT worth a thousand words" 1:22 - Meet Dr. Elisa Kreiss 2:11 - From Germany to Stanford: the move that shaped her research path 9:05 - Inside a NeurIPS Top Reviewer Award: what actually makes a review useful 14:31 - Introducing the vision-language translation problem 16:27 - The yellow banana problem: why word choice reveals hidden meaning 22:17 - Context Matters: rethinking what makes an image description "good" 27:10 - Why blind and low-vision users were left out of accessibility research 34:49 - CLIPScore's blind spot: it can't see context at all 40:48 - Can CLIPScore be patched to account for context? What she found 43:14 - When More Words Say Less: why longer descriptions can carry less information 47:29 - A new way to measure specificity, independent of length 51:15 - What happens when you just tell a model to "be concise" 54:22 - Charts vs. photos: a fundamentally different translation problem 1:00:21 - CHART-6: do models and humans fail at reading charts in the same way? 1:03:08 - Where models and humans actually diverge 1:05:20 - Why AI models still underperform on data visualizations 1:08:35 - Do our current metrics even work for charts? 1:11:05 - Is "visual understanding" one problem or several? 1:15:09 - The single biggest gap in how multimodal models handle the visual world Resources & Links: • Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics (aclanthology.org/2022.emnlp-main.309) • When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation (aclanthology.org/2026.acl-short.34/) • CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models (arxiv.org/abs/2505.17202) Elisa Kreiss is an Assistant Professor of Communication at UCLA and director of the Coalas Lab, where her work focuses on how humans and AI systems translate visual information into language - with a particular focus on accessibility and evaluation. Connect with Dr. Elisa Kreiss: • bsky.app/profile/elisakreiss.bsky.social • x.com/ElisaKreiss Women in AI Research (WiAIR) features in-depth conversations with researchers working on the hardest problems in AI. 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow WiAIR at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research/ ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #AIResearch #MultimodalAI #NLProc #WiAIR #WiAIRpodcast

















