Topics

Evaluation & Benchmarks

How do we know what a model can really do? These episodes look at how AI systems are measured: which benchmarks and metrics we rely on, what they miss, and how researchers design evaluations that hold up beyond the leaderboard.

6 episodes · 9 conference interviews

Is an Image Worth a Thousand Words? Hidden Failures of Multimodal Metrics, with Dr. Elisa Kreiss

Episode 24

Is an Image Worth a Thousand Words? Hidden Failures of Multimodal Metrics, with Dr. Elisa Kreiss

August 19, 2026 · 1 h 18 min · WiAIR Podcast • Season 2

An image is not worth a thousand words - it's worth an indefinite number of them. So why do the metrics we use to evaluate AI-generated image descriptions still assume there's one correct answer? In this episode of Women in AI Research, I talk with Elisa Kreiss (Assistant Professor of Communication at UCLA, director of the Coalas Lab) about what happens when you actually test the metrics the field relies on, and why CLIPScore, one of the most widely used measures for scoring image descriptions, stops correlating with human judgment the moment you introduce context. We also get into why longer descriptions aren't necessarily more informative, what happens when you just ask a model to "be concise," and whether AI models trip over charts and graphs the same way humans do. Elisa's research sits at the intersection of linguistics, accessibility, and multimodal AI, and this conversation covers the full arc of her work, from the theoretical question of why humans never describe images the same way twice, to the practical question of what that means for building systems that actually work for blind and low-vision users. In this episode: - Why "context matters" is more radical than it sounds for image description evaluation - The hidden reason CLIPScore breaks down once context enters the picture - Why length is a bad proxy for information density - and what to use instead - What happens when you prompt a model to just "be concise" - Why charts and photos need completely different evaluation approaches - Whether AI models make the same mistakes as humans when reading data visualizations - What NeurIPS's Top Reviewer Award taught her about writing a genuinely useful peer review Chapters: 0:00 - "An image is NOT worth a thousand words" 1:22 - Meet Dr. Elisa Kreiss 2:11 - From Germany to Stanford: the move that shaped her research path 9:05 - Inside a NeurIPS Top Reviewer Award: what actually makes a review useful 14:31 - Introducing the vision-language translation problem 16:27 - The yellow banana problem: why word choice reveals hidden meaning 22:17 - Context Matters: rethinking what makes an image description "good" 27:10 - Why blind and low-vision users were left out of accessibility research 34:49 - CLIPScore's blind spot: it can't see context at all 40:48 - Can CLIPScore be patched to account for context? What she found 43:14 - When More Words Say Less: why longer descriptions can carry less information 47:29 - A new way to measure specificity, independent of length 51:15 - What happens when you just tell a model to "be concise" 54:22 - Charts vs. photos: a fundamentally different translation problem 1:00:21 - CHART-6: do models and humans fail at reading charts in the same way? 1:03:08 - Where models and humans actually diverge 1:05:20 - Why AI models still underperform on data visualizations 1:08:35 - Do our current metrics even work for charts? 1:11:05 - Is "visual understanding" one problem or several? 1:15:09 - The single biggest gap in how multimodal models handle the visual world Resources & Links: • Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics (aclanthology.org/2022.emnlp-main.309) • When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation (aclanthology.org/2026.acl-short.34/) • CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models (arxiv.org/abs/2505.17202) Elisa Kreiss is an Assistant Professor of Communication at UCLA and director of the Coalas Lab, where her work focuses on how humans and AI systems translate visual information into language - with a particular focus on accessibility and evaluation. Connect with Dr. Elisa Kreiss: • bsky.app/profile/elisakreiss.bsky.social • x.com/ElisaKreiss Women in AI Research (WiAIR) features in-depth conversations with researchers working on the hardest problems in AI. 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow WiAIR at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research/ ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #AIResearch #MultimodalAI #NLProc #WiAIR #WiAIRpodcast

AI Safety Beyond Benchmarks --  Dr. Swabha Swayamdipta on Evaluation, Personalization, and Control

Episode 17

AI Safety Beyond Benchmarks -- Dr. Swabha Swayamdipta on Evaluation, Personalization, and Control

January 21, 2026 · 1 h 1 min · WiAIR Podcast • Season 2

As language models become more capable, the hardest questions are no longer just about performance, but about safety, interpretation, and control. In this episode of Women in AI Research, we speak with Swabha Swayamdipta, Assistant Professor of Computer Science at the University of Southern California and co-Associate Director of the USC Center for AI and Society. Swabha’s research examines how the design and deployment of language models intersect with real-world risks — from how models behave in unexpected ways to how seemingly technical choices can have broader societal consequences. We talk about AI safety from multiple angles: what it means when hidden inputs to models can sometimes be inferred from their outputs, why personalization introduces new trade-offs around privacy and user agency, and how assumptions about model behavior can quietly shape downstream harms. Rather than focusing only on accuracy or benchmarks, the conversation asks what kinds of evidence we actually need to trust these systems in practice. CHAPTERS 00:00 Swabha's Journey into NLP Research 04:25 Navigating Career Challenges and Building Networks 08:58 The Importance of AI Safety and Reliability 10:49 Addressing Security and Privacy Concerns in Language Models 13:41 Innovations in Language Model Inversion 20:33 Balancing Personalization and Privacy in AI 27:08 Incorporating Psychological Scaffolds in Language Models 30:19 The Duality of AI: Enhancing Human Decision-Making 32:17 AI in Social Issues: Addressing Homelessness 35:18 OATH-Frames: Analyzing Public Sentiment on Homelessness 46:40 Suicide Prevention: AI's Role in Critical Interventions 56:12 The Responsibility of AI Researchers: Balancing Capability and Safety REFERENCES 13:52 Better Language Model Inversion by Compactly Representing Next-Token Distributions (arxiv.org/abs/2506.17090) 27:18 Improving Language Model Personas via Rationalization with Psychological Scaffolds (ui.adsabs.harvard.edu/abs/2025arXiv250417993J/abstract) 35:21 OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants (arxiv.org/abs/2406.14883) 46:52 Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants (arxiv.org/abs/2508.18541) 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow us at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research/ ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #AISafety #LanguageModels #AIResearch #ResponsibleAI #WomenInAI #NLP #MachineLearning #AIAlignment #wiair #wiairpodcast

Do LLMs Understand Meaning? Neuroscience, Evaluation, and the Future of AI, with Dr. Maria Ryskina

Episode 15

Do LLMs Understand Meaning? Neuroscience, Evaluation, and the Future of AI, with Dr. Maria Ryskina

December 31, 2025 · 1 h 4 min · WiAIR Podcast • Season 1

Do large language models actually understand meaning — or are we over-interpreting impressive behavior? In this episode, we speak with Dr. Maria Ryskina, CIFAR AI Safety Postdoctoral Fellow at the Vector Institute for AI, whose research bridges neuroscience, cognitive science, and artificial intelligence. Together, we unpack what the brain can (and cannot) teach us about modern AI systems — and why current evaluation paradigms may be missing something fundamental. We explore how language models can predict brain activity in regions linked to visual processing, what this reveals about cross-modal knowledge, and why scale alone may not resolve deeper conceptual gaps in AI. The conversation also tackles the growing importance of interpretability, especially as AI systems become more embedded in high-stakes, real-world contexts. Beyond technical questions, Maria shares why community matters in AI research, particularly for underrepresented groups — and how diversity directly shapes the kinds of scientific questions we ask and the systems we ultimately build. CHAPTERS 00:00 Introduction - Dr. Maria Ryskina 02:42 Community Building in AI 10:40 Representation, Lived Experience, and AI Research 14:17 AI Meets Cognitive Neuroscience 23:04 Brain Alignment and Conceptual Meaning in Language Models 30:38 World Knowledge and the EWoK Framework 42:37 Evaluating What Models Know 51:15 Biological Inspiration, Alignment, and the Future of AI REFERENCES 11:41 Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf) 12:36 Stereotypes and Smut: The (Mis)representation of Non-cisgender Identities by Text-to-Image Models (aclanthology.org/2023.findings-acl.502) 17:38 Language models align with brain regions that represent concepts across modalities (arxiv.org/abs/2508.11536) 30:53 Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models (Paper - direct.mit.edu/tacl/article/doi/10.1162/TACL.a.38/133506, project - ewok-core.github.io) 49:13 Prompting is not a substitute for probability measurements in large language models (aclanthology.org/2023.emnlp-main.306/) 50:04 Auxiliary task demands mask the capabilities of smaller language models (openreview.net/forum?id=U5BUzSn4tD) 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow us at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research/ ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #ArtificialIntelligence #LanguageModels #AIResearch #Neuroscience #Interpretability #AIEvaluation #CognitiveScience #LLMs #AISafety #ResponsibleAI #WomenInAI #AIResearchers #ResearchPodcast #wiair #wiairpodcast

Unlocking LLM Reasoning, with Simeng Sophia Han

Episode 9

Unlocking LLM Reasoning, with Simeng Sophia Han

August 27, 2025 · 54 min · WiAIR Podcast • Season 1

How can we go beyond accuracy to truly understand large language models? In this episode of the Women in AI Research podcast, hosts Jekaterina Novikova and Malikeh Ehghaghi sit down with Simeng Sophia Han (PhD candidate at @yale, Research Scientist Intern at @meta , ex @googledeepmind, ex @amazon aws) to explore the future of 𝐋𝐋𝐌 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠, 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧, 𝐚𝐧𝐝 𝐞𝐱𝐩𝐥𝐚𝐢𝐧𝐚𝐛𝐥𝐞 𝐀𝐈. 🌟 What you’ll learn in this episode: • Why evaluating reasoning goes beyond correctness • How brain teasers uncover hidden strengths and weaknesses of LLMs • The importance of symbolic reasoning for complex problem solving • The role of mentorship and early research experiences in shaping careers • Why consistency in AI outputs is essential for building trust • How humans combine brute force and intuition — and what this means for AI ToC: 00:00 Introduction to LLM Reasoning and Evaluation 02:36 Simeng Sophia Han's Research Journey 06:26 Reflections on Early Research Experiences 11:25 Understanding LLM Reasoning Beyond Accuracy Metrics 16:16 Exploring Brain Teasers in LLM Reasoning 22:25 Example of Mathematical Problem Solving with Constraints 24:16 Cognitive Science Insights for Language Models 29:13 Challenges in Human-Written Reasoning Chains 32:16 Explaining the Black Box of LLMs 37:28 Consistency and Trustworthiness in AI Models 39:17 Symbolic Reasoning in LLMs 42:00 Neuro-Symbolic Reasoning Approaches 46:33 Future Directions in AI Research 49:11 Advice for Women in AI Research REFERENCES: 01:26 Simeng Sophia Han - Google Scholar profile (scholar.google.ca/citations?hl=en&user=D0dpploAAAAJ) 11:40 Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models (arxiv.org/abs/2505.10844) 25:35 HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning (arxiv.org/abs/2409.19381) 29:25 P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (arxiv.org/abs/2410.09207) 39:05 Folio: Natural Language Reasoning with First-Order Logic (arxiv.org/abs/2209.00840) 41:54 HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning (arxiv.org/abs/2409.19381) 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow us at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #LLM #AIResearch #ExplainableAI #Reasoning #aireasoning #MachineLearning #CognitiveScience #SymbolicReasoning #WiAIRPodcast #WiAIR

LLM Hallucinations and Machine Unlearning, with Dr. Abhilasha Ravichander

Episode 8

LLM Hallucinations and Machine Unlearning, with Dr. Abhilasha Ravichander

August 6, 2025 · 1 h 3 min · WiAIR Podcast • Season 1

In this episode of the Women in AI Research Podcast, hosts Jekaterina Novikova and Malikeh Ehghaghi engage with Abhilasha Ravichander lasharavichander.github.io/ to discuss the complexities of LLM hallucinations, the development of factuality benchmarks, and the importance of data transparency and machine unlearning in AI. The conversation also delves into personal experiences in academia and the future directions of research in responsible AI. CHAPTERS: 00:00 Episode Overview: LLM Hallucinations and Responsible AI 01:23 Introduction to Abhilasha Ravichander 03:24 Navigating Challenges in Research 16:33 Understanding LLM Hallucinations: Definitions and Implications 23:43 Factuality Benchmarks: Innovations in Evaluating LLMs 26:03 Innovative Benchmarks for Hallucination Detection 28:29 Implications of Hallucinations in LLMs 29:06 Atomic Units for Precise Verification 31:29 Classifying Hallucinations: Types A, B, and C 36:28 Mitigating Hallucinations in LLMs 39:14 Future Directions in LLM Architecture and Training 42:48 Tools for Data Control and Transparency 52:20 Machine Unlearning: Concepts and Challenges 59:02 Emerging Research Avenues in AI 01:01:11 Advice for Impactful Research REFERENCES: 01:24 Abhilasha Ravichander -- Google Scholar profile (scholar.google.ca/citations?user=6vLsKGsAAAAJ&hl=en&oi=ao) 25:05 WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries (arxiv.org/abs/2407.17468) 25:33 HALoGEN: Fantastic LLM Hallucinations and Where to Find Them (arxiv.org/abs/2501.08292) 30:02 FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (arxiv.org/abs/2305.14251) 34:05 What's In My Big Data? (arxiv.org/abs/2310.20707) 45:16 Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models (arxiv.org/abs/2503.12072) 54:15 RESTOR: Knowledge Recovery in Machine Unlearning (arxiv.org/abs/2411.00204) 54:39 Model State Arithmetic for Machine Unlearning (arxiv.org/abs/2506.20941) 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow us at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #LLMHallucinations #FactualityBenchmarks #MachineUnlearning #DataTransparency #ModelMemorization #ResponsibleAI #GenerativeAI #NLPResearch #WomenInAI #AIResearch #WiAIR #wiairpodcast

Generalization in AI, with Dr. Dieuwke Hupkes

Episode 7

Generalization in AI, with Dr. Dieuwke Hupkes

July 16, 2025 · 1 h 9 min · WiAIR Podcast • Season 1

A must-listen episode with Dr. Dieuwke Hupkes dieuwkehupkes.nl/, a research scientist at @meta AI Research, where we dive into AI generalization, LLM robustness, and model evaluation in large language models. We explore how LLMs handle grammar and hierarchy, how they generalize across tasks and languages, and what consistency tells us about AI alignment. We also talk about Dieuwke’s journey from physics to NLP, the challenges of peer review, and sustaining a career in research—plus, how pole dancing helps with focus 💪 REFERENCES: 00:28 Dieuwke Hupkes - Google Scholar profile (scholar.google.ca/citations?user=tAtSMTcAAAAJ&hl=en&oi=ao) 24:32 A taxonomy and review of generalization research in NLP (nature.com/articles/s42256-023-00729-y) 41:28 What's in My Big Data (wimbd.apps.allenai.org) 49:49 GenBench workshop (youtube.com/@GenBench-NLP , website - genbench.org) 52:30 Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses (arxiv.org/abs/2305.11662) 52:38 From form(s) to meaning: Probing the semantic depths of language models using multisense consistency (direct.mit.edu/coli/article/50/4/1507/123794) 01:02:44 MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages (arxiv.org/abs/2504.10356) 01:04:43 How much do language models memorize? (arxiv.org/abs/2505.24832) ToC: 00:00 Introduction to Dieuwke Hupkes and Her Journey 05:15 Navigating Challenges in Research 07:17 The Peer Review Process: Insights and Frustrations 16:23 Being a Woman in AI: Representation and Challenges 19:57 Balancing Research and Personal Life 23:37 Exploring Consistency and Generalization in Language Models 33:31 Generalization Across Languages and Modalities 35:15 Exploring Generalization Taxonomy 40:55 Challenges in Evaluating Generalization 44:12 Data Contamination and Generalization 50:43 Consistency in Language Models 57:23 The Intersection of Consistency and Alignment 01:01:15 Current Research Directions 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow us at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #LLMs #AIgeneralization #LLMrobustness #AIalignment #ModelEvaluation #MetaAIResearch #WiAIR #WiAIRpodcast

Researchers on this topic

From conferences and workshops

Interviews recorded on site

All topics