Welcome to
Women in AI Research

Can a language model learn a language no human ever could? Stanford PhD researcher Julie Kallini put Noam Chomsky's claim to the test, and won the ACL 2024 Best Paper Award for it. In this episode of Women in AI Research, Julie takes us from "impossible languages" to the hidden costs of tokenization, and shows how byte-level models could make LLMs faster, fairer across languages, and smarter about where they spend compute. We also talk about choosing a PhD advisor, leaving a software engineering job at Meta for a PhD, and what doing research at FAIR is really like. In this episode: → Why it would be unethical to raise a child on an invented language, but not a language model → Why linguists still can't agree on what makes a language "impossible" → Why ChatGPT couldn't count the R's in "strawberry", and what tokenization has to do with it → How tokenization can make the same request cost more in some languages than others → MrT5: teaching a byte-level model to decide which bytes it can safely drop → Fast BLT: speeding up byte-level generation with diffusion and self-speculation → Diffusion vs. autoregressive generation: practical advice for practitioners → What's next: adaptive compression, or why spend dozens of tokens on the MIT license? 📄 Papers discussed • Mission: Impossible Language Models (ACL 2024, Best Paper): arxiv.org/abs/2401.06416 • MrT5: Dynamic Token Merging for Efficient Byte-level Language Models (ICLR 2025): arxiv.org/abs/2410.20771 • Fast Byte Latent Transformer: arxiv.org/abs/2605.08044 • Byte Latent Transformer: Patches Scale Better Than Tokens (Meta FAIR): arxiv.org/abs/2412.09871 • Information Locality as an Inductive Bias for Neural Language Models (Someya et al., ACL 2025): arxiv.org/abs/2506.05136 👩‍🔬 About Julie Kallini Julie is a fourth-year Computer Science PhD student at Stanford, advised by Chris Potts and Dan Jurafsky. Her research spans computational linguistics, tokenization, efficient language model architectures, and byte-level modeling. ⏱️ Chapters 00:00 Finding the right PhD advisor 05:41 Painting as a creative outlet 07:40 From Meta software engineer to Stanford PhD 13:32 Can language models teach us how humans learn language? 18:57 Mission: Impossible Language Models (ACL 2024 Best Paper) 24:16 What an "impossible language" looks like 27:44 Why linguists can't define "impossible" 30:25 Could you build impossible languages from scratch? 33:30 From linguistics to tokenization 36:40 The hidden costs of subword tokenization 41:29 MrT5: dynamic token merging for byte-level models 46:26 How MrT5 learns what to delete 50:10 How close are byte-level models to everyday use? 52:37 Fast Byte Latent Transformer: diffusion & self-speculation 58:41 Research scientist at FAIR vs. academia 1:02:12 Diffusion vs. autoregressive: advice for practitioners 1:05:13 What people overlook about tokenization 1:08:48 What's next: adaptive compression 🎙️ About WiAIR Women in AI Research (WiAIR) features in-depth conversations with women shaping AI research: their work, their ideas, and the paths that led them there. 🔔 Subscribe for new episodes #AIResearch #Tokenization #LLM

Why don't bigger LLMs look more like the human brain? In the brain's language network, today's large models explain only slightly more variance than GPT-2 XL - and the reason says a lot about what those brain regions actually do. Dr. Greta Tuckute (Research Fellow at the Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard) joins Jekaterina Novikova on Women in AI Research to unpack what brain-LLM alignment does and does not tell us. Greta works where neuroscience, cognitive science and AI meet - and when asked to keep only one of those three labels, AI is the first one she drops. The conversation covers how brain alignment develops over training, how sparse autoencoders can turn the "you are just comparing one black box to another" critique into something testable, and what memory experiments reveal about how meaning is stored - from why "pineapple" sticks in memory and "light" does not, to which sentence embeddings predict what people remember. In this episode: - Brain alignment tracks formal linguistic competence, not reasoning - and it emerges after not much more than a developmentally plausible ~100M tokens, not 300B - Why better next-word prediction stops meaning "more brain-like" once a model has mastered language - Sparse autoencoder features plus surprisal: for one frontal brain region, surprisal alone does almost as well as 32,000 SAE features, while some voxels are captured by just six features - Brains and LLMs share the main, high-variance features of language - not the idiosyncratic ones - Why learning from BPE tokens puts brain-LLM comparisons on "pretty shaky ground", and what should come next - A distinctive meaning makes words and sentences memorable - and SBERT predicts human sentence memory better than the other embedding models tested - From running a photography business at 14 to a PhD at MIT, and why trying the other path first was worth it CHAPTERS 00:00 Preview 01:30 Greta Tuckute. Growing up between languages and countries 04:53 Neuroscience, AI or cognitive science: why AI is the label she'd drop 11:10 From a photography business at 14 to a research career 16:54 The unicorn demo that pulled her into language neuroscience 19:05 Why AI researchers should care how brains process language 21:32 From Language to Cognition (EMNLP 2025): alignment across training 24:40 You don't need 300B tokens to get a brain-like LLM 25:57 How LLMs outgrow the human language network 27:39 What "brain alignment" actually measures 31:16 Why better next-word prediction stops meaning more brain-like 33:39 Bigger models, barely better brain predictions 36:27 Isn't this just comparing two black boxes? 38:57 One number vs 32,000 SAE features: the surprisal result 41:35 Surprisal, and why fMRI can't resolve single words 44:58 Do brains and LLMs share the big features or the quirks? 50:23 What makes a representation "good" - and the BPE token problem 54:24 Memory as a window into meaning 57:14 Pineapple vs light: why some words stick 59:59 Misremembering her own paper title (which proves the point) 1:01:09 "Olive oil works well for tanning": what makes a sentence memorable 1:03:33 Which embeddings predict what people remember? 1:06:29 Can unusual syntax make a sentence memorable? 1:09:22 Open question: how do individual brains differ? 1:11:55 From static snapshots to dynamic transformations 1:12:55 What the answers would change, and where AI still struggles 1:15:04 Wrap-up PAPERS DISCUSSED • From Language to Cognition: How LLMs Outgrow the Human Language Network - aclanthology.org/2025.emnlp-main.1237/ • Interpreting Brain Responses to Language with Sparse Features from Language Models - arxiv.org/abs/2606.06857 • Driving and suppressing the human language network using large language models - nature.com/articles/s41562-023-01783-7 • Intrinsically memorable words have unique associations with their meanings - doi.org/10.1037/xge0001742 • A distinctive meaning makes a sentence memorable - doi.org/10.1016/j.jml.2025.104700 MENTIONED IN THIS EPISODE • Anna Ivanova on formal vs functional linguistic competence (WiAIR): youtu.be/e36ryy0Dsdo GRETA TUCKUTE Website: tuckute.com Bluesky: bsky.app/profile/gretatuckute.bsky.social X: x.com/GretaTuckute Women in AI Research (WiAIR) is a podcast and YouTube channel where Jekaterina Novikova talks with women doing AI research about their work, the questions driving it, and the paths that brought them there. #LLM #Neuroscience #AIResearch #WiAIR #WiAIRpodcast

An image is not worth a thousand words - it's worth an indefinite number of them. So why do the metrics we use to evaluate AI-generated image descriptions still assume there's one correct answer? In this episode of Women in AI Research, I talk with Elisa Kreiss (Assistant Professor of Communication at UCLA, director of the Coalas Lab) about what happens when you actually test the metrics the field relies on, and why CLIPScore, one of the most widely used measures for scoring image descriptions, stops correlating with human judgment the moment you introduce context. We also get into why longer descriptions aren't necessarily more informative, what happens when you just ask a model to "be concise," and whether AI models trip over charts and graphs the same way humans do. Elisa's research sits at the intersection of linguistics, accessibility, and multimodal AI, and this conversation covers the full arc of her work, from the theoretical question of why humans never describe images the same way twice, to the practical question of what that means for building systems that actually work for blind and low-vision users. In this episode: - Why "context matters" is more radical than it sounds for image description evaluation - The hidden reason CLIPScore breaks down once context enters the picture - Why length is a bad proxy for information density - and what to use instead - What happens when you prompt a model to just "be concise" - Why charts and photos need completely different evaluation approaches - Whether AI models make the same mistakes as humans when reading data visualizations - What NeurIPS's Top Reviewer Award taught her about writing a genuinely useful peer review Chapters: 0:00 - "An image is NOT worth a thousand words" 1:22 - Meet Dr. Elisa Kreiss 2:11 - From Germany to Stanford: the move that shaped her research path 9:05 - Inside a NeurIPS Top Reviewer Award: what actually makes a review useful 14:31 - Introducing the vision-language translation problem 16:27 - The yellow banana problem: why word choice reveals hidden meaning 22:17 - Context Matters: rethinking what makes an image description "good" 27:10 - Why blind and low-vision users were left out of accessibility research 34:49 - CLIPScore's blind spot: it can't see context at all 40:48 - Can CLIPScore be patched to account for context? What she found 43:14 - When More Words Say Less: why longer descriptions can carry less information 47:29 - A new way to measure specificity, independent of length 51:15 - What happens when you just tell a model to "be concise" 54:22 - Charts vs. photos: a fundamentally different translation problem 1:00:21 - CHART-6: do models and humans fail at reading charts in the same way? 1:03:08 - Where models and humans actually diverge 1:05:20 - Why AI models still underperform on data visualizations 1:08:35 - Do our current metrics even work for charts? 1:11:05 - Is "visual understanding" one problem or several? 1:15:09 - The single biggest gap in how multimodal models handle the visual world Resources & Links: • Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics (aclanthology.org/2022.emnlp-main.309) • When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation (aclanthology.org/2026.acl-short.34/) • CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models (arxiv.org/abs/2505.17202) Elisa Kreiss is an Assistant Professor of Communication at UCLA and director of the Coalas Lab, where her work focuses on how humans and AI systems translate visual information into language - with a particular focus on accessibility and evaluation. Connect with Dr. Elisa Kreiss: • bsky.app/profile/elisakreiss.bsky.social • x.com/ElisaKreiss Women in AI Research (WiAIR) features in-depth conversations with researchers working on the hardest problems in AI. 🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI. WiAIR website: ♾️ women-in-ai-research.github.io Follow WiAIR at: ♾️ LinkedIn: linkedin.com/company/women-in-ai-research/ ♾️ Bluesky: bsky.app/profile/wiair.bsky.social ♾️ X (Twitter): x.com/WiAIR_podcast #AIResearch #MultimodalAI #NLProc #WiAIR #WiAIRpodcast