Researchers · Interviewed at COLM 2026
MIT

COLM 2026 · October 21, 2026 · 8 min
Why does adding code to pre-training make a model better at maths - and why do some data mixtures quietly hold performance back? Kimia Hamidieh (PhD student, MIT CSAIL) shows that the data scaling exponent itself depends on what you train on, and turns that into a way to measure data synergy and pick better mixtures before you spend the compute.
Recorded at her COLM 2026 poster, "Domain-Aware Scaling Laws Uncover Data Synergy" (with Lester Mackey and David Alvarez-Melis). We talk about: - why a single scaling law fails to fit models trained on different data mixtures - first-order synergy: which pre-training domains speed up or slow down progress on which benchmarks (maths helps coding; books slow it down) - second-order synergy: how pairs of domains interact, modelled as "bonus tokens" - using small-scale runs to predict and optimise the mixture for a target benchmark - where the findings hold beyond pre-training - and where RL post-training may break them - who gets to decide what counts as a "domain"