Reading Makes You Smart: Why Next-Token Prediction Builds World Models

Predicting the next word is compression, and compression is intelligence — so reading is literally how brains build world models.

The core idea is a three-step chain. Step one: any system that assigns probabilities to the next token in a sequence is, mathematically, a lossless compressor. This isn't a metaphor — it's an exact equivalence via arithmetic coding, a classical result from information theory. The number of bits a model "spends" predicting a sequence equals the number of bits needed to transmit that sequence using that model as a code. Better prediction, shorter code. Same thing.

Step two: across modern large language models, compression rate on held-out text correlates almost linearly with performance on downstream tasks — reasoning, factual recall, code generation. A model that compresses 1% better is measurably smarter across the board. This empirical finding, from Huang et al., has a clean theoretical explanation: to compress a corpus containing physics, history, fiction, and code, you can't rely on memorization or shallow pattern-matching. You have to recover the latent structure of the world that produced the text. The compression objective forces world-model construction.

Step three: human readers do the same thing. Psycholinguistic research going back to Hale and Levy shows that reading time on a word is proportional to its surprisal — how unexpected it was given context. More surprising words take longer to process. Crucially, the better the language model used to estimate surprisal, the better it predicts human reading times and even fMRI/EEG brain responses. Goldstein et al. found that GPT-2's moment-to-moment computations during story processing match ECoG recordings from human listeners, including a continuous next-word prediction signal and an error response that scales with surprisal. The cortical language network is running a predictive coding loop that is computationally homologous to LLM training.

The paper formalizes this as a proposition: a reader whose cross-entropy on a long, rich text stream approaches the algorithmic information content of that stream necessarily shares algorithmic mutual information with the world process that generated it. In plain terms — attentive reading over diverse, high-quality text is gradient descent toward a compressive model of the world. That accumulated model is what psychologists call crystallized intelligence. The paper also flags real limits: skimming without engagement produces no training signal; reading only fiction builds a model of fiction's authorial process, not the external world; and reading without ever acting leaves the model unverified. Perhaps most pointed for the present moment — if the corpus increasingly consists of LLM-generated text, the "world process" behind it drifts away from the actual world, and the MAI bound still holds but the thing being recovered is less and less worth knowing.