Dissecting narrative construction and semantic search in LLMs with an entorhinal-hippocampal model

The entorhinal-hippocampal system is widely hypothesized to implement navigation over arbitrary cognitive and task spaces. Originally evolved in early nocturnal mammals for path integration and hidden state inference during spatial navigation, it has since been repurposed by cortical systems to support higher-level functions such as abstract reasoning and planning. In humans, intracranial recordings have implicated cortico-hippocampal sharp-wave ripples (SWRs) in the rapid binding of distributed information required for coherent narrative generation, although the computational principles linking spatial navigation mechanisms to language-based abstraction remain unclear. In parallel, large language models (LLMs) demonstrate strong performance across a wide range of language-based reasoning tasks but operate as largely opaque systems with poorly understood internal representations. Here, we bridge these perspectives by integrating mechanistic interpretability methods with entorhinal-hippocampal modeling to analyze language production in LLMs performing controlled narrative construction tasks, in which semantically disparate elements must be integrated into coherent stories, analogous to intracranial human experiments. Building on sparse autoencoder approaches for interpreting transformer residual streams, we introduce a biologically inspired framework that imposes structured inductive biases on latent dynamics. First, we extract a self-supervised generalized velocity signal from the model’s activity across layers during chain-of-thought (CoT), which drives a continuous attractor network with twisted toroidal topology, yielding entorhinal grid-cell-like representations in semantic space. These representations are then projected into a regularized softmax autoencoder that, akin to the hippocampus, discretizes continuous trajectories into a discrete set of latent states, inducing an interpretable hidden Markov model over state transitions. This formulation reveals latent inferential and semantic search processes underlying narrative generation, reframing transformer dynamics as structured probabilistic hierarchical state machines. Finally, we extend the framework beyond interpretability by incorporating an entorhino-hippocampal-inspired memory module in the form of retrieval-augmented generation (RAG) to support reasoning during CoT. We evaluate whether this addition enhances the integration of semantically distant concepts and improves narrative coherence and quality.