Improve Language Model and Brain Alignment via Associative Memory

Congchi Yin, Yongpeng Zhang, Xuyun Wen, Piji Li · 2025

← Back to overview
Evidence (2)
Information Integration # Continue PAPER_TPL BIO
Simulating associative memory in input text increased model-to-brain alignment across distributed cortical regions, especially ROIs implicated in associative memory.
"Generally speaking, large and continuous regions of the brain, including some areas of frontal gyrus, frontal sulcus and parietal lobule gain increase in brain score ranging from 0.0014 to 0.02. ... Moreover, we find random data augmentation leads to none and even negative growth of brain score, which supports the improvement of alignment is caused by introducing associative memory."
5.2 Associative Memory Score, p. 991
Introducing associative content into the stimulus boosted distributed cortical alignment with language-model activations, consistent with information integration linking context and associated knowledge in the brain.
"After achieving temporal alignment between X(l) and Yi through f, we seek to find a linear model g ∈ R^d to map language model activations to brain activity. Ridge regression with ℓ2-regularization is learned to predict brain activity: ... Finally, ... brain score R(X(l)) is defined as correlation between predicted brain activity and original brain activity."
3.2 Brain Score Calculation, p. 988
The alignment metric operationalizes integration as a shared representational structure between model and distributed human fMRI responses, enabling quantitative detection of integration gains.
"To investigate whether the alignment between language model and human brain can be improved by associative memory, we don’t directly simulate its process in the brain. Instead, we concretize the content of associative memory ... as natural language input to language models."
3.3 Data Augmentation with Simulated Associative Memory, p. 989
By concretizing associative memory content in language input, the study tests whether added associations enhance integrated processing measurable in brain-model correlations.
Limitations: Associative memory was simulated as appended text rather than manipulated at the neural level; fMRI’s low temporal resolution and linear mapping may obscure mechanistic pathways of integration, and no causal brain interventions were performed.
Causal Control # Continue PAPER_TPL AI
Supervised fine-tuning (LoRA or frozen-layer) with instructions encouraging associative memory causally increased brain alignment in ROIs (e.g., MTL).
"As shown in Figure 6, LLaMA-2 after supervised fine-tuning with both methods shows 2% to 7% gain in regions related to associative memory (i.e. medial temporal lobe (MTL)), which indicates the alignment between language model and brain is improved by associative memory instructed tuning."
5.3 Instruction Tuning Score, p. 993
Instruction-tuning the model to generate associative content functions as a targeted intervention that causally increases model–brain alignment in memory-related regions, evidencing causal control over representational access.
"Two supervised fine-tuning methods are tried: low-rank adaptation (LoRA) ... and frozen layers finetuning. ... We define instruction tuning score M as the growth percentage of supervised finetuned model compared to base model:"
3.4 Instruct LLM to Generate Associative Content, p. 989
The explicit definition of an instruction-tuning score and the use of LoRA/frozen-layer methods formalize the intervention and its measured effect on brain alignment, aligning with causal-control criteria.
Limitations: Causal claims are confined to model-side interventions and a specific naturalistic listening dataset; no ablations isolate which tuned components drive gains, and generalization beyond the tested corpus/ROIs remains to be shown.