TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction

"When combining any two modalities together, the encoding scores rise significantly compared to the unimodal models: the best bimodal model, obtained by combining text and video, achieves an encoding score of 0.30. Finally, combining the three modalities together yields an additional boost, bringing the value to 0.31. This hints to the fact that each modality plays a complementary role."

3.3 The benefit of multimodality, p. 6

This quantitative improvement from bimodal to trimodal inputs indicates integration of distributed information streams into unified representations that better match whole-brain activity, consistent with Information Integration as a core phenomenon in consciousness science (see also the cortical maps emphasizing associative areas) .

"These observations provide new insights on how multisensory integration may occurs in the human cortex."

3.4 The interplay between modalities, p. 7

Explicitly linking the results to multisensory integration supports the view that distributed modality-specific features are combined into unified cortical representations, aligning with Information Integration .

Figures

Figure 4 (p. 6) : Direct evidence that combining modalities improves brain-encoding performance, indicating integration across features and brain networks relevant to conscious content .

Figure 5 (p. 7) : Bimodal and trimodal patterns across cortex visualize where integration occurs, especially in associative regions implicated in global access and conscious processing .

"Modality dropout One desirable property of a multimodal encoding model is its ability to provide meaningful predictions in the absence of one or several modalities, for example for a silent movie or a podcast. To encourage this behaviour, while at the same time avoiding excessive reliance on one modality, we introduce modality dropout: during training, we randomly mask off each modality by zeroing out the corresponding input tensor with a probability p, ensuring that at least one modality is left unmasked."

2.4 Training details, p. 4

Masking/gating entire modalities implements selective routing pressure in the model, paralleling attentional gating and thalamocortical control motifs studied in biological consciousness research .

Tables

Table 3 (p. 13) : Table 3: Hyperparameters used in our model

"In fig. 6a, we demonstrate the importance of using a transformer and a multi-subject training scheme: the encoding score drops from 0.31 to 0.29 when training separately for each subject, and down to 0.23 when removing the transformer."

3.5 Ablations and scaling laws, p. 7

These controlled interventions (ablate transformer; change training regime) alter performance, providing causal evidence that specific modules and training choices govern computational competence—an AI-side parallel to lesion/TMS-like causal tests in consciousness studies .

Figures

Figure 6 (p. 8) : Performance drops under ablations indicate causal control of computation by specific architectural components, analogous to interventionist methods in neuroscience .

"To compress these embeddings while retaining both low-level and high-level information, for each modality, we split the layers into L groups, then average the tensor per group along the layer dimension, compressing to a shape [L,Dm]. We then concatenate the layers and feed the resulting vector through a linear layer with a shared output dimension D = 1024 followed by layer normalization. Finally, we concatenate the three modalities, leading to a time series of multimodal embeddings of shape 3× 1024. This will be the input to our transformer encoder."

2.3 Model, p. 4

The described embedding pipeline specifies a representational structure that organizes multimodal content in a shared subspace, aligning with the Representational Structure phenomenon and enabling downstream access/control by attention mechanisms .

Figures

Figure 2 (p. 2) : Schematic depicts how text, audio, and video embeddings are combined and processed, illustrating the model’s representational layout across modalities and layers .

"We extract windows of duration T = N × TR from these embedding time series, add learnable positional embeddings and a learnable subject embedding, then feed the result through a Transformer encoder with 8 layers and 8 attention heads. This enables information to be exchanged between timesteps."

2.3 Model, p. 4

Using positional embeddings and attention to exchange information across timesteps implements temporal coordination—an AI analogue of timing/phase mechanisms proposed to segment and bind conscious content in the brain .

Figures

Figure 2 (p. 2) : Pipeline illustration shows temporally binned, aligned embeddings feeding a transformer, consistent with coordinated temporal processing .