Preprint (arXiv:2407.07595v1)

"We employed the contrastive language-image pre-training (CLIP) [23] method, which uses self-supervised learning and hence does not require labels, on a large amount of paired EEG and speech data. Using CLIP we integrate the different modalities of EEG and speech and acquire representations that can be easily used for various downstream tasks, such as zero-shot classification of phrases, and speech reconstruction."

Introduction, p. 2

By aligning EEG and speech into a shared representation, this work demonstrates information integration across modalities—an AI analogue of unified access that is central to many theories of consciousness in the brain and machines .

Figures

Figure 1 (p. 2) : The pipeline explicitly aligns EEG and audio latents with a single contrastive objective, exemplifying integration of distributed signals into a unified representational space relevant for access and report .

"The cosine similarity matrix was computed between the EEG latents and audio latents of the 512 samples in the test set. For each EEG latent, the indices of the top-k most similar audio latents based on cosine similarity were extracted."

3.3 Evaluation, p. 4

This defines a structured latent geometry linking EEG and audio representations, enabling content-addressable retrieval consistent with organized representational subspaces in both brains and AI .

Figures

Figure 2 (p. 4) : Architectural details show how EEG is mapped into a latent space suitable for similarity-based retrieval, underscoring representational structure formation .

Tables

Table 2 (p. 5) : Table 2: Zero-shot segment classification accuracy across different audio encoders.

"The EEG latent and wav2vec2.0 latent representation were divided into 50 bins (=100ms) respectively, and the SD values were calculated along the time axis within each bin. These SD values were averaged across the features and binarized using a threshold to detect the onset of each syllable."

4 Results, p. 6

Temporal binning and onset detection show coordinated timing dynamics between EEG and audio latents, aligning with the role of temporal coordination mechanisms in segmenting and binding content across systems .

Figures

Figure 4 (p. 6) : Demonstrates that latent dynamics carry timing signals sufficient to segment speech from silence at ~100 ms resolution, evidencing temporal coordination relevant to content binding and access .

Tables

Table 3 (p. 6) : Table 3: Comparison of voice activity detection performance between different latents and raw EEG.

"The top-10 decoding performance from EMG signals dropped to around 3% for the models trained to disregard EMG signals using the mixed EMG and EEG signals This decline of performance to the near chance level of 2% suggests that the inference from EEG was not heavily influenced by EMG artifacts."

4.4 Influence of EMG contamination on decoding performance, p. 8

This targeted intervention functions like an ablation/causal control, showing that EEG—not EMG—drives decoding, an essential approach for establishing causal contribution of signal sources in consciousness-relevant readouts .

"The models trained on the EMG-mixed data failed to predict the speech only from the EMG signals as shown by the near-chance decoding accuracy from EMG signals alone (Table 4). These findings indicate that our model primarily relied on EEG signals."

6 Discussion, p. 9

Authors conclude that the causal manipulation supports EEG as the primary driver of decoding, strengthening claims about neural contributions rather than peripheral artifacts in report-linked signals .

Tables

Table 4 (p. 7) : Table 4: Low susceptibility to EMG artifacts. The susceptibility to EMG artifacts of EEG encoder was evaluated by the top-10 decoding accuracy from EMG.

"Figure 3(left, center) show that the classification accuracy improves as the amount of training data increases. Furthermore, the accuracy has not saturated with respect to the training data and continues to rise, suggesting that increasing the amount of data further would lead to further improvements in accuracy."

4.2 Data scaling in classification accuracy, p. 5

Monotonic performance gains with data suggest emergent capabilities with scale—analogous to emergent dynamics in large AI models and potentially relevant to capacity for richer content access with increased experience/data .

"Crucially, our results suggest that decoding accuracy can continue to increase significantly with further training data, based on a scaling law between performance and data size."

6 Discussion, p. 9

The authors explicitly frame a scaling-law relationship, reinforcing the view that higher-order performance emerges with increased data and training—paralleling emergent phenomena in modern AI systems .

Figures

Figure 3 (p. 5) : Shows non-saturating improvements in decoding performance with more data—an emergent behavior with scale .

Tables

Table 5 (p. 14) : Table 5: Zero-shot segment classification performance at the same amount of training data and number of segments as previous study [24].