Sohn Wonbum, Di Xin, Liang Zhen, Zhang Zhiguo, Biswal Bharat B
Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA.
Rutgers Biomedical and Health Sciences, Rutgers School of Graduate Studies, Newark, NJ 07039, USA.
Psychoradiology. 2024 Nov 4;4:kkae021. doi: 10.1093/psyrad/kkae021. eCollection 2024.
Naturalistic stimuli, such as videos, can elicit complex brain activations. However, the intricate nature of these stimuli makes it challenging to attribute specific brain functions to the resulting activations, particularly for higher-level processes such as social interactions.
We hypothesized that activations in different layers of a convolutional neural network (VGG-16) would correspond to varying levels of brain activation, reflecting the brain's visual processing hierarchy. Additionally, we aimed to explore which brain regions would be linked to the deeper layers of the network.
This study analyzed functional MRI data from participants watching a cartoon video. Using a pre-trained VGG-16 convolutional neural network, we mapped hierarchical features of the video to different levels of brain activation. Activation maps from various kernels and layers were extracted from video frames, and the time series of average activation patterns for each kernel were used in a voxel-wise model to examine brain responses.
Lower layers of the network were primarily associated with activations in lower visual regions, although some kernels also unexpectedly showed associations with the posterior cingulate cortex. Deeper layers were linked to more anterior and lateral regions of the visual cortex, as well as the supramarginal gyrus.
This analysis demonstrated both the potential and limitations of using convolutional neural networks to connect video content with brain functions, providing valuable insights into how different brain regions respond to varying levels of visual processing.
诸如视频之类的自然主义刺激可以引发复杂的大脑激活。然而，这些刺激的复杂性使得将特定的大脑功能归因于由此产生的激活具有挑战性，尤其是对于诸如社交互动等高级过程而言。
我们假设卷积神经网络（VGG - 16）不同层的激活将对应于不同水平的大脑激活，反映大脑的视觉处理层次结构。此外，我们旨在探索哪些脑区将与网络的更深层相关联。
本研究分析了参与者观看卡通视频时的功能磁共振成像数据。使用预训练的VGG - 16卷积神经网络，我们将视频的层次特征映射到不同水平的大脑激活。从视频帧中提取来自各种内核和层的激活图，并将每个内核的平均激活模式的时间序列用于体素级模型以检查大脑反应。
网络的较低层主要与较低视觉区域的激活相关，尽管一些内核也意外地显示出与后扣带回皮层的关联。更深层与视觉皮层的更靠前和外侧区域以及缘上回相关联。
该分析展示了使用卷积神经网络将视频内容与大脑功能联系起来的潜力和局限性，为不同脑区如何响应不同水平的视觉处理提供了有价值的见解。
