Yubo Zheng, Yingying Luo, Bing Zou, Lin Zhang, Lei Li
School of Artificial Intelligence, University of Posts and Telecommunications, Beijing, China.
Front Neurosci. 2022 Aug 16;16:973761. doi: 10.3389/fnins.2022.973761. eCollection 2022.
Pandemic-related sleep disorders affect human physical and mental health. The artificial intelligence (AI) based sleep staging with multimodal electrophysiological signals help people diagnose and treat sleep disorders. However, the existing AI-based methods could not capture more discriminative modalities and adaptively correlate these multimodal features. This paper introduces a multimodal attention network (MMASleepNet) to efficiently extract, perceive and fuse multimodal features of electrophysiological signals. The MMASleepNet has a multi-branch feature extraction (MBFE) module followed by an attention-based feature fusing (AFF) module. In the MBFE module, branches are designed to extract multimodal signals' temporal and spectral features. Each branch has two-stream convolutional networks with a unique kernel to perceive features of different time scales. The AFF module contains a modal-wise squeeze and excitation (SE) block to adjust the weights of modalities with more discriminative features and a Transformer encoder (TE) to generate attention matrices and extract the inter-dependencies among multimodal features. Our MMASleepNet outperforms state-of-the-art models in terms of different evaluation matrices on the datasets of Sleep-EDF and ISRUC-Sleep. The implementation code is available at: https://github.com/buptantEEG/MMASleepNet/.
与大流行相关的睡眠障碍会影响人类身心健康。基于人工智能（AI）的多模态电生理信号睡眠分期有助于人们诊断和治疗睡眠障碍。然而，现有的基于AI的方法无法捕捉更具判别力的模态并自适应地关联这些多模态特征。本文介绍了一种多模态注意力网络（MMASleepNet），以有效地提取、感知和融合电生理信号的多模态特征。MMASleepNet有一个多分支特征提取（MBFE）模块，后面跟着一个基于注意力的特征融合（AFF）模块。在MBFE模块中，各分支被设计用于提取多模态信号的时间和频谱特征。每个分支都有双流卷积网络，带有独特内核以感知不同时间尺度的特征。AFF模块包含一个模态维度挤压与激励（SE）块，用于调整具有更多判别性特征的模态权重，以及一个Transformer编码器（TE），用于生成注意力矩阵并提取多模态特征之间的相互依赖关系。在Sleep-EDF和ISRUC-Sleep数据集上，我们的MMASleepNet在不同评估指标方面优于现有模型。实现代码可在以下网址获取：https://github.com/buptantEEG/MMASleepNet/ 。
