Lee Hyung-Tak, Shim Miseon, Liu Xianghong, Cheon Hye-Ran, Kim Sang-Gyu, Han Chang-Hee, Hwang Han-Jeong
Department of Electronics and Information Engineering, Korea University, 2511, Sejong-ro, Jochiwon-eup, Sejong-si, 30019 Republic of Korea.
Interdisciplinary Graduate Program for Artificial Intelligence Smart Convergence Technology, Korea University, Sejong, Republic of Korea.
Biomed Eng Lett. 2025 Mar 22;15(4):587-618. doi: 10.1007/s13534-025-00469-5. eCollection 2025 Jul.
Human-computer interaction (HCI) focuses on designing efficient and intuitive interactions between humans and computer systems. Recent advancements have utilized multimodal approaches, such as electroencephalography (EEG)-based systems combined with other biosignals, along with deep learning to enhance performance and reliability. However, no systematic review has consolidated findings on EEG-based multimodal HCI systems. This review examined 124 studies published from 2016 to 2024, retrieved from the Web of Science database, focusing on hybrid EEG-based multimodal HCI systems employing deep learning. The keywords used for evaluation were as follows: 'Deep Learning' AND 'EEG' AND ('fNIRS' OR 'NIRS' OR 'MEG' OR 'fMRI' OR 'EOG' OR 'EMG' OR 'ECG' OR 'PPG' OR 'GSR'). Main topics explored are: (1) types of biosignals used with EEG, (2) neural network architectures, (3) fusion strategies, (4) system performance, and (5) target applications. Frequently paired signals, such as EOG, EMG, and fNIRS, effectively complement EEG by addressing its limitations. Convolutional neural networks are extensively employed for spatio-temporal-spectral feature extraction, with early and intermediate fusion strategies being the most commonly used. Applications, such as sleep stage classification, emotion recognition, and mental state decoding, have shown notable performance improvements. Despite these advancements, challenges remain, including the lack of real-time online systems, difficulties in signal synchronization, limited data availability, and insufficient explainable AI (XAI) methods to interpret signal interactions. Emerging solutions, such as portable systems, lightweight deep learning models, and data augmentation techniques, offer promising pathways to address these issues. This review highlights the potential of EEG-based multimodal HCI systems and emphasizes the need for advancements in real-time interaction, fusion algorithms, and XAI to enhance their adaptability, interpretability, and reliability.
人机交互（HCI）专注于设计人与计算机系统之间高效且直观的交互。最近的进展采用了多模态方法，例如基于脑电图（EEG）的系统与其他生物信号相结合，并利用深度学习来提高性能和可靠性。然而，尚无系统综述对基于EEG的多模态HCI系统的研究结果进行整合。本综述检索了科学网数据库中2016年至2024年发表的124项研究，重点关注采用深度学习的基于EEG的混合多模态HCI系统。用于评估的关键词如下：“深度学习”与“EEG”与（“功能近红外光谱技术（fNIRS）”或“近红外光谱技术（NIRS）”或“脑磁图（MEG）”或“功能磁共振成像（fMRI）”或“眼电图（EOG）”或“肌电图（EMG）”或“心电图（ECG）”或“光电容积脉搏波描记法（PPG）”或“皮肤电反应（GSR）”）。探讨的主要主题包括：（1）与EEG一起使用的生物信号类型，（2）神经网络架构，（3）融合策略，（4）系统性能，以及（5）目标应用。诸如EOG、EMG和fNIRS等经常配对的信号通过解决EEG的局限性有效地补充了EEG。卷积神经网络被广泛用于时空光谱特征提取，早期和中间融合策略是最常用的。诸如睡眠阶段分类、情绪识别和心理状态解码等应用已显示出显著的性能提升。尽管有这些进展，但挑战仍然存在，包括缺乏实时在线系统、信号同步困难、数据可用性有限以及用于解释信号交互的可解释人工智能（XAI）方法不足。诸如便携式系统、轻量级深度学习模型和数据增强技术等新兴解决方案为解决这些问题提供了有希望的途径。本综述强调了基于EEG的多模态HCI系统的潜力，并强调需要在实时交互、融合算法和XAI方面取得进展，以提高其适应性、可解释性和可靠性。
