Li Xinhui, Li Ao, Fu Wenyu, Song Xun, Li Fan, Ma Qiang, Peng Yong, Lv Zhao
Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and TechnologyAnhui University Hefei 230601 China.
CAAC Key Laboratory of Civil Aviation Flight Technology and Flight SafetyCivil Aviation Flight University of China Guanghan 618307 China.
IEEE J Transl Eng Health Med. 2025 Feb 14;13:85-97. doi: 10.1109/JTEHM.2025.3542408. eCollection 2025.
Pilots face complex working environments during flight missions, which can easily lead to excessive workload and affect flight safety. Physiological signals are commonly used to evaluate a pilot's workload because they are objective and can directly reflect physiological mental states. However, existing methods have shortcomings in temporal modeling, making it challenging to fully capture the dynamic characteristics of physiological signals over time. Moreover, fusing features of data from different modalities is also difficult.To address these problems, we proposed a temporal relation modeling and multimodal adversarial alignment network (TRM-MAAN) for pilot workload evaluation. Specifically, a Transformer-based temporal relationship modeling module was used to learn complex temporal relationships for better feature extraction. In addition, an adversarial alignment-based multi-modal fusion module was applied to capture and integrate multi-modal information, reducing distribution shifts between different modalities. The performance of the proposed TRM-MAAN method was evaluated via experiments of classifying three workload states using electroencephalogram (EEG) and electromyography (EMG) recordings of eight healthy pilots.Experimental results showed that the classification accuracy and F1 score of the proposed method were significantly better than the baseline model across different subjects, with an average recognition accuracy of [Formula: see text] and an F1 score of [Formula: see text].This work provides essential technical support for improving the accuracy and robustness of pilot workload evaluation and introduces a promising way for enhancing flight safety, offering broad application prospects. Clinical and Translational Impact Statement: The proposed scheme provides a promising solution for workload evaluation based on electrophysiological signals, with potential applications in aiding the clinical monitoring of fatigue, mental status, cognitive psychology, and other disorders.
飞行员在飞行任务中面临复杂的工作环境，这很容易导致工作量过大并影响飞行安全。生理信号通常用于评估飞行员的工作量，因为它们是客观的，并且可以直接反映生理和心理状态。然而，现有方法在时间建模方面存在不足，使得难以充分捕捉生理信号随时间的动态特征。此外，融合来自不同模态的数据特征也很困难。为了解决这些问题，我们提出了一种用于飞行员工作量评估的时间关系建模和多模态对抗对齐网络（TRM-MAAN）。具体而言，基于Transformer的时间关系建模模块用于学习复杂的时间关系，以更好地进行特征提取。此外，基于对抗对齐的多模态融合模块用于捕获和整合多模态信息，减少不同模态之间的分布差异。通过使用八名健康飞行员的脑电图（EEG）和肌电图（EMG）记录对三种工作量状态进行分类的实验，评估了所提出的TRM-MAAN方法的性能。实验结果表明，所提出方法的分类准确率和F1分数在不同受试者中均显著优于基线模型，平均识别准确率为[公式：见原文]，F1分数为[公式：见原文]。这项工作为提高飞行员工作量评估的准确性和鲁棒性提供了重要的技术支持，并为增强飞行安全引入了一种有前景的方法，具有广泛的应用前景。临床和转化影响声明：所提出的方案为基于电生理信号的工作量评估提供了一种有前景的解决方案，在辅助疲劳、精神状态、认知心理学和其他疾病的临床监测方面具有潜在应用。
