Sun Guangyu
The Basic Department, The Tourism College of Changchun University, Changchun, China.
Front Neurosci. 2025 Jan 7;18:1493163. doi: 10.3389/fnins.2024.1493163. eCollection 2024.
In the field of medical listening assessments,accurate transcription and effective cognitive load management are critical for enhancing healthcare delivery. Traditional speech recognition systems, while successful in general applications often struggle in medical contexts where the cognitive state of the listener plays a significant role. These conventional methods typically rely on audio-only inputs and lack the ability to account for the listener's cognitive load, leading to reduced accuracy and effectiveness in complex medical environments.
To address these limitations, this study introduces ClinClip, a novel multimodal model that integrates EEG signals with audio data through a transformer-based architecture. ClinClip is designed to dynamically adjust to the cognitive state of the listener, thereby improving transcription accuracy and robustness in medical settings. The model leverages cognitive-enhanced strategies, including EEG-based modulation and hierarchical fusion of multimodal data, to overcome the challenges faced by traditional methods.
Experiments conducted on four datasets-EEGEyeNet, DEAP, PhyAAt, and eSports Sensors-demonstrate that ClinClip significantly outperforms six state-of-the-art models in both Word Error Rate (WER) and Cognitive Modulation Efficiency (CME). These results underscore the model's effectiveness in handling complex medical audio scenarios and highlight its potential to improve the accuracy of medical listening assessments. By addressing the cognitive aspects of the listening process. ClinClip contributes to more reliable and effective healthcare delivery, offering a substantial advancement over traditional speech recognition approaches.
在医学听力评估领域，准确的转录和有效的认知负荷管理对于改善医疗服务至关重要。传统的语音识别系统虽然在一般应用中取得了成功，但在医学环境中往往面临困难，因为听者的认知状态起着重要作用。这些传统方法通常仅依赖音频输入，缺乏考虑听者认知负荷的能力，导致在复杂的医疗环境中准确性和有效性降低。
为了解决这些局限性，本研究引入了ClinClip，这是一种新颖的多模态模型，它通过基于Transformer的架构将脑电图（EEG）信号与音频数据集成在一起。ClinClip旨在动态适应听者的认知状态，从而提高医学环境中的转录准确性和鲁棒性。该模型利用认知增强策略，包括基于EEG的调制和多模态数据的分层融合，以克服传统方法面临的挑战。
在四个数据集——EEGEyeNet、DEAP、PhyAAt和电子竞技传感器——上进行的实验表明，ClinClip在字错误率（WER）和认知调制效率（CME）方面均显著优于六个最先进的模型。这些结果强调了该模型在处理复杂医学音频场景方面的有效性，并突出了其提高医学听力评估准确性的潜力。通过解决听力过程中的认知方面，ClinClip有助于实现更可靠、更有效的医疗服务，相对于传统语音识别方法有了实质性的进步。
