Moradizeyveh Sahar, Hanif Ambreen, Liu Sidong, Qi Yuankai, Beheshti Amin, Di Ieva Antonio
Computational NeuroSurgery (CNS) Lab, Macquarie Medical School, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney 2113, Australia.
Centre for Applied Artificial Intelligence, School of Computing, Faculty of Science and Engineering, Macquarie University, Sydney 2113, Australia.
Sensors (Basel). 2025 Jul 24;25(15):4575. doi: 10.3390/s25154575.
Interpreting diagnostic imaging and identifying clinically relevant features remain challenging tasks, particularly for novice radiologists who often lack structured guidance and expert feedback. To bridge this gap, we propose an Eye-Gaze Guided Multimodal Fusion framework that leverages expert eye-tracking data to enhance learning and decision-making in medical image interpretation. By integrating chest X-ray (CXR) images with expert fixation maps, our approach captures radiologists' visual attention patterns and highlights regions of interest (ROIs) critical for accurate diagnosis. The fusion model utilizes a shared backbone architecture to jointly process image and gaze modalities, thereby minimizing the impact of noise in fixation data. We validate the system's interpretability using Gradient-weighted Class Activation Mapping (Grad-CAM) and assess both classification performance and explanation alignment with expert annotations. Comprehensive evaluations, including robustness under gaze noise and expert clinical review, demonstrate the framework's effectiveness in improving model reliability and interpretability. This work offers a promising pathway toward intelligent, human-centered AI systems that support both diagnostic accuracy and medical training.
解读诊断成像并识别临床相关特征仍然是具有挑战性的任务，特别是对于那些常常缺乏结构化指导和专家反馈的新手放射科医生而言。为了弥补这一差距，我们提出了一种眼动引导的多模态融合框架，该框架利用专家眼动追踪数据来加强医学图像解读中的学习和决策。通过将胸部X光（CXR）图像与专家注视图相结合，我们的方法捕捉放射科医生的视觉注意力模式，并突出显示对准确诊断至关重要的感兴趣区域（ROI）。融合模型利用共享骨干架构来联合处理图像和注视模态，从而将注视数据中的噪声影响降至最低。我们使用梯度加权类激活映射（Grad-CAM）验证系统的可解释性，并评估分类性能以及与专家注释的解释一致性。包括注视噪声下的稳健性和专家临床审查在内的综合评估证明了该框架在提高模型可靠性和可解释性方面的有效性。这项工作为支持诊断准确性和医学培训的智能、以人为本的人工智能系统提供了一条充满希望的途径。
