Chi Jianning, Chen Jia-Hui, Wu Bo, Zhao Jin, Wang Kai, Yu Xiaosheng, Zhang Wenjun, Huang Ying
IEEE J Biomed Health Inform. 2025 Feb;29(2):1269-1282. doi: 10.1109/JBHI.2024.3472609. Epub 2025 Feb 10.
Contrast-enhanced ultrasound (CEUS) has been extensively employed as an imaging modality in thyroid nodule diagnosis due to its capacity to visualise the distribution and circulation of micro-vessels in organs and lesions in a non-invasive manner. However, current CEUS-based thyroid nodule diagnosis methods suffered from: 1) the blurred spatial boundaries between nodules and other anatomies in CEUS videos, and 2) the insufficient representations of the local structural information of nodule tissues by the features extracted only from CEUS videos. In this paper, we propose a novel dual-branch network with a cross-modality-attention mechanism for thyroid nodule diagnosis by integrating the information from tow related modalities, i.e., CEUS videos and ultrasound image. The mechanism has two parts: US-attention-from-CEUS transformer (UAC-T) and CEUS-attention-from-US transformer (CAU-T). As such, this network imitates the manner of human radiologists by decomposing the diagnosis into two correlated tasks: 1) the spatio-temporal features extracted from CEUS are hierarchically embedded into the spatial features extracted from US with UAC-T for the nodule segmentation; 2) the US spatial features are used to guide the extraction of the CEUS spatio-temporal features with CAU-T for the nodule classification. The two tasks are intertwined in the dual-branch end-to-end network and optimized with the multi-task learning (MTL) strategy. The proposed method is evaluated on our collected thyroid US-CEUS dataset. Experimental results show that our method achieves the classification accuracy of 86.92%, specificity of 66.41%, and sensitivity of 97.01%, outperforming the state-of-the-art methods. As a general contribution in the field of multi-modality diagnosis of diseases, the proposed method has provided an effective way to combine static information with its related dynamic information, improving the quality of deep learning based diagnosis with an additional benefit of explainability.
超声造影（CEUS）因其能够以非侵入性方式可视化器官和病变中微血管的分布和循环，已被广泛用作甲状腺结节诊断的成像方式。然而，当前基于CEUS的甲状腺结节诊断方法存在以下问题：1）CEUS视频中结节与其他解剖结构之间的空间边界模糊；2）仅从CEUS视频中提取的特征对结节组织局部结构信息的表征不足。在本文中，我们提出了一种具有跨模态注意力机制的新型双分支网络，通过整合超声造影视频（CEUS视频）和超声图像这两种相关模态的信息来进行甲状腺结节诊断。该机制有两个部分：从CEUS到超声的注意力变换器（UAC-T）和从超声到CEUS的注意力变换器（CAU-T）。因此，该网络通过将诊断分解为两个相关任务来模仿人类放射科医生的方式：1）利用UAC-T将从CEUS中提取的时空特征分层嵌入到从超声中提取的空间特征中，用于结节分割；2）利用CAU-T将超声空间特征用于指导CEUS时空特征的提取，用于结节分类。这两个任务在双分支端到端网络中相互交织，并采用多任务学习（MTL）策略进行优化。我们在收集的甲状腺超声-CEUS数据集中对所提出的方法进行了评估。实验结果表明，我们的方法实现了86.92%的分类准确率、66.41%的特异性和97.01%的灵敏度，优于现有方法。作为疾病多模态诊断领域的一项总体贡献，所提出的方法提供了一种将静态信息与其相关动态信息相结合的有效方法，提高了基于深度学习的诊断质量，并具有可解释性的额外优势。
