IEEE Trans Med Imaging. 2021 Feb;40(2):503-513. doi: 10.1109/TMI.2020.3030752. Epub 2021 Feb 2.
Multi-modal neuroimage retrieval has greatly facilitated the efficiency and accuracy of decision making in clinical practice by providing physicians with previous cases (with visually similar neuroimages) and corresponding treatment records. However, existing methods for image retrieval usually fail when applied directly to multi-modal neuroimage databases, since neuroimages generally have smaller inter-class variation and larger inter-modal discrepancy compared to natural images. To this end, we propose a deep Bayesian hash learning framework, called CenterHash, which can map multi-modal data into a shared Hamming space and learn discriminative hash codes from imbalanced multi-modal neuroimages. The key idea to tackle the small inter-class variation and large inter-modal discrepancy is to learn a common center representation for similar neuroimages from different modalities and encourage hash codes to be explicitly close to their corresponding center representations. Specifically, we measure the similarity between hash codes and their corresponding center representations and treat it as a center prior in the proposed Bayesian learning framework. A weighted contrastive likelihood loss function is also developed to facilitate hash learning from imbalanced neuroimage pairs. Comprehensive empirical evidence shows that our method can generate effective hash codes and yield state-of-the-art performance in cross-modal retrieval on three multi-modal neuroimage datasets.
多模态神经影像检索通过为医生提供（具有视觉相似神经影像）和相应治疗记录的先前案例，极大地提高了临床实践中的决策效率和准确性。然而，现有的图像检索方法在直接应用于多模态神经影像数据库时通常会失败，因为与自然影像相比，神经影像通常具有更小的类内变化和更大的模态差异。为此，我们提出了一种深度贝叶斯哈希学习框架，称为 CenterHash，它可以将多模态数据映射到共享的汉明空间，并从不平衡的多模态神经影像中学习有鉴别力的哈希码。解决小类内变化和大模态差异的关键思想是从不同模态学习相似神经影像的共同中心表示，并鼓励哈希码与其对应的中心表示明显接近。具体来说，我们测量哈希码与其对应中心表示之间的相似性，并将其视为所提出贝叶斯学习框架中的中心先验。还开发了加权对比似然损失函数，以促进从不平衡的神经影像对中进行哈希学习。全面的实证证据表明，我们的方法可以生成有效的哈希码，并在三个多模态神经影像数据集的跨模态检索中实现最先进的性能。
