Gao Jun, Lao Qicheng, Kang Qingbo, Liu Paul, Du Chenlin, Li Kang, Zhang Le
IEEE Trans Med Imaging. 2025 Jan;44(1):310-319. doi: 10.1109/TMI.2024.3440311. Epub 2025 Jan 2.
The recent advent of in-context learning (ICL) capabilities in large pre-trained models has yielded significant advancements in the generalization of segmentation models. By supplying domain-specific image-mask pairs, the ICL model can be effectively guided to produce optimal segmentation outcomes, eliminating the necessity for model fine-tuning or interactive prompting. However, current existing ICL-based segmentation models exhibit significant limitations when applied to medical segmentation datasets with substantial diversity. To address this issue, we propose a dual similarity checkup approach to guarantee the effectiveness of selected in-context samples so that their guidance can be maximally leveraged during inference. We first employ large pre-trained vision models for extracting strong semantic representations from input images and constructing a feature embedding memory bank for semantic similarity checkup during inference. Assuring the similarity in the input semantic space, we then minimize the discrepancy in the mask appearance distribution between the support set and the estimated mask appearance prior through similarity-weighted sampling and augmentation. We validate our proposed dual similarity checkup approach on eight publicly available medical segmentation datasets, and extensive experimental results demonstrate that our proposed method significantly improves the performance metrics of existing ICL-based segmentation models, particularly when applied to medical image datasets characterized by substantial diversity.
大型预训练模型中上下文学习(ICL)能力的近期出现,在分割模型的泛化方面取得了重大进展。通过提供特定领域的图像-掩码对,可以有效地引导ICL模型产生最佳分割结果,从而无需进行模型微调或交互式提示。然而,当前现有的基于ICL的分割模型在应用于具有显著多样性的医学分割数据集时表现出明显的局限性。为了解决这个问题,我们提出了一种双重相似性检查方法,以确保所选上下文样本的有效性,以便在推理过程中最大程度地利用它们的指导。我们首先使用大型预训练视觉模型从输入图像中提取强大的语义表示,并构建一个特征嵌入记忆库,用于在推理过程中进行语义相似性检查。在确保输入语义空间中的相似性之后,我们通过相似性加权采样和增强,最小化支持集与估计掩码外观先验之间掩码外观分布的差异。我们在八个公开可用的医学分割数据集上验证了我们提出的双重相似性检查方法,大量实验结果表明,我们提出的方法显著提高了现有基于ICL的分割模型的性能指标,特别是在应用于具有显著多样性的医学图像数据集时。