School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
Division of Liver Surgery, Department of General Surgery, West China Hospital, Sichuan University, Chengdu, 610044, China.
Med Image Anal. 2023 Oct;89:102890. doi: 10.1016/j.media.2023.102890. Epub 2023 Jul 8.
Recently, convolutional neural networks (CNNs) directly using whole slide images (WSIs) for tumor diagnosis and analysis have attracted considerable attention, because they only utilize the slide-level label for model training without any additional annotations. However, it is still a challenging task to directly handle gigapixel WSIs, due to the billions of pixels and intra-variations in each WSI. To overcome this problem, in this paper, we propose a novel end-to-end interpretable deep MIL framework for WSI analysis, by using a two-branch deep neural network and a multi-scale representation attention mechanism to directly extract features from all patches of each WSI. Specifically, we first divide each WSI into bag-, patch- and cell-level images, and then assign the slide-level label to its corresponding bag-level images, so that WSI classification becomes a MIL problem. Additionally, we design a novel multi-scale representation attention mechanism, and embed it into a two-branch deep network to simultaneously mine the bag with a correct label, the significant patches and their cell-level information. Extensive experiments demonstrate the superior performance of the proposed framework over recent state-of-the-art methods, in term of classification accuracy and model interpretability. All source codes are released at: https://github.com/xhangchen/MRAN/.
最近，直接使用全切片图像（WSI）进行肿瘤诊断和分析的卷积神经网络（CNN）引起了相当大的关注，因为它们仅利用幻灯片级别的标签进行模型训练，而无需任何其他注释。然而，由于每个 WSI 中的数十亿个像素和内部变化，直接处理千兆像素 WSI 仍然是一项具有挑战性的任务。为了解决这个问题，在本文中，我们提出了一种新颖的端到端可解释深度多实例学习（MIL）框架，用于 WSI 分析，该框架使用双分支深度神经网络和多尺度表示注意力机制，直接从每个 WSI 的所有补丁中提取特征。具体来说，我们首先将每个 WSI 划分为袋级、补丁级和细胞级图像，然后将幻灯片级标签分配给其对应的袋级图像，从而使 WSI 分类成为一个多实例学习问题。此外，我们设计了一种新颖的多尺度表示注意力机制，并将其嵌入到双分支深度网络中，同时挖掘具有正确标签的袋、重要补丁及其细胞级信息。大量实验表明，所提出的框架在分类准确性和模型可解释性方面优于最新的最先进方法。所有的源代码都可以在 https://github.com/xhangchen/MRAN/ 上找到。
