Department for Interdisciplinary Endoscopy, Hamburg, Germany; Center for Biomedical Artificial Intelligence (bAIome), Hamburg, Germany; Department of Computational Neuroscience, Hamburg, Germany.
Center for Biomedical Artificial Intelligence (bAIome), Hamburg, Germany; Department of Computational Neuroscience, Hamburg, Germany.
Med Image Anal. 2021 May;70:101996. doi: 10.1016/j.media.2021.101996. Epub 2021 Feb 18.
Histopathologic diagnosis relies on simultaneous integration of information from a broad range of scales, ranging from nuclear aberrations (≈O(0.1μm)) through cellular structures (≈O(10μm)) to the global tissue architecture (⪆O(1mm)). To explicitly mimic how human pathologists combine multi-scale information, we introduce a family of multi-encoder fully-convolutional neural networks with deep fusion. We present a simple block for merging model paths with differing spatial scales in a spatial relationship-preserving fashion, which can readily be included in standard encoder-decoder networks. Additionally, a context classification gate block is proposed as an alternative for the incorporation of global context. Our experiments were performed on three publicly available whole-slide images of recent challenges (PAIP 2019: hepatocellular carcinoma segmentation; BACH 2020: breast cancer segmentation; CAMELYON 2016: metastasis detection in lymph nodes). The multi-scale architectures consistently outperformed the baseline single-scale U-Nets by a large margin. They benefit from local as well as global context and particularly a combination of both. If feature maps from different scales are fused, doing so in a manner preserving spatial relationships was found to be beneficial. Deep guidance by a context classification loss appeared to improve model training at low computational costs. All multi-scale models had a reduced GPU memory footprint compared to ensembles of individual U-Nets trained on different image scales. Additional path fusions were shown to be possible at low computational cost, opening up possibilities for further, systematic and task-specific architecture optimisation. The findings demonstrate the potential of the presented family of human-inspired, end-to-end trainable, multi-scale multi-encoder fully-convolutional neural networks to improve deep histopathologic diagnosis by extensive integration of largely different spatial scales.
组织病理学诊断依赖于广泛范围内信息的同步整合，这些信息的尺度范围从核异常（≈O(0.1μm)）到细胞结构（≈O(10μm)）再到整个组织架构（⪆O(1mm)）。为了明确模拟人类病理学家如何整合多尺度信息，我们引入了一系列具有深度融合功能的多编码器全卷积神经网络。我们提出了一种简单的块，用于以保留空间关系的方式合并具有不同空间尺度的模型路径，这可以很容易地包含在标准的编码器-解码器网络中。此外，还提出了上下文分类门块作为合并全局上下文的替代方法。我们的实验是在三个公开的全切片图像数据集上进行的，包括最近的挑战（PAIP 2019：肝细胞癌分割；BACH 2020：乳腺癌分割；CAMELYON 2016：淋巴结转移检测）。多尺度架构始终以较大的优势优于基线单尺度 U-Net。它们受益于局部和全局上下文，特别是两者的结合。如果融合来自不同尺度的特征图，那么以保留空间关系的方式融合是有益的。在低计算成本的情况下，通过上下文分类损失进行深度引导似乎可以改善模型训练。与在不同图像尺度上训练的单个 U-Net 集合相比，所有多尺度模型的 GPU 内存占用都减少了。以低计算成本显示可以进行更多的路径融合，为进一步的、系统的和特定任务的架构优化开辟了可能性。研究结果表明，所提出的受人类启发的、端到端可训练的、多尺度多编码器全卷积神经网络系列具有潜力，可以通过广泛整合不同的空间尺度来提高深度组织病理学诊断。
