基于多模态数据的乳腺癌分类更丰富的融合网络。

BACKGROUND: Deep learning algorithms significantly improve the accuracy of pathological image classification, but the accuracy of breast cancer classification using only single-mode pathological images still cannot meet the needs of clinical practice. Inspired by the real scenario of pathologists reading pathological images for diagnosis, we integrate pathological images and structured data extracted from clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification. METHODS: In this paper, we propose a new richer fusion network for the classification of benign and malignant breast cancer based on multimodal data. To make pathological image can be integrated more sufficient with structured EMR data, we proposed a method to extract richer multilevel feature representation of the pathological image from multiple convolutional layers. Meanwhile, to minimize the information loss for each modality before data fusion, we use the denoising autoencoder as a way to increase the low-dimensional structured EMR data to high-dimensional, instead of reducing the high-dimensional image data to low-dimensional before data fusion. In addition, denoising autoencoder naturally generalizes our method to make the accurate prediction with partially missing structured EMR data. RESULTS: The experimental results show that the proposed method is superior to the most advanced method in terms of the average classification accuracy (92.9%). In addition, we have released a dataset containing structured data from 185 patients that were extracted from EMR and 3764 paired pathological images of breast cancer, which can be publicly downloaded from http://ear.ict.ac.cn/?page_id=1663 . CONCLUSIONS: We utilized a new richer fusion network to integrate highly heterogeneous data to leverage the structured EMR data to improve the accuracy of pathological image classification. Therefore, the application of automatic breast cancer classification algorithms in clinical practice becomes possible. Due to the generality of the proposed fusion method, it can be straightforwardly extended to the fusion of other structured data and unstructured data.

背景：深度学习算法显著提高了病理图像分类的准确性，但仅使用单模态病理图像进行乳腺癌分类的准确性仍无法满足临床实践的需求。受病理学家阅读病理图像进行诊断的真实场景启发，我们整合了病理图像和从临床电子病历（EMR）中提取的结构化数据，以进一步提高乳腺癌分类的准确性。

方法：在本文中，我们提出了一种新的基于多模态数据的良性和恶性乳腺癌分类更丰富的融合网络。为了使病理图像能够更充分地与结构化 EMR 数据融合，我们提出了一种从多个卷积层中提取病理图像更丰富的多层次特征表示的方法。同时，为了在数据融合之前最小化每种模态的信息损失，我们使用去噪自编码器作为一种将低维结构化 EMR 数据增加到高维的方法，而不是在数据融合之前将高维图像数据降低到低维。此外，去噪自编码器自然地将我们的方法推广到可以在结构化 EMR 数据部分缺失的情况下进行准确预测。

结果：实验结果表明，该方法在平均分类准确率（92.9%）方面优于最先进的方法。此外，我们还发布了一个包含从 EMR 中提取的 185 名患者的结构化数据和 3764 对乳腺癌病理图像的数据集，可以从 http://ear.ict.ac.cn/?page_id=1663 公开下载。

结论：我们利用一种新的更丰富的融合网络来整合高度异构的数据，利用结构化 EMR 数据来提高病理图像分类的准确性。因此，自动乳腺癌分类算法在临床实践中的应用成为可能。由于所提出的融合方法具有通用性，它可以直接扩展到其他结构化数据和非结构化数据的融合。