Institute of Vision Research, Department of Ophthalmology, Yonsei University College of Medicine, 50-1 Yonsei-ro, Seodaemun-gu, Seoul, 03722, South Korea.
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea.
Med Biol Eng Comput. 2019 Mar;57(3):677-687. doi: 10.1007/s11517-018-1915-z. Epub 2018 Oct 22.
Recently, researchers have built new deep learning (DL) models using a single image modality to diagnose age-related macular degeneration (AMD). Retinal fundus and optical coherence tomography (OCT) images in clinical settings are the most important modalities investigating AMD. Whether concomitant use of fundus and OCT data in DL technique is beneficial has not been so clearly identified. This experimental analysis used OCT and fundus image data of postmortems from the Project Macula. The DL based on OCT, fundus, and combination of OCT and fundus were invented to diagnose AMD. These models consisted of pre-trained VGG-19 and transfer learning using random forest. Following the data augmentation and training process, the DL using OCT alone showed diagnostic efficiency with area under the curve (AUC) of 0.906 (95% confidence interval, 0.891-0.921) and 82.6% (81.0-84.3%) accuracy rate. The DL using fundus alone exhibited AUC of 0.914 (0.900-0.928) and 83.5% (81.8-85.0%) accuracy rate. Combined usage of the fundus with OCT increased the diagnostic power with AUC of 0.969 (0.956-0.979) and 90.5% (89.2-91.8%) accuracy rate. The Delong test showed that the DL using both OCT and fundus data outperformed the DL using OCT alone (P value < 0.001) and fundus image alone (P value < 0.001). This multimodal random forest model showed even better performance than a restricted Boltzmann machine (P value = 0.002) and deep belief network algorithms (P value = 0.042). According to Duncan's multiple range test, the multimodal methods significantly improved the performance obtained by the single-modal methods. In this preliminary study, a multimodal DL algorithm based on the combination of OCT and fundus image raised the diagnostic accuracy compared to this data alone. Future diagnostic DL needs to adopt the multimodal process to combine various types of imaging for a more precise AMD diagnosis. Graphical abstract The basic architectural structure of the tested multimodal deep learning model based on pre-trained deep convolutional neural network and random forest using the combination of OCT and fundus image.
最近，研究人员使用单一的图像模态构建了新的深度学习 (DL) 模型来诊断年龄相关性黄斑变性 (AMD)。在临床环境中，视网膜眼底和光学相干断层扫描 (OCT) 图像是研究 AMD 的最重要的模态。DL 技术中是否同时使用眼底和 OCT 数据是否有益尚不清楚。本实验分析使用了 Project Macula 的死后 OCT 和眼底图像数据。基于 OCT、眼底和 OCT 与眼底结合的 DL 被发明来诊断 AMD。这些模型由预先训练的 VGG-19 和使用随机森林的迁移学习组成。在进行数据增强和训练过程后，单独使用 OCT 的 DL 显示出诊断效率，曲线下面积 (AUC) 为 0.906(95%置信区间，0.891-0.921)和 82.6%(81.0-84.3%)准确率。单独使用眼底的 DL 显示 AUC 为 0.914(0.900-0.928)和 83.5%(81.8-85.0%)准确率。眼底与 OCT 的联合使用提高了诊断能力，AUC 为 0.969(0.956-0.979)和 90.5%(89.2-91.8%)准确率。DeLong 检验表明，同时使用 OCT 和眼底数据的 DL 优于单独使用 OCT 的 DL (P 值<0.001)和单独使用眼底图像的 DL (P 值<0.001)。这种多模态随机森林模型的性能甚至优于受限玻尔兹曼机 (P 值=0.002)和深度置信网络算法 (P 值=0.042)。根据 Duncan 多重范围检验，多模态方法显著提高了单模态方法的性能。在这项初步研究中，基于 OCT 和眼底图像组合的多模态 DL 算法与单独使用这些数据相比，提高了诊断准确性。未来的诊断 DL 需要采用多模态过程，结合各种类型的成像，以实现更精确的 AMD 诊断。
