Lopez Kevin, Fodeh Samah J, Allam Ahmed, Brandt Cynthia A, Krauthammer Michael
Program of Computational Biology and Bioinformatics, Yale University, New Haven, CT, United States.
Department of Emergency Medicine, Yale School of Medicine, New Haven, CT, United States.
Front Big Data. 2020 Jun 2;3:19. doi: 10.3389/fdata.2020.00019. eCollection 2020.
Choosing an optimal data fusion technique is essential when performing machine learning with multimodal data. In this study, we examined deep learning-based multimodal fusion techniques for the combined classification of radiological images and associated text reports. In our analysis, we (1) compared the classification performance of three prototypical multimodal fusion techniques: , and fusion, (2) assessed the performance of multimodal compared to unimodal learning; and finally (3) investigated the amount of labeled data needed by multimodal vs. unimodal models to yield comparable classification performance. Our experiments demonstrate the potential of multimodal fusion methods to yield competitive results using less training data (labeled data) than their unimodal counterparts. This was more pronounced using the and less so using the fusion approaches. With increasing amount of training data, unimodal models achieved comparable results to multimodal models. Overall, our results suggest the potential of multimodal learning to decrease the need for labeled training data resulting in a lower annotation burden for domain experts.
在使用多模态数据进行机器学习时，选择最佳的数据融合技术至关重要。在本研究中，我们研究了基于深度学习的多模态融合技术，用于放射图像和相关文本报告的联合分类。在我们的分析中，我们（1）比较了三种典型多模态融合技术的分类性能： 、 以及 融合；（2）评估了多模态与单模态学习相比的性能；最后（3）研究了多模态模型与单模态模型为产生可比分类性能所需的标注数据量。我们的实验表明，与单模态对应方法相比，多模态融合方法有潜力使用更少的训练数据（标注数据）产生有竞争力的结果。使用 时这种情况更明显，而使用 融合方法时则不太明显。随着训练数据量的增加，单模态模型取得了与多模态模型相当的结果。总体而言，我们的结果表明多模态学习有潜力减少对标注训练数据的需求，从而降低领域专家的标注负担。
