Institute for Computer and Systems Engineering, Technische Universität Ilmenau, Ilmenau, Germany.
PLoS One. 2021 Jan 12;16(1):e0245230. doi: 10.1371/journal.pone.0245230. eCollection 2021.
Humans' decision making process often relies on utilizing visual information from different views or perspectives. However, in machine-learning-based image classification we typically infer an object's class from just a single image showing an object. Especially for challenging classification problems, the visual information conveyed by a single image may be insufficient for an accurate decision. We propose a classification scheme that relies on fusing visual information captured through images depicting the same object from multiple perspectives. Convolutional neural networks are used to extract and encode visual features from the multiple views and we propose strategies for fusing these information. More specifically, we investigate the following three strategies: (1) fusing convolutional feature maps at differing network depths; (2) fusion of bottleneck latent representations prior to classification; and (3) score fusion. We systematically evaluate these strategies on three datasets from different domains. Our findings emphasize the benefit of integrating information fusion into the network rather than performing it by post-processing of classification scores. Furthermore, we demonstrate through a case study that already trained networks can be easily extended by the best fusion strategy, outperforming other approaches by large margin.
人类的决策过程通常依赖于利用来自不同视角的视觉信息。然而，在基于机器学习的图像分类中，我们通常仅从显示对象的单个图像推断对象的类别。特别是对于具有挑战性的分类问题，单个图像传达的视觉信息可能不足以做出准确的决策。我们提出了一种分类方案，该方案依赖于融合通过从多个视角拍摄的同一对象的图像捕获的视觉信息。使用卷积神经网络从多个视图中提取和编码视觉特征，并提出了融合这些信息的策略。更具体地说，我们研究了以下三种策略：（1）融合在不同网络深度的卷积特征图；（2）在分类之前融合瓶颈潜在表示；以及（3）得分融合。我们在来自不同领域的三个数据集上系统地评估了这些策略。我们的研究结果强调了将信息融合到网络中而不是通过分类分数的后处理来执行的好处。此外，我们通过案例研究证明，已经训练好的网络可以通过最佳融合策略轻松扩展，并且大大优于其他方法。
