Amerineni Rajesh, Gupta Resh S, Gupta Lalit
Department of Electrical and Computer Engineering, Southern Illinois University, Carbondale, IL 62901, USA.
Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN 37232, USA.
Brain Sci. 2019 Jan 2;9(1):3. doi: 10.3390/brainsci9010003.
Two multimodal classification models aimed at enhancing object classification through the integration of semantically congruent unimodal stimuli are introduced. The feature-integrating model, inspired by multisensory integration in the subcortical superior colliculus, combines unimodal features which are subsequently classified by a multimodal classifier. The decision-integrating model, inspired by integration in primary cortical areas, classifies unimodal stimuli independently using unimodal classifiers and classifies the combined decisions using a multimodal classifier. The multimodal classifier models are implemented using multilayer perceptrons and multivariate statistical classifiers. Experiments involving the classification of noisy and attenuated auditory and visual representations of ten digits are designed to demonstrate the properties of the multimodal classifiers and to compare the performances of multimodal and unimodal classifiers. The experimental results show that the multimodal classification systems exhibit an important aspect of the "inverse effectiveness principle" by yielding significantly higher classification accuracies when compared with those of the unimodal classifiers. Furthermore, the flexibility offered by the generalized models enables the simulations and evaluations of various combinations of multimodal stimuli and classifiers under varying uncertainty conditions.
本文介绍了两种多模态分类模型，旨在通过整合语义一致的单模态刺激来增强目标分类。受皮层下上丘多感官整合的启发，特征整合模型将单模态特征进行组合，随后由多模态分类器进行分类。受初级皮层区域整合的启发，决策整合模型使用单模态分类器对单模态刺激进行独立分类，并使用多模态分类器对组合决策进行分类。多模态分类器模型使用多层感知器和多元统计分类器来实现。设计了涉及对十个数字的噪声和衰减听觉及视觉表示进行分类的实验，以证明多模态分类器的特性，并比较多模态和单模态分类器的性能。实验结果表明，与单模态分类器相比，多模态分类系统在分类准确率上显著更高，展现出了“逆有效性原则”的一个重要方面。此外，广义模型所提供的灵活性使得在不同不确定性条件下对多模态刺激和分类器的各种组合进行模拟和评估成为可能。
