Liu Fen, Chen Jianfeng, Li Kemeng, Tan Weijie, Cai Chang, Ayub Muhammad Saad
School of Marine Science and Technology, Northwestern Polytechnical University, Xi'an 710072, China.
College of Mathematics and Computer Science, Yan'an University, Yan'an 716000, China.
Entropy (Basel). 2022 Dec 16;24(12):1836. doi: 10.3390/e24121836.
Multi-modal fusion can exploit complementary information from various modalities and improve the accuracy of prediction or classification tasks. In this paper, we propose a parallel, multi-modal, factorized, bilinear pooling method based on a semi-tensor product (STP) for information fusion in emotion recognition. Initially, we apply the STP to factorize a high-dimensional weight matrix into two low-rank factor matrices without dimension matching constraints. Next, we project the multi-modal features to the low-dimensional matrices and perform multiplication based on the STP to capture the rich interactions between the features. Finally, we utilize an STP-pooling method to reduce the dimensionality to get the final features. This method can achieve the information fusion between modalities of different scales and dimensions and avoids data redundancy due to dimension matching. Experimental verification of the proposed method on the emotion-recognition task using the IEMOCAP and CMU-MOSI datasets showed a significant reduction in storage space and recognition time. The results also validate that the proposed method improves the performance and reduces both the training time and the number of parameters.
多模态融合可以利用来自各种模态的互补信息，并提高预测或分类任务的准确性。在本文中，我们提出了一种基于半张量积（STP）的并行、多模态、因式分解双线性池化方法，用于情感识别中的信息融合。首先，我们应用STP将高维权重矩阵分解为两个低秩因子矩阵，而无需维度匹配约束。接下来，我们将多模态特征投影到低维矩阵上，并基于STP进行乘法运算，以捕捉特征之间丰富的交互。最后，我们利用STP池化方法降低维度以获得最终特征。该方法可以实现不同尺度和维度的模态之间的信息融合，并避免由于维度匹配导致的数据冗余。使用IEMOCAP和CMU-MOSI数据集对所提出的方法在情感识别任务上进行的实验验证表明，存储空间和识别时间显著减少。结果还验证了所提出的方法提高了性能，并减少了训练时间和参数数量。
