IEEE Trans Image Process. 2017 Aug;26(8):3995-4004. doi: 10.1109/TIP.2017.2707858. Epub 2017 May 24.
Cross-modal recognition and matching with privileged information are important challenging problems in the field of computer vision. The cross-modal scenario deals with matching across different modalities and needs to take care of the large variations present across and within each modality. The privileged information scenario deals with the situation that all the information available during training may not be available during the testing stage, and hence, algorithms need to leverage the extra information from the training stage itself. We show that for multi-modal data, either one of the above situations may arise if one modality is absent during testing. Here, we propose a novel framework, which can handle both these scenarios seamlessly with applications to matching multi-modal data. The proposed approach jointly uses data from the two modalities to build a canonical representation, which encompasses information from both the modalities. We explore four different types of canonical representations for different types of data. The algorithm computes dictionaries and canonical representation for data from both the modalities, such that the transformed sparse coefficients of both the modalities are equal to that of the canonical representation. The sparse coefficients are finally matched using Mahalanobis metric. Extensive experiments on different data sets, involving RGBD, text-image, and audio-image data, show the effectiveness of the proposed framework.
跨模态识别和匹配与特权信息是计算机视觉领域的重要挑战性问题。跨模态场景涉及跨不同模式的匹配，需要注意不同模式之间以及每个模式内部存在的巨大差异。特权信息场景涉及的情况是，在训练阶段可用的所有信息在测试阶段可能不可用，因此，算法需要利用训练阶段本身的额外信息。我们表明，如果在测试阶段缺少一种模式，则对于多模态数据，上述两种情况中的任何一种都可能出现。在这里，我们提出了一个新的框架，可以无缝地处理这两种情况，并应用于多模态数据的匹配。所提出的方法联合使用两个模态的数据来构建一个规范表示，该表示包含来自两个模态的信息。我们针对不同类型的数据探索了四种不同类型的规范表示。该算法为两个模态的数据计算字典和规范表示，使得两个模态的变换稀疏系数等于规范表示的稀疏系数。最后，使用马氏距离匹配稀疏系数。在涉及 RGBD、文本-图像和音频-图像数据的不同数据集上进行的广泛实验表明了所提出框架的有效性。
