Jacobs Robert A, Xu Chenliang
Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, USA.
Department of Computer Science, University of Rochester, Rochester, NY, USA.
J Vis. 2019 Sep 3;19(11):1. doi: 10.1167/19.11.1.
Although real-world environments are often multisensory, visual scientists typically study visual learning in unisensory environments containing visual signals only. Here, we use deep or artificial neural networks to address the question, Can multisensory training aid visual learning? We examine a network's internal representations of objects based on visual signals in two conditions: (a) when the network is initially trained with both visual and haptic signals, and (b) when it is initially trained with visual signals only. Our results demonstrate that a network trained in a visual-haptic environment (in which visual, but not haptic, signals are orientation-dependent) tends to learn visual representations containing useful abstractions, such as the categorical structure of objects, and also learns representations that are less sensitive to imaging parameters, such as viewpoint or orientation, that are irrelevant for object recognition or classification tasks. We conclude that researchers studying perceptual learning in vision-only contexts may be overestimating the difficulties associated with important perceptual learning problems. Although multisensory perception has its own challenges, perceptual learning can become easier when it is considered in a multisensory setting.
尽管现实世界的环境通常是多感官的，但视觉科学家通常在仅包含视觉信号的单感官环境中研究视觉学习。在此，我们使用深度神经网络或人工神经网络来解决这个问题：多感官训练能否辅助视觉学习？我们在两种情况下，基于视觉信号检查网络对物体的内部表征：(a) 当网络最初同时使用视觉和触觉信号进行训练时，以及 (b) 当它最初仅使用视觉信号进行训练时。我们的结果表明，在视觉 - 触觉环境中训练的网络（其中视觉信号而非触觉信号是方向依赖的）倾向于学习包含有用抽象信息的视觉表征，例如物体的类别结构，并且还学习对成像参数（如视角或方向）不太敏感的表征，这些参数与物体识别或分类任务无关。我们得出结论，在仅视觉环境中研究知觉学习的研究人员可能高估了与重要知觉学习问题相关的困难。尽管多感官知觉有其自身的挑战，但在多感官环境中考虑时，知觉学习会变得更容易。
