Onat Selim, Libertus Klaus, König Peter
Institute of Cognitive Science, University of Osnabrück, Germany.
J Vis. 2007 Jul 25;7(10):11.1-16. doi: 10.1167/7.10.11.
In everyday life, our brains decide about the relevance of huge amounts of sensory input. Further complicating this situation, this input is distributed over different modalities. This raises the question of how different sources of information interact for the control of overt attention during free exploration of the environment under natural conditions. Different modalities may work independently or interact to determine the consequent overt behavior. To answer this question, we presented natural images and lateralized natural sounds in a variety of conditions and we measured the eye movements of human subjects. We show that, in multimodal conditions, fixation probabilities increase on the side of the image where the sound originates showing that, at a coarser scale, lateralized auditory stimulation topographically increases the salience of the visual field. However, this shift of attention is specific because the probability of fixation of a given location on the side of the sound scales with the saliency of the visual stimulus, meaning that the selection of fixation points during multimodal conditions is dependent on the saliencies of both auditory and visual stimuli. Further analysis shows that a linear combination of both unimodal saliencies provides a good model for this integration process, which is optimal according to information-theoretical criteria. Our results support a functional joint saliency map, which integrates different unimodal saliencies before any decision is taken about the subsequent fixation point. These results provide guidelines for the performance and architecture of any model of overt attention that deals with more than one modality.
在日常生活中，我们的大脑会对大量的感官输入信息的相关性进行判断。而使这种情况更为复杂的是，这些输入信息分布于不同的感觉模态。这就引出了一个问题：在自然条件下自由探索环境时，不同的信息源是如何相互作用以控制外显注意力的。不同的感觉模态可能独立起作用，也可能相互作用来决定随之产生的外显行为。为了回答这个问题，我们在多种条件下呈现自然图像和侧向化的自然声音，并测量了人类受试者的眼球运动。我们发现，在多模态条件下，注视概率会在声音发出一侧的图像区域增加，这表明在较粗略的尺度上，侧向化的听觉刺激会在地形学上增加视野的显著性。然而，这种注意力的转移是特定的，因为声音一侧给定位置的注视概率会随着视觉刺激的显著性而变化，这意味着在多模态条件下注视点的选择取决于听觉和视觉刺激的显著性。进一步的分析表明，两种单模态显著性的线性组合为这种整合过程提供了一个良好的模型，根据信息理论标准，这个模型是最优的。我们的结果支持了一种功能性联合显著性图谱，它在对后续注视点做出任何决定之前，会整合不同的单模态显著性。这些结果为任何处理多种感觉模态的外显注意力模型的性能和架构提供了指导。
