Research School of Behavioural and Cognitive Neuroscience (BCN), University of Groningen, Groningen, The Netherlands; Laboratory of Experimental Ophthalmology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands; Department of Otorhinolaryngology - Head and Neck Surgery, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.
Institute of Acoustics, Technische Hochschule Lübeck, Lübeck, Germany.
Vision Res. 2021 Mar;180:51-62. doi: 10.1016/j.visres.2020.12.002. Epub 2020 Dec 24.
Emotion recognition requires optimal integration of the multisensory signals from vision and hearing. A sensory loss in either or both modalities can lead to changes in integration and related perceptual strategies. To investigate potential acute effects of combined impairments due to sensory information loss only, we degraded the visual and auditory information in audiovisual video-recordings, and presented these to a group of healthy young volunteers. These degradations intended to approximate some aspects of vision and hearing impairment in simulation. Other aspects, related to advanced age, potential health issues, but also long-term adaptation and cognitive compensation strategies, were not included in the simulations. Besides accuracy of emotion recognition, eye movements were recorded to capture perceptual strategies. Our data show that emotion recognition performance decreases when degraded visual and auditory information are presented in isolation, but simultaneously degrading both modalities does not exacerbate these isolated effects. Moreover, degrading the visual information strongly impacts recognition performance and on viewing behavior. In contrast, degrading auditory information alongside normal or degraded video had little (additional) effect on performance or gaze. Nevertheless, our results hold promise for visually impaired individuals, because the addition of any audio to any video greatly facilitates performance, even though adding audio does not completely compensate for the negative effects of video degradation. Additionally, observers modified their viewing behavior to degraded video in order to maximize their performance. Therefore, optimizing the hearing of visually impaired individuals and teaching them such optimized viewing behavior could be worthwhile endeavors for improving emotion recognition.
情感识别需要对视听觉多模态信号进行最佳整合。单一或两者模态的感觉丧失都会导致整合和相关感知策略发生变化。为了研究仅由于感觉信息丧失导致的联合损伤的潜在急性影响，我们对视听视频记录中的视觉和听觉信息进行了降级，并将这些信息呈现给一组健康的年轻志愿者。这些降级旨在模拟视觉和听觉损伤的某些方面。其他方面，与年龄增长、潜在健康问题有关，但也与长期适应和认知补偿策略有关，并未包含在模拟中。除了情绪识别的准确性之外，还记录了眼动以捕捉感知策略。我们的数据表明，当单独呈现降级的视觉和听觉信息时，情绪识别性能会下降，但同时降级两种模式并不会加剧这些孤立的影响。此外，降级视觉信息会强烈影响识别性能和观看行为。相比之下，在正常或降级视频中同时降级听觉信息对性能或注视几乎没有（额外）影响。然而，我们的结果为视力受损个体带来了希望，因为向任何视频添加任何音频都会极大地提高性能，即使添加音频并不能完全弥补视频降级的负面影响。此外，观察者为了最大限度地提高性能，会调整他们观看降级视频的行为。因此，优化视力受损个体的听力并教他们这种优化的观看行为可能是提高情绪识别的值得努力的方向。
