Faculty of Applied Linguistics, University of Warsaw, Warsaw, Poland.
School of Politics, Philosophy, Language and Communication Studies, University of East Anglia, Norwich, United Kingdom.
PLoS One. 2024 Oct 7;19(10):e0306251. doi: 10.1371/journal.pone.0306251. eCollection 2024.
Every day, millions of viewers worldwide engage with subtitled content, and an increasing number choose to watch without sound. In this mixed-methods study, we examine the impact of sound presence or absence on the viewing experience of both first-language (L1) and second-language (L2) viewers when they watch subtitled videos. We explore this novel phenomenon through comprehension and recall post-tests, self-reported cognitive load, immersion, and enjoyment measures, as well as gaze pattern analysis using eye tracking. We also investigate viewers' motivations for opting for audiovisual content without sound and explore how the absence of sound impacts their viewing experience, using in-depth, semi-structured interviews. Our goal is to ascertain whether these effects are consistent among L2 and L1 speakers from different language varieties. To achieve this, we tested L1-British English, L1-Australian English and L2-English (L1-Polish) language speakers (n = 168) while they watched English-language audiovisual material with English subtitles with and without sound. The findings show that when watching videos without sound, viewers experienced increased cognitive load, along with reduced comprehension, immersion and overall enjoyment. Examination of participants' gaze revealed that the absence of sound significantly affected the viewing experience, increasing the need for subtitles and thus increasing the viewers' propensity to process them more thoroughly. The absence of sound emerged as a global constraint that made reading more effortful. Triangulating data from multiple sources made it possible to tap into some of the metacognitive strategies employed by viewers to maintain comprehension in the absence of sound. We discuss the implications within the context of the growing trend of watching subtitled videos without sound, emphasising its potential impact on cognitive processes and the viewing experience.
每天，全世界数以百万计的观众都在观看字幕内容，越来越多的人选择在没有声音的情况下观看。在这项混合方法研究中，我们研究了当观众观看带有字幕的视频时，声音的存在或缺失对母语（L1）和第二语言（L2）观众的观看体验的影响。我们通过理解和回忆后的测试、自我报告的认知负荷、沉浸感和享受度的测量，以及使用眼动追踪的注视模式分析来探索这一新颖的现象。我们还调查了观众选择无声音的视听内容的动机，并探讨了声音的缺失如何影响他们的观看体验，采用了深度半结构化访谈。我们的目标是确定这些影响在来自不同语言变体的 L2 和 L1 说话者中是否一致。为了实现这一目标，我们测试了母语为英语（L1-英国英语、L1-澳大利亚英语）和第二语言为英语（L2-英语（L1-波兰语））的语言说话者（n=168），他们观看带有英文字幕的英语视听材料，有声音和无声音。研究结果表明，在观看无声音视频时，观众体验到认知负荷增加，理解能力、沉浸感和整体享受度降低。对参与者的注视进行检查后发现，声音的缺失显著影响了观看体验，增加了对字幕的需求，从而增加了观众更仔细地处理字幕的倾向。声音的缺失成为了一种全局约束，使得阅读更加费力。从多个来源获取的数据可以深入了解观众在没有声音的情况下维持理解所采用的一些元认知策略。我们在观看无声音字幕视频的趋势不断增长的背景下讨论了这些影响，强调了其对认知过程和观看体验的潜在影响。
