Li Mingtong, Aussems Suzanne, Kita Sotaro
Department of Psychology, University of Warwick, Coventry, CV4 7AL, UK.
Psychon Bull Rev. 2025 May 12. doi: 10.3758/s13423-025-02698-2.
Iconicity is the resemblance or similarity between the form of a signal and its meaning. In two studies, we investigated whether adults interpret iconicity in speech and gesture via a modality-independent mechanism (Study 1, N = 40; Study 2, N = 348). Participants in both studies completed two verb-action matching tasks. In these tasks, either spoken verbs or co-speech gestures were manipulated at different rates to iconically represent actions (i.e., manners of locomotion) performed at corresponding rates. We hypothesized that adult participants would map verbs spoken at a fast rate to fast actions and verbs spoken at a slow rate to slow actions. Similarly, we hypothesized that adult participants would map gestures produced with faster hand movements to fast actions and gestures produced with slower hand movements to slow actions. Moreover, we predicted a positive correlation between the two task performances. In both a pilot study (Study 1) and a preregistered replication (Study 2), participants successfully used iconic cues to match verbs to actions in both verb-action matching tasks. Crucially, adult participants who performed well in the verb-action matching task with iconic speech cues also performed well in the verb-action matching task with iconic gesture cues. This weak positive correlation remained significant even after controlling verbal working memory performance. These findings suggest that adults may rely on a modality-independent cognitive mechanism for interpreting iconicity both within and across different sensory modalities, using both visual information from gestures and auditory information from speech to identify referent actions.
象似性是指信号形式与其意义之间的相似性。在两项研究中，我们调查了成年人是否通过一种模态独立机制来理解言语和手势中的象似性（研究1，N = 40；研究2，N = 348）。两项研究中的参与者都完成了两个动词 - 动作匹配任务。在这些任务中，口语动词或伴随言语的手势以不同速率被操控，以象似性地表现以相应速率执行的动作（即移动方式）。我们假设成年参与者会将快速说出的动词映射到快速动作上，将慢速说出的动词映射到慢速动作上。同样，我们假设成年参与者会将手部动作较快产生的手势映射到快速动作上，将手部动作较慢产生的手势映射到慢速动作上。此外，我们预测两项任务表现之间存在正相关。在一项先导研究（研究1）和一项预先注册的重复研究（研究2）中，参与者在两个动词 - 动作匹配任务中都成功地利用象似性线索将动词与动作进行了匹配。至关重要的是，在带有象似性言语线索的动词 - 动作匹配任务中表现良好的成年参与者，在带有象似性手势线索的动词 - 动作匹配任务中也表现良好。即使在控制了言语工作记忆表现之后，这种微弱的正相关仍然显著。这些发现表明，成年人可能依赖一种模态独立的认知机制，利用来自手势的视觉信息和来自言语的听觉信息来识别所指动作，从而在不同感官模态内和跨不同感官模态理解象似性。
