Rühlemann Christoph, Trujillo James
Deutsches Seminar - Germanistische Linguistik, University of Freiburg, Freiburg, Germany.
Institute for Logic, Language and Computation, University of Amsterdam, Amsterdam, Netherlands.
Front Psychol. 2024 Dec 27;15:1477263. doi: 10.3389/fpsyg.2024.1477263. eCollection 2024.
The key function of storytelling is a meeting of hearts: a resonance in the recipient(s) of the story narrator's emotion toward the story events. This paper focuses on the role of gestures in engendering emotional resonance in conversational storytelling. The paper asks three questions: Does story narrators' gesture expressivity increase from story onset to climax offset (RQ #1)? Does gesture expressivity predict specific EDA responses in story participants (RQ #2)? How important is the contribution of gesture expressivity to emotional resonance compared to the contribution of other predictors of resonance (RQ #3)? 53 conversational stories were annotated for a large number of variables including Protagonist, Recency, Group composition, Group size, Sentiment, and co-occurrence with quotation. The gestures in the stories were coded for gesture phases and gesture kinematics including Size, Force, Character view-point, Silence during gesture, Presence of hold phase, Co-articulation with other bodily organs, and Nucleus duration. The Gesture Expressivity Index (GEI) provides an average of these parameters. Resonating gestures were identified, i.e., gestures exhibiting concurrent specific EDA responses by two or more participants. The first statistical model, which addresses RQ #1, suggested that story narrators' gestures become more expressive from story onset to climax offset. The model constructed to adress RQ #2 suggested that increased gesture expressivity increases the probability of specific EDA responses. To address RQ #3 a Random Forest for emotional resonance as outcome variable and the seven GEI parameters as well as six more variables as predictors was constructed. All predictors were found to impact Eemotional resonance. Analysis of variable importance showed Group composition to be the most impactful predictor. Inspection of ICE plots clearly indicated combined effects of individual GEI parameters and other factors, including Group size and Group composition. This study shows that more expressive gestures are more likely to elicit physiological resonance between individuals, suggesting an important role for gestures in connecting people during conversational storytelling. Methodologically, this study opens up new avenues of multimodal corpus linguistic research by examining the interplay of emotion-related measurements and gesture at micro-analytic kinematic levels and using advanced machine-learning methods to deal with the inherent collinearity of multimodal variables.
故事叙述者对故事事件的情感在故事接受者心中产生共鸣。本文聚焦于手势在对话式讲故事中引发情感共鸣的作用。本文提出了三个问题：故事叙述者的手势表现力从故事开始到高潮结束是否增强（研究问题1）？手势表现力能否预测故事参与者特定的皮肤电反应（研究问题2）？与其他共鸣预测因素的贡献相比，手势表现力对情感共鸣的贡献有多重要（研究问题3）？53个对话式故事针对大量变量进行了标注，包括主人公、近期性、群体构成、群体规模、情感以及与引语的共现情况。故事中的手势根据手势阶段和手势运动学进行编码，包括大小、力度、角色视角、手势过程中的沉默、是否存在保持阶段、与其他身体器官的协同发音以及核心时长。手势表现力指数（GEI）是这些参数的平均值。识别出了共鸣手势，即两个或更多参与者同时表现出特定皮肤电反应的手势。解决研究问题1的第一个统计模型表明，故事叙述者的手势从故事开始到高潮结束变得更具表现力。为解决研究问题2构建的模型表明，增强的手势表现力会增加特定皮肤电反应的可能性。为解决研究问题3，构建了一个以情感共鸣为结果变量、七个GEI参数以及另外六个变量为预测因素的随机森林模型。发现所有预测因素都会影响情感共鸣。变量重要性分析表明群体构成是最具影响力的预测因素。对ICE图的检查清楚地表明了各个GEI参数与其他因素（包括群体规模和群体构成）的综合作用。这项研究表明，更具表现力的手势更有可能引发个体之间的生理共鸣，这表明手势在对话式讲故事过程中连接人们方面发挥着重要作用。在方法论上，这项研究通过在微观分析运动学层面研究与情感相关的测量指标和手势之间的相互作用，并使用先进的机器学习方法来处理多模态变量固有的共线性，开辟了多模态语料库语言学研究的新途径。
