Mayn Alexandra, Demberg Vera
Department of Language of Science and Technology, Saarland University.
Department of Computer Science, Saarland University.
Open Mind (Camb). 2023 Jun 1;7:156-178. doi: 10.1162/opmi_a_00077. eCollection 2023.
Formal probabilistic models, such as the Rational Speech Act model, are widely used for formalizing the reasoning involved in various pragmatic phenomena, and when a model achieves good fit to experimental data, that is interpreted as evidence that the model successfully captures some of the underlying processes. Yet how can we be sure that participants' performance on the task is the result of successful reasoning and not of some feature of experimental setup? In this study, we carefully manipulate the properties of the stimuli that have been used in several pragmatics studies and elicit participants' reasoning strategies. We show that certain biases in experimental design inflate participants' performance on the task. We then repeat the experiment with a new version of stimuli which is less susceptible to the identified biases, obtaining a somewhat smaller effect size and more reliable estimates of individual-level performance.
形式概率模型，如理性言语行为模型，被广泛用于将各种语用现象中涉及的推理形式化，当一个模型与实验数据拟合良好时，这被解释为该模型成功捕捉了一些潜在过程的证据。然而，我们如何确定参与者在任务中的表现是成功推理的结果，而不是实验设置的某些特征导致的呢？在本研究中，我们仔细操纵了几项语用学研究中使用的刺激属性，并引出了参与者的推理策略。我们表明，实验设计中的某些偏差夸大了参与者在任务中的表现。然后，我们用一个新版本的刺激重复实验，该刺激对已识别的偏差不太敏感，从而获得了略小的效应量和更可靠的个体水平表现估计。
