Zhang Sam, Heck Patrick R, Meyer Michelle N, Chabris Christopher F, Goldstein Daniel G, Hofman Jake M
Department of Applied Mathematics, University of Colorado, Boulder, CO 80309.
Office of Research, Consumer Financial Protection Bureau, Washington, DC 20552.
Proc Natl Acad Sci U S A. 2023 Aug 15;120(33):e2302491120. doi: 10.1073/pnas.2302491120. Epub 2023 Aug 9.
Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.
传统上，与结果变异性（即个体结果的可预测性）相比，科学家更强调传达推断不确定性（即统计估计的精度）。在此，我们表明这可能导致对科学结果含义的重大误解。具体而言，我们展示了三个预先注册的随机实验，参与者看到相同的科学发现，分别以仅显示推断不确定性、仅显示结果变异性或两者都显示的方式呈现，并回答关于所展示发现的规模和重要性的问题。我们的结果由医学专业人员、专业数据科学家和终身教职教员的回答组成，表明仅可视化推断不确定性的普遍形式会导致对治疗效果的显著高估，即使在训练有素的专家中也是如此。相比之下，我们发现描绘推断不确定性和结果变异性两者会导致对结果的更准确认知，同时平均而言似乎不会改变对结果的其他主观印象。
