Rastogi Charvi, Song Xiangchen, Jin Zhijing, Stelmakh Ivan, Daumé Hal, Zhang Kun, Shah Nihar B
Machine Learning Department, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
Computer Science Department, ETH Zürich, Zürich, Switzerland.
PLoS One. 2024 Dec 27;19(12):e0315674. doi: 10.1371/journal.pone.0315674. eCollection 2024.
Many peer-review processes involve reviewers submitting their independent reviews, followed by a discussion between the reviewers of each paper. A common question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this question by conducting a randomized controlled trial at the Conference on Uncertainty in Artificial Intelligence (UAI) 2022 conference where reviewer discussions were conducted over a typed forum. We randomly split the reviewers and papers into two conditions-one with anonymous discussions and the other with non-anonymous discussions. We also conduct an anonymous survey of all reviewers to understand their experience and opinions. We compare the two conditions in terms of the amount of discussion, influence of seniority on the final decisions, politeness, reviewers' self-reported experiences and preferences. Overall, this experiment finds small, significant differences favoring the anonymous discussion setup based on the evaluation criteria considered in this work.
许多同行评审过程包括评审人员提交独立评审意见，然后对每篇论文的评审人员进行讨论。政策制定者们普遍关心的一个问题是，在讨论过程中，论文的评审人员是否应该相互匿名。我们通过在2022年人工智能不确定性会议（UAI）上进行一项随机对照试验，来阐明这个问题，该会议的评审人员讨论是在一个打字论坛上进行的。我们将评审人员和论文随机分为两种情况——一种是匿名讨论，另一种是非匿名讨论。我们还对所有评审人员进行了匿名调查，以了解他们的经验和意见。我们从讨论量、资历对最终决定的影响、礼貌程度、评审人员自我报告的经验和偏好等方面对这两种情况进行比较。总体而言，根据这项工作中考虑的评估标准，该实验发现了一些有利于匿名讨论设置的微小但显著的差异。
