Machine Learning Department and Computer Science Department, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
PLoS One. 2023 Jun 21;18(6):e0286206. doi: 10.1371/journal.pone.0286206. eCollection 2023.
There is widespread debate on whether to anonymize author identities in peer review. The key argument for anonymization is to mitigate bias, whereas arguments against anonymization posit various uses of author identities in the review process. The Innovations in Theoretical Computer Science (ITCS) 2023 conference adopted a middle ground by initially anonymizing the author identities from reviewers, revealing them after the reviewer had submitted their initial reviews, and allowing the reviewer to change their review subsequently. We present an analysis of the reviews pertaining to the identification and use of author identities. Our key findings are: (I) A majority of reviewers self-report not knowing and being unable to guess the authors' identities for the papers they were reviewing. (II) After the initial submission of reviews, 7.1% of reviews changed their overall merit score and 3.8% changed their self-reported reviewer expertise. (III) There is a very weak and statistically insignificant correlation of the rank of authors' affiliations with the change in overall merit; there is a weak but statistically significant correlation with respect to change in reviewer expertise. We also conducted an anonymous survey to obtain opinions from reviewers and authors. The main findings from the 200 survey responses are: (i) A vast majority of participants favor anonymizing author identities in some form. (ii) The "middle-ground" initiative of ITCS 2023 was appreciated. (iii) Detecting conflicts of interest is a challenge that needs to be addressed if author identities are anonymized. Overall, these findings support anonymization of author identities in some form (e.g., as was done in ITCS 2023), as long as there is a robust and efficient way to check conflicts of interest.
关于是否在同行评审中对作者身份进行匿名化存在广泛的争论。匿名化的主要论点是减轻偏见，而反对匿名化的论点则认为作者身份在评审过程中有各种用途。2023 年理论计算机科学创新会议（ITCS）采取了中间立场，最初对评审员的作者身份进行了匿名处理，在评审员提交初步评审后揭示了这些身份，并允许评审员随后更改其评审。我们对与识别和使用作者身份有关的评论进行了分析。我们的主要发现是：（一）大多数评审员自我报告说，他们不知道也无法猜测他们正在评审的论文的作者身份。（二）在初始评论提交后，7.1%的评论改变了他们对论文整体价值的评分，3.8%改变了他们自我报告的评审员专业知识。（三）作者所属机构的排名与整体价值的变化之间只有非常微弱且无统计学意义的相关性；与评审员专业知识的变化之间存在较弱但具有统计学意义的相关性。我们还进行了一项匿名调查，以获取评审员和作者的意见。从 200 份调查回复中得出的主要发现是：（一）绝大多数参与者赞成以某种形式对作者身份进行匿名化。（二）2023 年 ITCS 的“中间立场”倡议受到赞赏。（三）如果对作者身份进行匿名化，那么检测利益冲突是一个需要解决的挑战。总的来说，这些发现支持以某种形式对作者身份进行匿名化（例如，在 ITCS 2023 中所做的那样），只要有一个强大而高效的方法来检查利益冲突。
