Baxt W G, Waeckerle J F, Berlin J A, Callaham M L
Annals of Emergency Medicine, Irving, TX, USA.
Ann Emerg Med. 1998 Sep;32(3 Pt 1):310-7. doi: 10.1016/s0196-0644(98)70006-x.
To determine whether a fictitious manuscript into which purposeful errors were placed could be used as an instrument to evaluate peer reviewer performance.
An instrument for reviewer evaluation was created in the form of a fictitious manuscript into which deliberate errors were placed in order to develop an approach for the analysis of peer reviewer performance. The manuscript described a double-blind, placebo control study purportedly demonstrating that intravenous propranolol reduced the pain of acute migraine headache. There were 10 major and 13 minor errors placed in the manuscript. The work was distributed to all reviewers of Annals of Emergency Medicine for review.
The manuscript was sent to 262 reviewers; 203 (78%) reviews were returned. One-hundred ninety-nine reviewers recommended a disposition for the manuscript: 15 recommended acceptance, 117 rejection, and 67 revision. The 15 who recommended acceptance identified 17.3% (95% confidence interval [CI] 11.3% to 23.4%) of the major and 11.8% (CI 7.3% to 16.3%) of the minor errors. The 117 who recommended rejection identified 39.1 % (CI 36.3% to 41.9%) of the major and 25.2% (CI 23.0% to 27.4%) of the minor errors. The 67 who recommended revision identified 29.6% (CI 26.1% to 33.1%) of the major and 22.0% (CI 19.3% to 24.8%) of the minor errors. The number of errors identified differed significantly across recommended disposition. Sixty-eight percent of the reviewers did not realize that the conclusions of the work were not supported by the results.
These data suggest that the use of a preconceived manuscript into which purposeful errors are placed may be a viable approach to evaluate reviewer performance. Peer reviewers in this study failed to identify two thirds of the major errors in such a manuscript.
确定一份故意插入错误的虚构手稿能否用作评估同行评审员表现的工具。
以一份虚构手稿的形式创建了一个评审员评估工具，其中故意插入错误，以开发一种分析同行评审员表现的方法。该手稿描述了一项双盲、安慰剂对照研究，据称证明静脉注射普萘洛尔可减轻急性偏头痛的疼痛。手稿中设置了10个主要错误和13个次要错误。该作品分发给了《急诊医学年鉴》的所有评审员进行评审。
该手稿被发送给262名评审员；203份（78%）评审意见被返回。199名评审员对手稿给出了处理建议：15人建议接受，117人建议拒绝，67人建议修改。建议接受的15人发现了17.3%（95%置信区间[CI]11.3%至23.4%）的主要错误和11.8%（CI 7.3%至16.3%）的次要错误。建议拒绝的117人发现了39.1%（CI 36.3%至41.9%）的主要错误和25.2%（CI 23.0%至27.4%）的次要错误。建议修改的67人发现了29.6%（CI 26.1%至33.1%）的主要错误和22.0%（CI 19.3%至24.8%）的次要错误。不同处理建议所发现的错误数量有显著差异。68%的评审员没有意识到该研究结果不支持其结论。
这些数据表明，使用一份预先设定并故意插入错误的手稿可能是评估评审员表现的一种可行方法。本研究中的同行评审员未能识别出此类手稿中三分之二的主要错误。
