Weill Cornell Medical College, New York, NY, USA.
Spine J. 2010 Mar;10(3):209-11. doi: 10.1016/j.spinee.2009.12.003.
It is commonly believed that the peer-review process is reliable and consistent. It appears, however, that depending on the journal and the editorial leadership, agreement by reviewers on whether to publish submitted articles varies widely; from substantial to slightly greater than one would expect with random assignments of acceptance or rejection.
The purpose was to assess peer-review agreement in major spine journals.
STUDY DESIGN/SETTING: This study is for the assessment of reviewer agreement.
The study consisted of consecutive reviews of 200 submitted articles.
Agreement via Kappa statistics.
Group A consisted of 200 consecutive article reviews for which the senior author was involved in the review or editorial process over the past 8 years for two major spine journals. Reviewers' recommendations were placed into one of two groups: accept/minimal revisions or major revision/reject. Standard Kappa statistics were used to assess reviewer agreement. Group B consisted of a similar set, but with wholly randomly generated recommendations. Again, Kappa statistics were used.
Kappa for Group A was 0.155 with a range of 0.017 to 0.294 at 95% confidence interval and agreement at 0.6; suggesting "slight" reviewer agreement. Kappa for Group B behaved as expected, with "poor" agreement.
Agreement regarding peer-review recommendations for publication in spine journals appears to be better than would be expected in the random situation; but still only "slight." This suggests that review methodology varies considerably among reviewers and that further study should be undertaken to determine "ideal" agreement levels and ways to increase review consistency/quality commensurate with the editorial missions of the journals.
人们普遍认为同行评审过程是可靠且一致的。然而，似乎取决于期刊和编辑领导层，审稿人是否同意发表提交的文章差异很大；从实质性的到略高于随机分配接受或拒绝的预期。
评估主要脊柱期刊的同行评审一致性。
研究设计/设置：本研究旨在评估审稿人意见的一致性。
该研究包括对 200 篇提交文章的连续审查。
一致性通过 Kappa 统计量评估。
A 组由 200 篇连续的文章审查组成，在过去 8 年中，资深作者参与了这两个主要脊柱期刊的审查或编辑过程。审稿人的建议分为两组之一：接受/最小修改或重大修改/拒绝。使用标准 Kappa 统计量评估审稿人意见的一致性。B 组由类似的一组组成，但建议是完全随机生成的。同样，使用 Kappa 统计量。
A 组的 Kappa 值为 0.155，95%置信区间范围为 0.017 至 0.294，一致性为 0.6，表明审稿人意见“略有”一致。B 组的 Kappa 值表现符合预期，表明“较差”的一致性。
在脊柱期刊上发表的同行评审建议的一致性似乎优于随机情况下的预期；但仍只是“略有”。这表明审稿方法在审稿人之间存在很大差异，应进一步研究以确定“理想”的一致性水平，并找到提高审查一致性/质量的方法，以符合期刊的编辑使命。
