Nylenna M, Riis P, Karlsson Y
Journal of the Norwegian Medical Association, Lysaker.
JAMA. 1994 Jul 13;272(2):149-51.
To study the association between referee characteristics and their manuscript assessments, the influence of manuscript language on referees' judgments, and the usefulness, quality, and extent of referees' free-text comments.
Two nonauthentic, but realistic, short manuscripts with a number of common methodological flaws were sent to 180 Scandinavian referees. Through randomization, each referee received one of the manuscripts in English and the other manuscript in the national language. A structured assessment of the manuscript quality was expressed on a 5-point scale, and the impact of referee characteristics (age, gender, experience, and so on) was analyzed by multiple linear regression.
Manuscript quality assessed by referees.
A total of 312 reviews from 156 referees could be used for the study of referee characteristics and language. With increasing experience, the referees gave lower quality scores (P < .05). A tendency toward stricter assessment with younger age was seen (P < .05). No influence of referees' gender, specialty, or nationality was found. For the test manuscript of the poorest quality, the English version was assessed to be better than the national-language version (P < .05). A total of 159 of 312 reviews included free-text comments applicable for the methodological study. In 54 reviews (34%), no methodological comments accompanied the assessment, and in six reviews they were only incomplete. Wrong sampling unit was mentioned by one fourth of 80 referees. Only one referee mentioned the incorrect use of a parametric test in the analysis of data whose distribution was nonparametric.
Experienced and young referees gave a stricter assessment of the manuscripts than their less experienced and older colleagues. An English version seemed to be accepted more easily than a national-language version of the same manuscript. Most referees spontaneously mentioned the shortcomings of the manuscripts only as part of their overall judgment.
研究审稿人特征与其稿件评估之间的关联、稿件语言对审稿人判断的影响，以及审稿人自由文本评论的有用性、质量和范围。
将两篇存在一些常见方法学缺陷的非真实但现实的短文稿发送给180名斯堪的纳维亚审稿人。通过随机分组，每位审稿人收到一篇英文稿件和另一篇本国语言稿件。稿件质量的结构化评估采用5分制，通过多元线性回归分析审稿人特征（年龄、性别、经验等）的影响。
审稿人评估的稿件质量。
来自156名审稿人的312份评审意见可用于研究审稿人特征和语言。随着经验增加，审稿人给出的质量评分更低（P < .05）。发现有年龄越小评估越严格的趋势（P < .05）。未发现审稿人的性别、专业或国籍有影响。对于质量最差的测试稿件，英文版本的评估结果优于本国语言版本（P < .05）。312份评审意见中有159份包含适用于方法学研究的自由文本评论。在54份评审意见（34%）中，评估时未附带任何方法学评论，6份评审意见中的评论不完整。80名审稿人中有四分之一提到了错误的抽样单位。在对分布为非参数的数据进行分析时，只有一名审稿人提到了参数检验的错误使用。
经验丰富和年轻的审稿人对稿件的评估比经验不足和年长的同事更严格。同一稿件的英文版本似乎比本国语言版本更容易被接受。大多数审稿人只是在整体判断中自发提及稿件的缺点。
