Bhandari Mohit, Templeman David, Tornetta Paul
Department of Clinical Epidemiology, McMaster University, Hamilton, Ontario, Canada.
Clin Orthop Relat Res. 2004 Jun(423):217-21. doi: 10.1097/01.blo.0000127584.02606.00.
Only a small proportion of submitted abstracts to the annual meeting of the Orthopaedic Trauma Association can be accepted for podium presentation. Annual program committee members must ensure that the selection of abstracts is free from bias and transparent to investigators. The objectives of this study are to examine the consistency of reviewers in grading abstracts submitted for podium presentations at the 2001 and 2002 Annual Meetings of the Orthopaedic Trauma Association and to evaluate whether the grades of the actual podium presentations at the meeting are consistent with the grades based on abstracts only. Reviewers independently graded all abstracts submitted to the Orthopaedic Trauma Association for presentation in a blinded manner. Abstracts submitted by members of the review panel were independently adjudicated by six reviewers who were not members of the committee. Before final decision-making, all reviewers met to discuss the abstracts submitted for oral presentation. Among the 440 papers reviewed in 2001 and 438 papers reviewed in 2002, the interreviewer reliability for abstract review was 0.23 and 0.27, respectively. Despite disagreements in the quality of the abstracts, reviewers achieved consensus by discussions to determine the final program. Agreement among unblinded reviewers of the 67 and 73 podium presentations during the 2001 and 2002 meetings, respectively, did not improve interreviewer agreement. Of the papers of the 2002 meeting that ultimately ranked in the top 20 after the full presentation of the papers, 15 papers originally had been ranked less than 20 in the initial grading. Only one of the top three papers of the meeting originally was ranked in the top three before the meeting.
提交给骨科创伤协会年会的摘要中，只有一小部分能够被接受进行大会发言。年会项目委员会成员必须确保摘要的选择过程没有偏见，且对研究者透明。本研究的目的是检验评审者在对提交至2001年和2002年骨科创伤协会年会进行大会发言的摘要评分时的一致性，并评估会议上实际大会发言的评分是否仅与基于摘要的评分一致。评审者以盲法独立对所有提交给骨科创伤协会进行发言的摘要进行评分。评审小组成员提交的摘要由六名非委员会成员的评审者独立评判。在做出最终决定前，所有评审者会面讨论提交进行口头报告的摘要。在2001年评审的440篇论文和2002年评审的438篇论文中，摘要评审的评审者间信度分别为0.23和0.27。尽管在摘要质量上存在分歧，但评审者通过讨论达成共识以确定最终议程。2001年和2002年会议期间，分别对67个和73个大会发言进行非盲评审的评审者之间的一致性并未提高评审者间的一致性。在2002年会议的论文在完整报告后最终排名前20的论文中，有15篇在最初评分时排名低于20。会议排名前三的论文中只有一篇在会前最初排名前三。
