Office of Consultation and Research in Medical Education, University of Iowa Carver Collegeof Medicine, Iowa City, IA 52242, USA.
Teach Learn Med. 2013;25(1):103-7. doi: 10.1080/10401334.2012.741548.
BACKGROUND. Admission decisions require that information about an applicant be combined using either holistic (human judges) or statistical (actuarial) methods. For optimizing a defined measureable outcome, there is a consistent body of research evidence demonstrating that statistical methods yield superior decisions compared to those generated by judges. It is possible, however, that the benefits of holistic decisions are reflected in unmeasured outcomes. If such benefits exist, they would necessarily appear as systematic variance in raters' scores that deviate from statistically-based decisions. PURPOSE. To estimate this variance, we propose a design examining the interrater reliability of difference scores (i.e., the difference between observed committee rankings and rankings based on statistical approaches). METHODS. Example calculations and G study models are presented to demonstrate how rater agreement on difference scores can be analyzed under various circumstances. High interrater reliability of difference scores would support but not prove the assertion that the holistic process adds useful information beyond that achieved by much less costly statistical approaches. Conversely, if the interrater reliability of difference scores is near zero, this would clearly demonstrate that committee judgments add random error to the decision process. RESULTS. Evidence to conduct such studies already exists within most highly selective medical schools and graduate programs and the proposed validity research could be conducted on existing data. CONCLUSIONS. Such research evidence is critical for establishing the validity of widely used holistic admission approaches.
入学决策需要综合运用整体（人类评判者）或统计（评估）方法来整合申请人的信息。为了优化特定的可衡量的结果，有大量的研究证据表明，与评判者生成的决策相比，统计方法可做出更优的决策。然而，整体决策的优势可能反映在未被衡量的结果中。如果确实存在这种优势，那么这些优势必然会表现为评分者的分数存在系统差异，与基于统计的决策不同。目的：为了估计这种差异，我们提出了一种设计，检验差异评分（即观察到的委员会排名与基于统计方法的排名之间的差异）的评分者间信度。方法：通过示例计算和 G 研究模型展示了如何在各种情况下分析差异评分的评分者间一致性。差异评分的高评分者间信度将支持但不能证明整体过程在统计方法之外提供了有用信息的说法。相反，如果差异评分的评分者间信度接近零，这将清楚地表明委员会判断为决策过程增加了随机误差。结果：在大多数高选择性医学院校和研究生项目中，已经存在进行此类研究的证据，并且可以在现有数据上开展拟议的有效性研究。结论：这种研究证据对于确立广泛使用的整体入学方法的有效性至关重要。
