Department of Psychiatry, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
Acad Med. 2012 Oct;87(10):1335-40. doi: 10.1097/ACM.0b013e3182674629.
Although multiple independent sampling (MIS) has been adapted for admissions interviews, its application for assessing written materials in the admissions file has been limited. Currently, admissions file review at the University of Toronto medical school involves one rater per file to enable holistic assessment, which may introduce a halo effect-that is, impressions of one component influencing the evaluation of other components. The authors examined whether MIS file review, through which multiple raters evaluate specific file components independently, may reduce this effect.
The authors selected a stratified random sample of 300 applicant files from the 2010-2011 admissions cycle for rescoring by MIS. They divided each of the 300 applicant files into their four components (academic transcript, autobiographical sketch, personal statement, reference letters) and rebundled them into packages of 38 same-component items (purposely creating some overlap among packages to assess inter-rater reliability). The authors distributed each package to 1 of 36 raters; thus, each rater evaluated only one of four components across many applicants. The authors compared the inter-component reliability and factor analysis of MIS with that of holistic scoring.
Ratings were returned for all applicants. Inter-component reliability (Cronbach alpha) was 0.69 for holistic scoring and 0.29 for MIS. Factor analysis showed all components loading heavily onto one factor in the holistic approach and onto three factors in the MIS method.
Using MIS to assess the admissions file may reduce the halo effect and should be considered when evaluating applicants' written submissions.
尽管多独立样本（MIS）已被应用于招生面试，但它在评估招生档案中的书面材料方面的应用却受到限制。目前，多伦多大学医学院的招生档案评审涉及每份档案由一名评审员进行整体评估，这可能会引入晕轮效应，即对一个组成部分的印象会影响对其他组成部分的评估。作者研究了多独立样本评审是否可以通过多个评审员独立评估特定档案组成部分来减少这种影响。
作者从 2010-2011 年招生周期中选择了 300 份申请人档案的分层随机样本进行多独立样本重评。他们将 300 份申请人档案中的每一份分为四个组成部分（学术成绩单、自传概要、个人陈述、推荐信），并将它们重新捆绑成 38 个相同组成部分的项目包（故意在包之间创建一些重叠，以评估评审员之间的可靠性）。作者将每个包分发给 36 名评审员中的 1 名；因此，每个评审员仅在许多申请人中评估四个组成部分中的一个。作者比较了多独立样本与整体评分的跨组成部分可靠性和因子分析。
所有申请人的评分均已返回。整体评分的跨组成部分可靠性（Cronbach alpha）为 0.69，多独立样本评分的跨组成部分可靠性为 0.29。因子分析表明，整体方法中的所有组成部分都强烈地加载到一个因素上，而多独立样本方法中的所有组成部分都加载到三个因素上。
使用多独立样本评估招生档案可能会减少晕轮效应，在评估申请人的书面材料时应予以考虑。
