Department of Medicine, University of Otago Wellington, Wellington, New Zealand.
University of Otago Christchurch, Christchurch, New Zealand.
Med Educ. 2021 Aug;55(8):951-960. doi: 10.1111/medu.14534. Epub 2021 Apr 21.
Assessment of healthcare professionals should be authentic to clinical practice. As clinicians regularly use resources in practice, similar resources should be available to those sitting assessment. There is limited information on the impacts of open-book (resource) assessments on standard setting for use in high-stakes assessments. This research aims to explore the effects on standard setting and student perceptions when open vs closed resources are available in high-stakes assessment of medical students.
Students sat multiple-choice question (MCQ) examinations under both closed- and open-resource conditions in a randomised crossover design. A standard setting panel set pass-marks for both closed- and open-resource conditions of delivery, and we compared these pass-marks with each other and with actual performance. Students responded to a survey on perceptions of open-resource assessments.
The pass-mark was set higher when panellists considered open-resource conditions compared to closed conditions (59% vs 47%), but actual student performance showed no difference in scores between the two conditions. The net effect was that the pass rate was higher for closed than open-resource conditions (71% vs 34%). Open-resource conditions increased the time to complete the questions. The students perceived that open resource was more authentic but was more time-consuming and would require different preparation. Regarding the acceptability of including open resources in high-stakes assessment, the responses of students were mixed.
Pass standards based solely on judgements by panellists experienced in closed-resource conditions might not be applicable under open conditions. Questions vary in how much time accessing resources may take and the degree of assistance in selecting the correct answer. A programme of assessment could be constructed to include both closed- and open-resource condition assessments depending on the questions' content and format. Open-resource conditions may promote assessment preparation that focuses more on seeking and evaluating resources rather than learning facts.
评估医疗保健专业人员的方法应与临床实践相符。由于临床医生在实践中经常使用资源，因此应向参加评估的人员提供类似的资源。关于在高风险评估中使用开卷（资源）评估对标准制定的影响，相关信息有限。本研究旨在探讨在高风险评估中为医学生提供开卷和闭卷资源时，对标准制定和学生感知的影响。
学生采用随机交叉设计，在闭卷和开卷资源条件下参加多项选择题（MCQ）考试。一个标准制定小组为闭卷和开卷资源条件设定了及格分数，我们将这些及格分数相互比较，并与实际表现进行比较。学生对开卷评估的看法进行了调查。
与闭卷条件相比，当小组成员考虑开卷条件时，及格分数设定得更高（59%比 47%），但学生在两种条件下的实际表现没有差异。净效应是闭卷条件的及格率高于开卷条件（71%比 34%）。开卷条件增加了完成问题的时间。学生认为开卷更真实，但更耗时，需要不同的准备。关于在高风险评估中包含开卷资源的可接受性，学生的反应喜忧参半。
仅基于闭卷资源条件下有经验的小组成员的判断设定的及格标准可能不适用于开卷条件。问题在获取资源所需的时间和选择正确答案的帮助程度上有所不同。根据问题的内容和格式，可以构建一个评估方案，包括闭卷和开卷资源条件的评估。开卷条件可能会促进更注重寻找和评估资源而不是学习事实的评估准备。
