标准设定变得简单：验证临床考试设定临界分数的等 Z 分数（EZ）法。

Standard setting made easy: validating the Equal Z-score (EZ) method for setting cut-score for clinical examinations.

机构信息

University of New South Wales, Sydney, Australia.

National Yang-Ming University, Taipei, Taiwan.

出版信息

BMC Med Educ. 2020 May 25;20(1):167. doi: 10.1186/s12909-020-02080-x.

DOI:10.1186/s12909-020-02080-x

PMID:32450878

原文链接:https://pmc.ncbi.nlm.nih.gov/articles/PMC7249316/

Abstract

BACKGROUND

This study aims to assess the feasibility, reliability and validity of the panel-based Equal Z-score (EZ) method applied to objective structural clinical examination (OSCE) of Chinese medical students and undertaking a comparison with the statistical techniques-based Borderline Regression Method (BRM).

METHODS

Data received from two cohorts of 6th and 7th year medical students in Taiwan who set the mock OSCE as a formative assessment. Traditionally this medical school uses BRM to set the pass/fail cut-score. For the current study, 31 OSCE panellists volunteered to participate in the EZ method in parallel to the BRM.

RESULTS

In the conduct of this study, each panel completed this task for an OSCE exam comprising 12 stations within less than 60 min. Moreover, none of the 31 panellists, whose are busy clinicians, had indicated that the task was too difficult or too time-consuming. Although EZ method yielded higher cut-scores than the BRM it was found reliable. Intraclass correlation (ICC) measuring absolute agreement, across the three groups of panellists was .893 and .937 for the first and second rounds respectively, demonstrating high level of agreement across groups with the EZ method and the alignment between the BRM and the EZ method was visually observed. The paired t-test results identified smaller differences between the cut-scores within methods than across methods.

CONCLUSIONS

Overall this study suggests that the EZ method is a feasible, reliable and valid standard setting method. The EZ method requires relatively little resources (takes about an hour to assess a 12 station OSCE); the calculation of the cut-score is simple and requires basic statistical skills; it is highly reliable even when only 10 panellists participate in the process; and its validity is supported by comparison to BRM. This study suggests that the EZ method is a feasible, reliable and valid standard setting method.

摘要

背景

本研究旨在评估基于面板的等效 Z 分数（EZ）方法应用于中国医学生客观结构临床考试（OSCE）的可行性、可靠性和有效性，并与基于边界回归方法（BRM）的统计技术进行比较。

方法

本研究数据来自台湾的 6 年级和 7 年级医学生的两个队列，他们将模拟 OSCE 作为形成性评估。传统上，这所医学院使用 BRM 来设置及格/不及格的临界分数。在当前的研究中，31 名 OSCE 考官自愿参与 EZ 方法，同时进行 BRM。

结果

在进行这项研究时，每个小组在不到 60 分钟的时间内完成了一项包含 12 个站点的 OSCE 考试。此外，没有一位忙碌的临床医生表示任务太难或太耗时。虽然 EZ 方法得出的临界分数高于 BRM，但它被发现是可靠的。三位考官小组的 ICC 测量绝对一致性，第一轮和第二轮分别为.893 和.937，表明 EZ 方法在小组之间具有高度的一致性，并且 BRM 和 EZ 方法之间的一致性是通过视觉观察得到的。配对 t 检验结果表明，方法内的临界分数差异小于方法间的差异。

结论

总的来说，这项研究表明 EZ 方法是一种可行、可靠和有效的标准设定方法。EZ 方法需要相对较少的资源（评估 12 个站点的 OSCE 需要大约一个小时）；临界分数的计算简单，只需要基本的统计技能；即使只有 10 名考官参与这个过程，它也是高度可靠的；并且与 BRM 的比较支持其有效性。这项研究表明，EZ 方法是一种可行、可靠和有效的标准设定方法。