Coetzee Karen, Monteiro Sandra, Amirthalingam Luxshi
Assessment Consultant and Psychometrician at Construct Measures, An Organization with a Focus on Assessment within Regulated Health Professions, Toronto, Ontario, Canada.
Department of Medicine at McMaster, University in Hamilton, Hamilton, Ontario, Canada.
J Eval Clin Pract. 2025 Feb;31(1):e14114. doi: 10.1111/jep.14114. Epub 2024 Jul 29.
Objective Structured Clinical Examinations (OSCEs) are widely used for assessing clinical competence, especially in high-stakes environments such as medical licensure. However, the reuse of OSCE cases across multiple administrations raises concerns about parameter stability, known as item parameter drift (IPD). AIMS & OBJECTIVES: This study aims to investigate IPD in reused OSCE cases while accounting for examiner scoring effects using a Many-facet Rasch Measurement (MFRM) model.
Data from 12 OSCE cases, reused over seven administrations of the Internationally Educated Nurse Competency Assessment Program (IENCAP), were analyzed using the MFRM model. Each case was treated as an item, and examiner scoring effects were accounted for in the analysis.
The results indicated that despite accounting for examiner effects, all cases exhibited some level of IPD, with an average absolute IPD of 0.21 logits. Three cases showed positive directional trends. IPD significantly affected score decisions in 1.19% of estimates, at an invariance violation of 0.58 logits.
These findings suggest that while OSCE cases demonstrate sufficient stability for reuse, continuous monitoring is essential to ensure the accuracy of score interpretations and decisions. The study provides an objective threshold for detecting concerning levels of IPD and underscores the importance of addressing examiner scoring effects in OSCE assessments. The MFRM model offers a robust framework for tracking and mitigating IPD, contributing to the validity and reliability of OSCEs in evaluating clinical competence.
客观结构化临床考试（OSCEs）被广泛用于评估临床能力，尤其是在诸如医学执照考试等高风险环境中。然而，OSCE案例在多次考试中的重复使用引发了对参数稳定性的担忧，即所谓的项目参数漂移（IPD）。
本研究旨在调查重复使用的OSCE案例中的IPD情况，同时使用多面Rasch测量（MFRM）模型考虑考官评分效应。
使用MFRM模型分析了国际教育护士能力评估项目（IENCAP）七次考试中重复使用的12个OSCE案例的数据。每个案例被视为一个项目，并在分析中考虑了考官评分效应。
结果表明，尽管考虑了考官效应，但所有案例都表现出一定程度的IPD，平均绝对IPD为0.21对数单位。三个案例呈现出正向趋势。在不变性违反0.58对数单位的情况下，IPD在1.19%的估计中显著影响分数决策。
这些发现表明，虽然OSCE案例在重复使用时表现出足够的稳定性，但持续监测对于确保分数解释和决策的准确性至关重要。该研究提供了一个检测IPD相关水平的客观阈值，并强调了在OSCE评估中解决考官评分效应的重要性。MFRM模型为跟踪和减轻IPD提供了一个强大的框架，有助于提高OSCE在评估临床能力方面的有效性和可靠性。
