School of Education and Faculty of Medicine, The University of Queensland, Brisbane, QLD, 4072, Australia.
School of Nursing and Midwifery, Queen's University Belfast, Belfast, BT9 7BL, UK.
Adv Health Sci Educ Theory Pract. 2023 Mar;28(1):27-46. doi: 10.1007/s10459-022-10139-1. Epub 2022 Aug 9.
Examiners' judgements play a critical role in competency-based assessments such as objective structured clinical examinations (OSCEs). The standardised nature of OSCEs and their alignment with regulatory accountability assure their wide use as high-stakes assessment in medical education. Research into examiner behaviours has predominantly explored the desirable psychometric characteristics of OSCEs, or investigated examiners' judgements from a cognitive rather than a sociocultural perspective. This study applies cultural historical activity theory (CHAT) to address this gap in exploring examiners' judgements in a high-stakes OSCE. Based on the idea that OSCE examiners' judgements are socially constructed and mediated by their clinical roles, the objective was to explore the sociocultural factors that influenced examiners' judgements of student competence and use the findings to inform examiner training to enhance assessment practice. Seventeen semi-structured interviews were conducted with examiners who assessed medical student competence in progressing to the next stage of training in a large-scale OSCE at one Australian university. The initial thematic analysis provided a basis for applying CHAT iteratively to explore the sociocultural factors and, specifically, the contradictions created by interactions between different elements such as examiners and rules, thus highlighting the factors influencing examiners' judgements. The findings indicated four key factors that influenced examiners' judgements: examiners' contrasting beliefs about the purpose of the OSCE; their varying perceptions of the marking criteria; divergent expectations of student competence; and idiosyncratic judgement practices. These factors were interrelated with the activity systems of the medical school's assessment practices and the examiners' clinical work contexts. Contradictions were identified through the guiding principles of multi-voicedness and historicity. The exploration of the sociocultural factors that may influence the consistency of examiners' judgements was facilitated by applying CHAT as an analytical framework. Reflecting upon these factors at organisational and system levels generated insights for creating fit-for-purpose examiner training to enhance assessment practice.
考官的判断在基于能力的评估中起着至关重要的作用,例如客观结构化临床考试(OSCE)。OSCE 的标准化性质及其与监管问责制的一致性确保了它们作为医学教育中高风险评估的广泛使用。对考官行为的研究主要探讨了 OSCE 的理想心理测量特征,或者从认知而不是社会文化的角度研究了考官的判断。本研究应用文化历史活动理论(CHAT)来解决这一差距,即在高风险 OSCE 中探索考官的判断。基于 OSCE 考官的判断是由他们的临床角色构建和调节的观点,本研究的目的是探讨影响考官对学生能力判断的社会文化因素,并利用这些发现为考官培训提供信息,以提高评估实践。在澳大利亚一所大学的大规模 OSCE 中,对评估医学生是否进入下一阶段培训的能力的考官进行了 17 次半结构化访谈。初始主题分析为应用 CHAT 提供了基础,以探讨社会文化因素,特别是不同元素(如考官和规则)之间相互作用产生的矛盾,从而突出影响考官判断的因素。研究结果表明,有四个关键因素影响了考官的判断:考官对 OSCE 目的的信念差异;他们对评分标准的不同看法;对学生能力的不同期望;以及独特的判断实践。这些因素与医学院评估实践的活动系统以及考官的临床工作环境相互关联。通过多声部和历史性的指导原则,确定了矛盾。通过应用 CHAT 作为分析框架,探讨了可能影响考官判断一致性的社会文化因素。在组织和系统层面上思考这些因素,为创造适合目的的考官培训提供了见解,以提高评估实践。