直接观察在委托决策中的关键作用。

The critical role of direct observation in entrustment decisions.

作者信息

Sibbald Matthew, Mansoor Muqtasid, Tsang Michael, Blissett Sarah, Norman Geoffrey

机构信息

McMaster Faculty of Health Sciences Education Research, Innovation and Program (MERIT), McMaster University, Ontario, Canada.

McMaster University, Ontario, Canada.

出版信息

Can Med Educ J. 2021 Nov 1;12(5):18-23. doi: 10.36834/cmej.72040. eCollection 2021 Nov.

DOI:10.36834/cmej.72040

PMID:34804284

原文链接:https://pmc.ncbi.nlm.nih.gov/articles/PMC8603883/

Abstract

BACKGROUND

Entrustment decisions may be retrospective (based on past experiences with a trainee) or real-time (based on direct observation). We investigated judgments of entrustment based on assessor prior knowledge of candidates and based on systematic direct observation, conducted in an objective structured clinical exam (OSCE).

METHODS

Sixteen faculty examiners provided 287 retrospective and real-time entrustment ratings of 16 cardiology trainees during OSCE stations in 2019 and 2020. Reliability and validity of these ratings were assessed by comparing correlations across stations as a measure of reliability, differences across postgraduate years as an index of construct validity, correlation to standardized in-training exam (ITE) as a measure of criterion validity, and reclassification of entrustment as a measure of consequential validity.

RESULTS

Both retrospective and real-time assessments were highly reliable (all intra-class correlations >0.86). Both increased with a year of postgraduate training. Real-time entrustment ratings were significantly correlated with standardized ITE scores; retrospective ratings were not. Real-time ratings explained 37% (2019) and 46% (2020) of variance in examination scores vs. 21% (2019) and 7% (2020) for retrospective ratings. Direct observation resulted in a different level of entrustment compared with retrospective ratings in 44% of cases ( = <0.001).

CONCLUSIONS

Ratings based on direct observation made unique contributions to entrustment decisions.

摘要

背景

委托决策可以是回顾性的（基于对学员过去经历的考量）或实时的（基于直接观察）。我们在客观结构化临床考试（OSCE）中，研究了基于评估者对考生的先验知识以及基于系统直接观察做出的委托判断。

方法

16名教员考官在2019年和2020年的OSCE站点中，对16名心脏病学实习医生提供了287项回顾性和实时委托评级。通过比较各站点间的相关性作为可靠性指标、比较研究生年级间的差异作为结构效度指标、与标准化在职培训考试（ITE）的相关性作为效标效度指标以及将委托重新分类作为结果效度指标，来评估这些评级的可靠性和有效性。

结果

回顾性和实时评估都具有高度可靠性（所有组内相关性>0.86）。两者都随着研究生培训年限的增加而提高。实时委托评级与标准化ITE分数显著相关；回顾性评级则不然。实时评级解释了考试分数方差的37%（2019年）和46%（2020年），而回顾性评级分别为21%（2019年）和7%（2020年）。在44%的病例中，直接观察导致的委托水平与回顾性评级不同（P<0.001）。