Division of Emergency Medicine, McMaster University, Hamilton, Ontario, Canada.
Department of Emergency Medicine, Stanford University, Stanford, California, USA.
Teach Learn Med. 2020 Jun-Jul;32(3):319-329. doi: 10.1080/10401334.2019.1708365. Epub 2020 Feb 4.
This study seeks to determine validity evidence for the Quality of Assessment for Learning score (QuAL score), which was created to evaluate short qualitative comments that are related to specific scores entered into a workplace-based assessment, common within the competency-based medical education (CBME) context. In the age of CBME, qualitative comments play an important role in clarifying the quantitative scores rendered by observers at the bedside. Currently there are few practical tools that evaluate mixed data (e.g. associated score-and-comment data), other than the comprehensive Completed Clinical Evaluation Report Rating tool (CCERR) that was originally derived to rate end-of-rotation reports. A multi-center, randomized cohort-based rating exercise was conducted to evaluate the rating properties of the QuAL score as compared to the CCERR. One group rated comments using the QuAL score, and the other group rated comments using the CCERR. A generalizability study (G-Study) and a decision study (D-study) were conducted to determine the number of meta-raters for a reliable rating (phi-coefficient target of >0.80). Both scores were correlated against rater's gestalt perceptions of utility for both faculty and residents reading the scores. Twenty-five meta-raters from 20 sites participated in this rating exercise. The G-study revealed that the CCERR group (n = 13) rated the comments with a very high reliability (Phi = 0.97). Meanwhile, the QuAL group (n = 12) rated the comments with a similarly high reliability (Phi = 0.97). The QuAL score required only two raters to reach an acceptable target reliability of >0.80, while the CCERR required three. The QuAL score correlated with perceptions of utility (Meta-rater usefulness, Pearson's r = 0.69, p < 0.001; Perceived usefulness for trainee, r = 0.74, p < 0.001). The CCERR performed similarly, correlating with perceived faculty (r = 0.67, <0.001) and resident utility (0.79, <0.001). The QuAL score is reliable rating score that correlates well with perceptions of utility. The QuAL score may be useful for rating shorter comments generated by workplace-based assessments.
本研究旨在确定评估学习质量评分（QuAL 评分）的效度证据，该评分旨在评估与工作场所评估中输入的特定分数相关的简短定性评论，这在基于能力的医学教育（CBME）背景下很常见。在 CBME 时代，定性评论在澄清观察者在床边给出的定量分数方面发挥着重要作用。目前，除了最初用于评估轮次结束报告的综合完成临床评估报告评分工具（CCERR）之外，很少有实用工具可以评估混合数据（例如，相关评分和评论数据）。进行了一项多中心、随机队列的评分练习，以评估 QuAL 评分与 CCERR 的评分特性。一组使用 QuAL 评分对评论进行评分，另一组使用 CCERR 对评论进行评分。进行了一项可推广性研究（G 研究）和一项决策研究（D 研究），以确定可靠评分所需的元评分者数量（phi 系数目标值>0.80）。两个评分都与评分者对评分对教师和住院医师阅读评分的效用的整体感知相关。来自 20 个地点的 25 名元评分者参加了此次评分练习。G 研究表明，CCERR 组（n=13）对评论的评分具有很高的可靠性（Phi=0.97）。与此同时，QuAL 组（n=12）对评论的评分也具有相似的高可靠性（Phi=0.97）。QuAL 评分仅需两名评分者即可达到可接受的>0.80 的目标可靠性，而 CCERR 则需要三名评分者。QuAL 评分与效用感知相关（元评分者有用性，Pearson's r=0.69，p<0.001；对学员的有用性，r=0.74，p<0.001）。CCERR 的表现类似，与感知教师（r=0.67，<0.001）和住院医师效用（0.79，<0.001）相关。QuAL 评分是一种可靠的评分，与效用感知密切相关。QuAL 评分可能有助于对工作场所评估产生的较短评论进行评分。
