Sebok-Syer Stefanie S, Klinger Don A, Sherbino Jonathan, Chan Teresa M
S.S. Sebok-Syer is instructor of education, Queen's University, Kingston, Ontario, Canada. D.A. Klinger is professor of education, Queen's University, Kingston, Ontario, Canada. J. Sherbino is associate professor of medicine, McMaster University, Hamilton, Ontario, Canada. T.M. Chan is assistant professor of medicine, McMaster University, Hamilton, Ontario, Canada; ORCID: http://orcid.org/0000-0001-6104-462.
Acad Med. 2017 Dec;92(12):1774-1779. doi: 10.1097/ACM.0000000000001743.
The shift toward broader, programmatic assessment has revolutionized the approaches that many take in assessing medical competence. To understand the association between quantitative and qualitative evaluations, the authors explored the relationships that exist among assessors' checklist scores, task ratings, global ratings, and written comments.
The authors collected and analyzed, using regression analyses, data from the McMaster Modular Assessment Program. The data were from emergency medicine residents in their first or second year of postgraduate training from 2012 through 2014. Additionally, using content analysis, the authors analyzed narrative comments corresponding to the "done" and "done, but needs attention" checklist score options.
The regression analyses revealed that the task ratings, provided by faculty assessors, are associated with the use of the "done, but needs attention" checklist score option. Analyses also identified that the "done, but needs attention" option is associated with a narrative comment that is balanced, providing both strengths and areas for improvement. Analysis of qualitative comments revealed differences in the type of comments provided to higher- and lower-performing residents.
This study highlights some of the relationships that exist among checklist scores, rating scales, and written comments. The findings highlight that task ratings are associated with checklist options while global ratings are not. Furthermore, analysis of written comments supports the notion of a "hidden code" used to communicate assessors' evaluation of medical competence, especially when communicating areas for improvement or concern. This study has implications for how individuals should interpret information obtained from qualitative assessments.
向更广泛的、基于项目的评估转变已经彻底改变了许多人评估医学能力的方法。为了理解定量和定性评估之间的关联，作者探讨了评估者的检查表评分、任务评级、整体评级和书面评语之间存在的关系。
作者使用回归分析收集并分析了来自麦克马斯特模块化评估项目的数据。这些数据来自2012年至2014年接受研究生培训第一年或第二年的急诊医学住院医师。此外，作者使用内容分析方法，分析了与检查表评分选项“完成”和“完成但需关注”相对应的叙述性评语。
回归分析显示，教员评估者给出的任务评级与检查表评分选项“完成但需关注”的使用有关。分析还发现，“完成但需关注”选项与一条平衡的叙述性评语相关，该评语既指出了优点也提出了改进的方面。对定性评语的分析揭示了针对表现较好和较差的住院医师所提供的评语类型存在差异。
本研究突出了检查表评分、评级量表和书面评语之间存在的一些关系。研究结果表明，任务评级与检查表选项相关，而整体评级则不然。此外，对书面评语的分析支持了存在一种“隐藏代码”的观点，这种代码用于传达评估者对医学能力的评估，尤其是在传达改进或关注的方面时。本研究对个人应如何解读从定性评估中获得的信息具有启示意义。
