Hatala Rose, Sawatsky Adam P, Dudek Nancy, Ginsburg Shiphra, Cook David A
R. Hatala is associate professor of medicine, Faculty of Medicine, and director, Clinical Educator Fellowship, Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada. A.P. Sawatsky is assistant professor of medicine and senior associate consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota. N. Dudek is associate professor, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. S. Ginsburg is professor, Department of Medicine, Faculty of Medicine, University of Toronto, scientist, Wilson Centre for Research in Education, University Health Network/University of Toronto, and staff physician, Mount Sinai Hospital, Toronto, Ontario, Canada. D.A. Cook is professor of medicine and medical education, associate director, Mayo Clinic Online Learning, and consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota.
Acad Med. 2017 Jun;92(6):868-879. doi: 10.1097/ACM.0000000000001506.
In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for assessing medical students and residents and to identify evidence gaps.
Reviewers searched for relevant English-language studies in MEDLINE, EMBASE, Scopus, and ERIC (last search June 5, 2015), and in reference lists and author files. They included all original studies that evaluated ITERs for qualitative assessment of medical students and residents. Working in duplicate, they selected articles for inclusion, evaluated quality, and abstracted information on validity evidence using Kane's framework (inferences of scoring, generalization, extrapolation, and implications).
Of 777 potential articles, 22 met inclusion criteria. The scoring inference is supported by studies showing that rich narratives are possible, that changing the prompt can stimulate more robust narratives, and that comments vary by context. Generalization is supported by studies showing that narratives reach thematic saturation and that analysts make consistent judgments. Extrapolation is supported by favorable relationships between ITER narratives and numeric scores from ITERs and non-ITER performance measures, and by studies confirming that narratives reflect constructs deemed important in clinical work. Evidence supporting implications is scant.
The use of ITER narratives for trainee assessment is generally supported, except that evidence is lacking for implications and decisions. Future research should seek to confirm implicit assumptions and evaluate the impact of decisions.
培训期间评估报告（ITERs）是医学生和研究生医师培训学员（住院医师）评估的一个组成部分。ITERs的叙述性评语受到的关注少于数字评分。作者旨在确定哪些效度证据为使用ITERs的叙述性评语评估医学生和住院医师提供依据，并找出证据差距。
评审人员在MEDLINE、EMBASE、Scopus和ERIC（最后一次检索时间为2015年6月5日）以及参考文献列表和作者文件中搜索相关的英文研究。他们纳入了所有评估ITERs用于医学生和住院医师定性评估的原始研究。他们两人一组工作，选择纳入的文章，评估质量，并使用凯恩框架（评分推断、概括、外推和影响）提取关于效度证据的信息。
在777篇潜在文章中，22篇符合纳入标准。评分推断得到了一些研究的支持，这些研究表明丰富的叙述是可能的，改变提示可以激发更有力的叙述，并且评语因背景而异。概括得到了一些研究的支持，这些研究表明叙述达到了主题饱和，并且分析人员做出了一致的判断。外推得到了ITER叙述与ITER数字评分和非ITER绩效指标之间良好关系的支持，以及证实叙述反映了临床工作中被认为重要的结构的研究的支持。支持影响的证据很少。
一般支持使用ITER叙述进行学员评估，但缺乏关于影响和决策的证据。未来的研究应寻求确认隐含假设并评估决策的影响。
