Acad Med. 2024 Sep 1;99(9):976-980. doi: 10.1097/ACM.0000000000005769. Epub 2024 May 15.
Many non-workplace-based assessments do not provide good evidence of a learner's problem representation or ability to provide a rationale for a clinical decision they have made. Exceptions include assessment formats that require resource-intensive administration and scoring. This article reports on research efforts toward building a scalable non-workplace-based assessment format that was specifically developed to capture evidence of a learner's ability to justify a clinical decision.
The authors developed a 2-step item format called SHARP (SHort Answer, Rationale Provision), referring to the 2 tasks that comprise the item. In collaboration with physician-educators, the authors integrated short-answer questions into a patient medical record-based item starting in October 2021 and arrived at an innovative item format in December 2021. In this format, a test-taker interprets patient medical record data to make a clinical decision, types in their response, and pinpoints medical record details that justify their answers. In January 2022, a total of 177 fourth-year medical students, representing 20 U.S. medical schools, completed 35 SHARP items in a proof-of-concept study.
Primary outcomes were item timing, difficulty, reliability, and scoring ease. There was substantial variability in item difficulty, with the average item answered correctly by 44% of students (range, 4%-76%). The estimated reliability (Cronbach α ) of the set of SHARP items was 0.76 (95% confidence interval, 0.70-0.80). Item scoring is fully automated, minimizing resource requirements.
A larger study is planned to gather additional validity evidence about the item format. This study will allow comparisons between performance on SHARP items and other examinations, examination of group differences in performance, and possible use cases for formative assessment. Cognitive interviews are also planned to better understand the thought processes of medical students as they work through the SHARP items.
许多非工作场所评估并不能很好地证明学习者的问题表示或为他们所做的临床决策提供理由的能力。例外情况包括需要资源密集型管理和评分的评估格式。本文报告了构建可扩展的非工作场所评估格式的研究工作，该格式专门用于捕捉学习者证明临床决策合理性的能力的证据。
作者开发了一种称为 SHARP（Short Answer，Rationale Provision）的 2 步项目格式，指的是组成项目的 2 项任务。作者与医师教育工作者合作，从 2021 年 10 月开始将简答题整合到基于患者病历的项目中，并于 2021 年 12 月提出了一种创新的项目格式。在这种格式中，应试者解释患者的病历数据以做出临床决策，输入他们的答案，并指出支持他们答案的病历细节。2022 年 1 月，共有来自 20 所美国医学院的 177 名四年级医学生在概念验证研究中完成了 35 项 SHARP 项目。
主要结果是项目时间、难度、可靠性和评分的便利性。项目难度差异很大，平均每个项目有 44%的学生答对（范围为 4%-76%）。整套 SHARP 项目的估计可靠性（Cronbach α）为 0.76（95%置信区间，0.70-0.80）。项目评分完全自动化，最大限度地减少资源需求。
计划进行一项更大的研究，以收集有关项目格式的更多有效性证据。这项研究将允许比较 SHARP 项目和其他考试的表现，考察表现的群体差异，以及形成性评估的可能用例。还计划进行认知访谈，以更好地了解医学生在处理 SHARP 项目时的思维过程。
