J.J.H. Cheung is assistant professor, Department of Medical Education, University of Illinois College of Medicine at Chicago, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-6037-5811 .
Y.S. Park is associate professor, Harvard Medical School, and director of health professions education research, Massachusetts General Hospital, Boston, Massachusetts; ORCID: https://orcid.org/0000-0001-8583-4335 .
Acad Med. 2022 Nov 1;97(11S):S15-S21. doi: 10.1097/ACM.0000000000004908. Epub 2022 Aug 9.
Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback.
Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time).
Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores.
Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams.
用于评估学生临床推理能力的标准化患者（SP）后患者记录，给传统评分的教师带来了巨大的时间负担。为了减轻这种负担，作者之前报告了一种复杂的教师开发评分方法，用于评估由非临床医生评分的患者记录。本研究探讨了简化非临床评分程序是否可以通过减少时间、成本和为形成性反馈创造更多机会来进一步优化患者记录评估。
10 名非临床评分者通过识别案例特定的患者记录清单项目，对 5 个 SP 案例中的 141 名学生的患者记录进行评分。作者使用记录中正确识别的项目比例（百分制得分）和案例特定的教师生成的评分公式（公式得分）来确定学生的最低五分位数。5 名教师使用全球评分量表对来自表现较差、边缘和表现较好的学生的一小部分记录（n = 30 名学生）进行评分。作者进行了分析，以收集百分制得分的有效性证据（即与其他变量的关系），研究其可靠性（即可推广性研究），并评估其成本（即教师时间）。
非临床百分制和公式制得分高度相关（r =.88），并确定了类似的表现较差学生名单。这两种方法在通过/不通过的判定上彼此之间具有良好的一致性（Kappa =.68），与教师的总体评分也具有良好的一致性（Kappa 百分制得分 =.61；Kappa 公式得分 =.66）。百分制得分的 G 系数为.52，有 38%的变异性归因于嵌套在案例中的清单项目。与公式制得分相比，使用百分制得分可在开发成本方面为每个 SP 案例节省约 746 美元（包括 6 小时的教师时间）。
非临床百分制得分可在无需复杂的教师开发评分公式的情况下，可靠地识别表现较差的学生。结合非临床评分者的分析评分和教师的整体评分可以减少患者记录评分的时间和成本，让教师有更多时间指导有风险的学生，并为高风险的总结性考试提供有针对性的评估输入。
