B. Kinnear is associate professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-0052-4130 .
M. Kelleher is assistant professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio.
Acad Med. 2021 Jul 1;96(7S):S64-S69. doi: 10.1097/ACM.0000000000004112.
Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system.
To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time.
The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps.
Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.
在以能力为基础的医学教育模式中，许多医学教育项目都非常依赖基于工作场所的评估（WBA）来为学习者的总结性决策提供数据。然而，WBA 非常复杂，需要除心理测量分析之外的有效性证据。在这里，作者描述了他们如何使用修辞论证过程来为基于委托的 WBA 系统的总结性决策制定有效性证据图谱。
为了组织证据，作者交叉参考了 2 个当代有效性框架，一个强调证据来源（Messick），另一个强调论证中的推理（Kane）。他们使用 4 个步骤构建了一个有效性图谱：（1）针对既定的解释和使用提出批判性问题，（2）寻找作为回应的有效性证据，（3）使用 Messick 和 Kane 的框架对证据进行分类，以及（4）构建收集和组织证据的可视化表示。作者采用迭代方法，随着时间的推移不断添加新的批判性问题和证据。
最初的图谱草案产生了 25 个框的证据，其中包括 Messick 详细描述的 5 种证据来源，以及 Kane 描述的 4 种推理中的每一种。修辞性的问题-回应过程允许对 WBA 系统进行结构化的批判性评估，从而确定证据差距。
未来的图谱迭代将整合证据质量指标，并允许更深入地研究证据。作者打算与研究生医学教育利益相关者（如认证机构、机构领导、学习者、患者）分享他们的图谱，以了解它是否为评估他们的 WBA 项目的有效性论证增加了价值。
