The Wilson Centre, Department of Medicine, University of Toronto/University Health Network, 200 Elizabeth Street, 1ES-565, Toronto, ON, M5G 2C4, Canada.
Post-MD Education (Post-Graduate Medical Education/Continued Professional Development), University of Toronto, Toronto, ON, Canada.
Adv Health Sci Educ Theory Pract. 2018 May;23(2):323-338. doi: 10.1007/s10459-017-9800-3. Epub 2017 Oct 27.
Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane's validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating "Scoring" included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating "Generalizability" included: (a) a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating "Extrapolation" included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating "Implications" included low error rates in practice. Interpreting our findings according to Kane's framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane's validity framework to evaluate, and iteratively develop and refine assessment strategies.
临床能力评估较为复杂，需要进行推理。可靠且具有可辩护性的评估过程必须具有良好的有效性证据，尤其是在决策被认为具有高风险时。我们旨在使用 Kane 的有效性框架来组织、收集和解释高风险模拟评估策略的有效性证据，该框架据一些人报告难以实施。我们描述了使用该框架的经验，确定了挑战、决策点、解释和经验教训。我们考虑了与评估过程中的四个推理（评分、推广、推断、影响）相关的数据，并将有效性视为我们必须评估的一系列假设，从而产生了几个假设和拟议的分析。然后，我们根据四个推理来解释我们的发现，判断证据是否支持或反驳我们对评估数据的拟议使用。评估“评分”的相关数据包括：（a）具有可接受的项目间相关性的理想工具特征；（b）具有强项目-总分相关性；（c）项目和评分者的误差方差较低；（d）具有强评分者间可靠性。评估“可推广性”的相关数据包括：（a）一种强大的抽样策略，可涵盖大部分相关医疗指令、技能和国家能力，具有良好的整体和站间可靠性。评估“推断”的相关数据包括：维度评估得分与实践中的临床错误之间的相关性较低。评估“影响”的相关数据包括：实践中的错误率较低。根据 Kane 的框架来解释我们的发现，我们建议评分、推广和影响的证据支持我们将基于模拟的急救员评估策略作为认证考试使用；然而，推断证据较弱，表明考试成绩并不能预测临床错误率。我们的分析为其他使用 Kane 的有效性框架评估和迭代开发和完善评估策略的人提供了一个可遵循的实例。
