Selim Omar, Dueck Andrew, Walsh Catharine M, Brydges Ryan, Okrainec Allan
Temerty-Chang Telesimulation Centre, University of Toronto, Toronto, Ontario, Canada; Division of Vascular Surgery, University Health Network, Toronto, Ontario, Canada.
Division of Vascular Surgery, Schulich Heart Centre, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada; Department of Surgery, University of Toronto, Toronto, Ontario, Canada.
J Vasc Surg. 2021 Feb;73(2):689-697. doi: 10.1016/j.jvs.2020.07.066. Epub 2020 Jul 22.
Diabetic foot wounds account for up to one-third of diabetes-related health care expenditure and are the greatest cause of extremity amputation in Canada. Physicians encounter patients with such wounds in all specialties, particularly as generalists in medical wards and emergency departments. However, there is a dearth of literature on the optimal way to teach and to assess the management of these patients. Given the importance of assessment for learning in the shift toward competency-based medical education, we aimed to develop an assessment tool and to build validity evidence for its use in this context.
A consensus process involving nine Canadian experts in diabetic wound management was used to develop the Diabetic Wound Assessment Learning Tool (DiWALT) items and two 10-minute simulation-based testing scenarios. The simulators used were modified from commercially available models to serve the testing scenarios. Validity evidence for the DiWALT was subsequently evaluated by assessing 24 physician participants' performance during the two scenarios. All participants were novices (<50 cases managed). Two assessors independently rated participants using the DiWALT. Evidence was organized using Kane's validity framework and included Cronbach α for interitem consistency as well as test-retest and inter-rater reliability using the intra-class correlation coefficient (ICC).
Cronbach α was 0.92, implying high internal consistency. Test-retest reliability was also excellent with ICC of 0.89 (confidence interval [CI], 0.76-0.95) for single measures and ICC of 0.94 (CI, 0.86-0.98) for average measures. Inter-rater reliability was fair for single measures with ICC of 0.68 (CI, 0.65-0.71) and good for average measures with ICC of 0.81 (CI, 0.79-0.83).
These results demonstrate that the DiWALT consistently and reliably evaluates competence in diabetic wound management during simulated cases using a small, homogeneous sample of physicians. Further work is necessary to quantify sources of error in the assessment scores, to establish validity evidence when it is used to assess larger and more heterogeneous participants, and to identify how well the DiWALT differentiates between different experience levels.
糖尿病足伤口占糖尿病相关医疗保健支出的三分之一，是加拿大肢体截肢的最大原因。所有专科的医生都会遇到患有此类伤口的患者，尤其是在医疗病房和急诊科的全科医生。然而，关于教授和评估这些患者管理的最佳方法的文献却很匮乏。鉴于在向基于能力的医学教育转变过程中评估对学习的重要性，我们旨在开发一种评估工具，并为其在此背景下的使用建立效度证据。
采用由九位加拿大糖尿病伤口管理专家参与的共识过程，来制定糖尿病伤口评估学习工具（DiWALT）项目以及两个基于模拟的10分钟测试场景。所使用的模拟器是从市售模型修改而来，以服务于测试场景。随后，通过评估24名医生参与者在这两个场景中的表现，来评估DiWALT的效度证据。所有参与者均为新手（处理病例数<50例）。两名评估者使用DiWALT对参与者进行独立评分。证据按照凯恩的效度框架进行整理，包括用于项目间一致性的克朗巴哈α系数，以及使用组内相关系数（ICC）的重测信度和评分者间信度。
克朗巴哈α系数为0.92，表明内部一致性较高。重测信度也非常好，单次测量的ICC为0.89（置信区间[CI]，0.76 - 0.95），平均测量的ICC为0.94（CI，0.86 - 0.98）。评分者间信度对于单次测量而言一般，ICC为0.68（CI，0.65 - 0.71），对于平均测量而言良好，ICC为0.81（CI，0.79 - 0.83）。
这些结果表明，DiWALT使用一小群同质化的医生样本，在模拟病例中能够一致且可靠地评估糖尿病伤口管理能力。有必要进一步开展工作，以量化评估分数中的误差来源，在用于评估更大规模和更多样化的参与者时建立效度证据，并确定DiWALT区分不同经验水平的能力有多强。
