Everett Tobias C, Ng Elaine, Power Daniel, Marsh Christopher, Tolchard Stephen, Shadrina Anna, Bould Matthew D
Department of Anesthesiology and Pain Medicine, The Hospital for Sick Children, University of Toronto, Toronto, ON, Canada.
Paediatr Anaesth. 2013 Dec;23(12):1117-23. doi: 10.1111/pan.12212. Epub 2013 Jun 26.
The use of simulation-based assessments for high-stakes physician examinations remains controversial. The Managing Emergencies in Paediatric Anaesthesia course uses simulation to teach evidence-based management of anesthesia crises to trainee anesthetists in the United Kingdom (UK) and Canada. In this study, we investigated the feasibility and reliability of custom-designed scenario-specific performance checklists and a global rating scale (GRS) assessing readiness for independent practice.
After research ethics board approval, subjects were videoed managing simulated pediatric anesthesia crises in a single Canadian teaching hospital. Each subject was randomized to two of six different scenarios. All 60 scenarios were subsequently rated by four blinded raters (two in the UK, two in Canada) using the checklists and GRS. The actual and predicted reliability of the tools was calculated for different numbers of raters using the intraclass correlation coefficient (ICC) and the Spearman-Brown prophecy formula.
Average measures ICCs ranged from 'substantial' to 'near perfect' (P ≤ 0.001). The reliability of the checklists and the GRS was similar. Single measures ICCs showed more variability than average measures ICC. At least two raters would be required to achieve acceptable reliability.
We have established the reliability of a GRS to assess the management of simulated crisis scenarios in pediatric anesthesia, and this tool is feasible within the setting of a research study. The global rating scale allows raters to make a judgement regarding a participant's readiness for independent practice. These tools may be used in the future research examining simulation-based assessment.
将基于模拟的评估用于高风险的医师考试仍存在争议。小儿麻醉应急处理课程利用模拟向英国和加拿大的实习麻醉医师传授基于证据的麻醉危机管理方法。在本研究中，我们调查了定制的特定场景表现检查表和评估独立执业准备情况的整体评分量表（GRS）的可行性和可靠性。
经研究伦理委员会批准后，在加拿大一家教学医院对受试者处理模拟小儿麻醉危机的过程进行录像。每位受试者被随机分配到六个不同场景中的两个。随后，四名不知情的评分者（两名来自英国，两名来自加拿大）使用检查表和GRS对所有60个场景进行评分。使用组内相关系数（ICC）和斯皮尔曼-布朗预测公式计算不同评分者数量下工具的实际和预测可靠性。
平均测量ICC值范围从“高度一致”到“几乎完美”（P≤0.001）。检查表和GRS的可靠性相似。单次测量ICC值的变异性大于平均测量ICC值。至少需要两名评分者才能达到可接受的可靠性。
我们已确定GRS评估小儿麻醉模拟危机场景处理情况的可靠性，并且该工具在研究环境中是可行的。整体评分量表使评分者能够对参与者独立执业的准备情况做出判断。这些工具未来可能用于基于模拟的评估研究。
