Kim John, Neilipovitz David, Cardinal Pierre, Chiu Michelle
Division of Critical Care Medicine and Department of Medicine (J.M.), University of Ottawa/The Ottawa Hospital, Gloucester, ON, Canada.
Simul Healthc. 2009 Spring;4(1):6-16. doi: 10.1097/SIH.0b013e3181880472.
Crisis resource management (CRM) skills are a set of nonmedical skills required to manage medical emergencies. There is currently no gold standard for evaluation of CRM performance. A prior study examined the use of a global rating scale (GRS) to evaluate CRM performance. This current study compared the use of a GRS and a checklist as formal rating instruments to evaluate CRM performance during simulated emergencies.
First-year and third-year residents participated in two simulator scenarios each. Three raters then evaluated resident performance in CRM using edited video recordings using both a GRS and a checklist. The Ottawa GRS provides a seven-point anchored ordinal scale for performance in five categories of CRM, and an overall performance score. The Ottawa CRM checklist provides 12 items in the five categories of CRM, with a maximum cumulative score of 30 points. Construct validity was measured on the basis of content validity, response process, internal structure, and response to other variables. T-test analysis of Ottawa GRS scores was conducted to examine response to the variable of level of training. Intraclass correlation coefficient (ICC) scores were used to measure inter-rater reliability for both scenarios.
Thirty-two first-year and 28 third-year residents participated in the study. Third-year residents produced higher mean scores for overall CRM performance than first-year residents (P < 0.05), and in all individual categories within the Ottawa GRS (P < 0.05) and the Ottawa CRM checklist (P < 0.05). This difference was noted for both scenarios and for each individual rater (P < 0.05). No statistically significant difference in resident scores was observed between scenarios for both instruments. ICC scores of 0.59 and 0.61 were obtained for Scenarios 1 and 2 with the Ottawa GRS, whereas ICC scores of 0.63 and 0.55 were obtained with the Ottawa CRM checklist. Users indicated a strong preference for the Ottawa GRS given ease of scoring, presence of an overall score, and the potential for formative evaluation.
Construct validity seems to be present when using both the Ottawa GRS and CRM checklist to evaluate CRM performance during simulated emergencies. Data also indicate the presence of moderate inter-rater reliability when using both the Ottawa GRS and CRM checklist.
危机资源管理（CRM）技能是应对医疗紧急情况所需的一系列非医学技能。目前尚无评估CRM表现的金标准。此前一项研究探讨了使用整体评分量表（GRS）来评估CRM表现。本研究比较了使用GRS和检查表作为正式评分工具在模拟紧急情况下评估CRM表现的情况。
一年级和三年级住院医师各参与两个模拟场景。然后，三名评分者使用编辑后的录像，通过GRS和检查表对住院医师在CRM中的表现进行评估。渥太华GRS为CRM的五个类别表现提供了一个七点锚定顺序量表以及一个整体表现分数。渥太华CRM检查表在CRM的五个类别中有12项内容，最高累计分数为30分。结构效度基于内容效度、反应过程、内部结构以及对其他变量的反应来衡量。对渥太华GRS分数进行t检验分析，以检验对培训水平变量的反应。组内相关系数（ICC）分数用于衡量两种场景下评分者间的可靠性。
32名一年级和28名三年级住院医师参与了研究。三年级住院医师在CRM整体表现上的平均得分高于一年级住院医师（P < 0.05），在渥太华GRS的所有单个类别（P < 0.05）以及渥太华CRM检查表（P < 0.05）中也是如此。两种场景以及每位评分者都出现了这种差异（P < 0.05）。两种工具在不同场景下住院医师得分均未观察到统计学上的显著差异。使用渥太华GRS时，场景1和场景2的ICC分数分别为0.59和0.61，而使用渥太华CRM检查表时ICC分数分别为0.63和0.55。鉴于评分简便、存在整体分数以及形成性评估的潜力，使用者表示强烈倾向于渥太华GRS。
在模拟紧急情况下使用渥太华GRS和CRM检查表评估CRM表现时，似乎存在结构效度。数据还表明，使用渥太华GRS和CRM检查表时存在中等程度的评分者间可靠性。
