Florez Amy R, Shepard Lindsay N, Frey Mary E, Justice Lindsey B, Constand Sara E, Gilbert Gregory E, Kessler David O, Kerrey Benjamin T, Calhoun Aaron W
From the Division of Cardiology (A.R.F, L.B.J) and Emergency Medicine (M.E.F), Cincinnati Children's Hospital Medical Center; Division of Emergency Medicine (B.T.K), University of Cincinnati and Cincinnati Children's Hospital Medical Center, Cincinnati, OH; Division of Pediatrics (L.N.S), Columbia University Medical Center; Division of Emergency Medicine (D.O.K), Columbia University and Columbia University Medical Center, New York, NY; University of Cincinnati School of Medicine (S.E.C), Cincinnati, OH; ∑igma∑tats Consulting, LLC (G.E.G), Charleston, SC; ICON plc (G.E.G), Biostatistics and Medical Writing, Real World Evidence Strategy & Analytics, ICON Commercialization & Outcomes Services, North Wales, PA; and Division of Pediatric Critical Care (A.W.C.), University of Louisville School of Medicine and Norton Children's Hospital, Louisville, KY.
Simul Healthc. 2023 Feb 1;18(1):24-31. doi: 10.1097/SIH.0000000000000669. Epub 2022 May 5.
Resuscitation events in pediatric critical and emergency care are high risk, and strong leadership is an important component of an effective response. The Concise Assessment of Leadership Management (CALM) tool, designed to assess the strength of leadership skills during pediatric crises, has shown promising validity and reliability in simulated settings. The objective of this study was to generate further validity and reliability evidence for the CALM by applying it to real-life emergency events.
A prospective, video-based study was conducted in an academic pediatric emergency department. Three reviewers independently applied the CALM tool to the assessment of pediatric emergency department physicians as they led both a cardiac arrest and a sepsis event. Time to critical event (epinephrine, fluid, and antibiotic administration) was collected via video review. Based on Kane's framework, we conducted fully crossed, person × event × rater generalizability (G) and decision (D) studies. Interrater reliability was calculated using Gwet AC 2 and intraclass correlation coefficients. Time to critical events was correlated with CALM scores using Spearman coefficient.
Nine team leaders were assessed in their leadership of 2 resuscitations each. The G coefficient was 0.68, with 26% subject variance, 20% rater variance, and no case variance. Thirty-three percent of the variance (33%) was attributed to third-order interactions and unknown factors. Gwet AC 2 was 0.3 and intraclass correlation was 0.58. The CALM score and time to epinephrine correlated at -0.79 ( P = 0.01). The CALM score and time to fluid administration correlated at -0.181 ( P = 0.64).
This study provides additional validity evidence for the CALM tool's use in this context if used with multiple raters, aligning with data from the previous simulation-based CALM validity study. Further development may improve reliability. It also serves as an exemplar of the rigors of conducting validity work within medical simulation.
儿科重症和急救护理中的复苏事件风险很高，强有力的领导是有效应对的重要组成部分。简明领导力管理评估（CALM）工具旨在评估儿科危机期间领导技能的强度，在模拟环境中已显示出有前景的有效性和可靠性。本研究的目的是通过将CALM应用于实际紧急事件来生成更多关于其有效性和可靠性的证据。
在一家学术性儿科急诊科进行了一项基于视频的前瞻性研究。三名评审员在儿科急诊科医生领导心脏骤停和脓毒症事件时，独立应用CALM工具对他们进行评估。通过视频回顾收集关键事件（肾上腺素、液体和抗生素给药）的时间。基于凯恩的框架，我们进行了完全交叉的人×事件×评分者泛化性（G）和决策（D）研究。使用Gwet AC 2和组内相关系数计算评分者间信度。使用斯皮尔曼系数将关键事件时间与CALM评分相关联。
对九名团队领导者在其各自领导的两次复苏事件中的表现进行了评估。G系数为0.68，其中26%为受试者方差，20%为评分者方差，无病例方差。33%的方差归因于三阶交互作用和未知因素。Gwet AC 2为0.3，组内相关系数为0.58。CALM评分与肾上腺素给药时间的相关性为-0.79（P = 0.01）。CALM评分与液体给药时间的相关性为-0.181（P = 0.64）。
本研究为CALM工具在这种情况下与多名评分者一起使用提供了额外的有效性证据，与之前基于模拟的CALM有效性研究的数据一致。进一步的开发可能会提高可靠性。它还作为在医学模拟中进行有效性工作严谨性的一个范例。
