Soghikian Sierra, Chipman Micheline, Holmes Jeffrey, Calhoun Aaron W, Mallory Leah A
Maine Track Program, Tufts University School of Medicine, Boston, USA.
Medical Education and Simulation, Hannaford Center for Safety, Innovation and Simulation, MaineHealth Brighton Campus, Portland, USA.
Cureus. 2024 Sep 6;16(9):e68810. doi: 10.7759/cureus.68810. eCollection 2024 Sep.
Introduction Neonatal resuscitation is a high-acuity, low-occurrence event that requires ongoing practice by interprofessional teams to maintain proficiency. Simulation provides an ideal platform for team training and evaluation of team performance. Our simulation center supports a longitudinal in situ simulation training program for delivery room teams. In addition to adherence to the Neonatal Resuscitation Program standards, team performance assessment is an essential component of program evaluation and participant feedback. Multiple published teamwork assessment tools exist. Our objective was to select the tool with the best validity evidence for our program's needs. Methods We used Messick's framework to assess the validity of evidence for potential teamwork assessment tools. Four possible tools were identified from the literature: the Mayo High Performance Teamwork Scale (Mayo), Team Performance Observation Tool (TPOT), Clinical Teamwork Scale (CTS), and Team Emergency Assessment Measure (TEAM). Relevant context included team versus individual focus, external evaluator versus self-evaluation, and ease of use (which included efficiency, clarity of interpretation, and overall assessment). Three simulation experts identified consensus anchors for each tool and independently reviewed and scored 10 pre-recorded neonatal resuscitation simulations. Raters assigned each tool a rating according to efficiency, ease of interpretation, and completeness of teamwork assessment. Interrater reliability (IRR) was calculated using intraclass correlation for each tool across the three raters. Average team performance scores for each tool were correlated with neonatal resuscitation adherence scores for each video using Spearman's rank coefficient. Results There was a range of IRR between the tools, with Mayo having the best (single 0.55 and multi 0.78). Each of the three raters ranked Mayo optimally in terms of efficiency (mean 4.66 + 0.577) and ease of use (4+1). However, TPOT and CTS scored highest (mean 4.66 ± 0.577) for overall completeness of teamwork assessment. There was no significant correlation to NRP adherence scores for any teamwork tool. Conclusion Of the four tools assessed, Mayo demonstrated moderate IRR and scored highest for its ease of use and efficiency, though not completeness of assessment. The remaining three tools had poor IRR, which is not an uncommon problem with teamwork assessment tools. Our process emphasizes the fact that assessment tool validity is contextual. Factors such as a relatively narrow (and high) performance distribution and clinical context may have contributed to reliability challenges for tools that offered a more complete teamwork assessment.
引言 新生儿复苏是一种高风险、低发生率的事件，跨专业团队需要持续练习以保持熟练程度。模拟为团队培训和团队绩效评估提供了一个理想平台。我们的模拟中心为产房团队支持一项纵向现场模拟培训计划。除了遵守新生儿复苏计划标准外，团队绩效评估是计划评估和参与者反馈的重要组成部分。有多种已发表的团队合作评估工具。我们的目标是为我们计划的需求选择具有最佳效度证据的工具。方法 我们使用梅西克框架来评估潜在团队合作评估工具的证据效度。从文献中确定了四种可能的工具：梅奥高效团队合作量表（梅奥）、团队绩效观察工具（TPOT）、临床团队合作量表（CTS）和团队紧急评估措施（TEAM）。相关背景包括团队与个人重点、外部评估者与自我评估以及易用性（包括效率、解释清晰度和整体评估）。三位模拟专家为每种工具确定了共识锚点，并独立审查和评分10个预先录制的新生儿复苏模拟。评分者根据效率、解释的难易程度和团队合作评估的完整性为每种工具分配一个评分。使用组内相关系数计算三位评分者对每种工具的评分者间信度（IRR）。使用斯皮尔曼等级系数将每种工具的平均团队绩效得分与每个视频的新生儿复苏依从性得分相关联。结果 这些工具之间的IRR存在差异，梅奥的IRR最佳（单个为0.55，多个为0.78）。三位评分者中的每一位在效率（平均4.66 + 0.577）和易用性（4 + 1）方面都将梅奥评为最佳。然而，TPOT和CTS在团队合作评估的整体完整性方面得分最高（平均4.66 ± 0.577）。任何团队合作工具与新生儿复苏计划依从性得分均无显著相关性。结论 在评估的四种工具中，梅奥表现出中等的IRR，在易用性和效率方面得分最高，尽管在评估完整性方面并非如此。其余三种工具的IRR较差，这在团队合作评估工具中并非罕见问题。我们的过程强调了评估工具效度是与背景相关的这一事实。诸如相对狭窄（且高）的绩效分布和临床背景等因素可能导致提供更完整团队合作评估的工具在可靠性方面面临挑战。
