Department of Anesthesia, Women's College Hospital, 76 Grenville Street, Toronto, ON M5S 1B2, Canada.
BMJ Qual Saf. 2012 Jan;21(1):78-82. doi: 10.1136/bmjqs-2011-000296. Epub 2011 Oct 12.
To determine the effectiveness of high-fidelity simulation for team training, a valid and reliable tool is required. This study investigated the internal consistency, inter-rater reliability and test-retest reliability of two newly developed tools to assess obstetrical team performance.
After research ethics board approval, multidisciplinary obstetrical teams participated in three sessions separated by 5-9 months and managed four high-fidelity simulation scenarios. Two tools, an 18-item Assessment of Obstetric Team Performance (AOTP) and a six-item Global Assessment of Obstetric Team Performance (GAOTP) were used.(5) Eight reviewers rated the DVDs of all teams' performances.
Two AOTP items were consistently incomplete and omitted from the analyses. Cronbach's α for the 16-item AOTP was 0.96, and 0.91 for the six-item GAOTP. The eight-rater α for the GAOTP was 0.81 (single-rater intra-class correlation coefficient, 0.34) indicating acceptable inter-rater reliability. The 'four-scenario' α for the 12 teams was 0.79 for session 1, 0.88 for session 2, and 0.86 for session 3, suggesting that performance is not being strongly affected by the context specificity of the cases. Pearson's correlation of team performance scores for the four scenarios were 0.59, 0.35, 0.40 and 0.33, and for the total score across scenarios it was 0.47, indicating moderate test-retest reliability.
The results from this study indicate that the GAOTP would be a sufficient assessment tool for obstetrical team performance using simulation provided that it is used to assess teams with at least eight raters to ensure a sufficiently stable score. This could allow the quantitative evaluation of an educational intervention.
为了确定高保真模拟在团队培训中的有效性，需要使用一种有效且可靠的工具。本研究旨在调查两种新开发的评估产科团队绩效的工具的内部一致性、评分者间信度和重测信度。
在获得研究伦理委员会的批准后，多学科产科团队参加了三个相隔 5-9 个月的会议，并管理了四个高保真模拟场景。使用两种工具，即 18 项产科团队绩效评估工具（AOTP）和 6 项产科团队整体绩效评估工具（GAOTP）。八名评审员对所有团队表现的 DVD 进行评分。
有两个 AOTP 项目始终不完整，并从分析中排除。16 项 AOTP 的 Cronbach's α 为 0.96，6 项 GAOTP 的 Cronbach's α 为 0.91。GAOTP 的八名评分者 α 为 0.81（单一评分者组内相关系数为 0.34），表明评分者间信度可接受。在三个会议中，12 个团队的“四场景”α 值分别为 0.79、0.88 和 0.86，表明绩效不会受到案例情境特异性的强烈影响。四个场景的团队绩效评分之间的 Pearson 相关系数分别为 0.59、0.35、0.40 和 0.33，以及整个场景的总分相关系数为 0.47，表明重测信度中等。
本研究结果表明，GAOTP 是一种用于评估模拟中产科团队绩效的充分评估工具，前提是使用至少八名评分者来评估团队，以确保评分稳定。这可以允许对教育干预进行定量评估。
