Nuffield Department of Clinical Neurosciences, University of Oxford, Level 6, West Wing, John Radcliffe Hospital, Oxford, OX3 9DU, England.
The Medical Specialist Group LLP Guernsey, Saint Peter Port, Guernsey.
BMC Med Educ. 2023 Mar 11;23(1):153. doi: 10.1186/s12909-023-04108-4.
Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.
Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.
Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.
The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.
非技术技能（NTS）评估工具广泛用于为医疗保健专业人员提供形成性和总结性评估，现在有很多这样的工具。本研究检查了三种为类似环境设计的不同工具，并收集证据来测试它们的有效性和可用性。
三位经验丰富的教师使用三种专为英国设计的 NTS 评估工具来审查模拟心脏骤停场景的标准化视频：ANTS（麻醉师非技术技能）、牛津 NOTECHS（牛津非技术技能）和 OSCAR（复苏观察技能基础临床评估工具）。对每个工具进行了内部一致性、评分者间可靠性以及可用性的定量和定性分析。
在 NTS 类别和要素方面，三个工具的内部一致性和评分者间可靠性（IRR）差异很大。三位专家评分者的组内相关系数得分范围从差（ANTS 中的任务管理 [0.26]和牛津 NOTECHS 中的情境意识 [0.34]）到非常好（牛津 NOTECHS 中的问题解决 [0.81]和合作 [0.84]以及 OSCAR 中的 SA [0.87]）。此外，每个工具的 IRR 的不同统计检验产生了不同的结果。对可用性的定量和定性检查也揭示了使用每个工具的挑战。
缺乏 NTS 评估工具的标准化以及对其使用的培训对医疗保健教育者和学生没有帮助。教育者需要在使用 NTS 评估工具评估个别医疗保健专业人员或医疗保健团队方面得到持续支持。使用 NTS 评估工具进行总结性或高风险的考试，至少应由两名评估员进行评分，以提供共识评分。鉴于人们对模拟作为一种教育工具的重新关注，以支持和加强 COVID-19 后的培训恢复，标准化、简化和提供充足培训来支持这些重要技能的评估变得更加重要。
