Cold Kristoffer Mazanti, Agbontaen Kaladerhan, Nielsen Anne Orholm, Andersen Christian Skjoldvang, Singh Suveer, Konge Lars
Copenhagen Academy for Medical Education and Simulation (CAMES), Rigshospitalet, University of Copenhagen, the Capital Region of Denmark, Copenhagen, Denmark.
Department of Intensive Care Unit, Chelsea and Westminster Hospital, Chelsea, London, UK.
J Thorac Dis. 2024 Sep 30;16(9):5718-5726. doi: 10.21037/jtd-24-841. Epub 2024 Sep 6.
Bronchoscopy is a challenging technical procedure, and assessment of competence currently relies on expert raters. Human rating is time consuming and prone to rater bias. The aim of this study was to evaluate if a bronchial segment identification system based on artificial intelligence (AI) could automatically, instantly, and objectively assess competencies in flexible bronchoscopy in a valid way.
Participants were recruited at the Clinical Skills Zone of the European Respiratory Society Annual Conference in Milan, 9-13 September 2023. The participants performed one full diagnostic bronchoscopy in a simulated setting and were rated immediately by the AI according to its four outcome measures: diagnostic completeness (DC), structured progress (SP), procedure time (PT), and mean intersegmental time (MIT). The procedures were video-recorded and rated after the conference by two blinded, expert raters using a previously validated assessment tool with nine items regarding anatomy and dexterity.
Fifty-two participants from six different continents were included. All four outcome measures of the AI correlated significantly with the experts' anatomy-ratings (Pearson's correlation coefficient, P value): DC (r=0.47, P<0.001), SP (r=0.57, P<0.001), PT (r=-0.32, P=0.02), and MIT (r=-0.55, P<0.001) and also with the experts' dexterity-ratings: DC (r=0.38, P=0.006), SP (r=0.53, P<0.001), PT (r=-0.34, P=0.014), and MIT (r=-0.47, P<0.001).
The study provides initial validity evidence for AI-based immediate and automatic assessment of anatomical and navigational competencies in flexible bronchoscopy. SP provided stronger correlations with human experts' ratings than the traditional DC.
支气管镜检查是一项具有挑战性的技术操作，目前对操作能力的评估依赖于专家评分。人工评分耗时且容易出现评分偏差。本研究的目的是评估基于人工智能（AI）的支气管节段识别系统能否以有效的方式自动、即时且客观地评估柔性支气管镜检查的操作能力。
在2023年9月9日至13日于米兰举行的欧洲呼吸学会年会临床技能区招募参与者。参与者在模拟环境中进行一次完整的诊断性支气管镜检查，并由人工智能根据其四项结果指标立即进行评分：诊断完整性（DC）、结构化进展（SP）、操作时间（PT）和平均节段间时间（MIT）。操作过程进行了视频记录，会议结束后由两名不知情的专家评分者使用先前验证的包含九项关于解剖结构和操作灵活性的评估工具进行评分。
纳入了来自六大洲的52名参与者。人工智能的所有四项结果指标与专家的解剖结构评分均显著相关（Pearson相关系数，P值）：DC（r = 0.47，P < 0.001）、SP（r = 0.57，P < 0.001）、PT（r = -0.32，P = 0.02）和MIT（r = -0.55，P < 0.001），并且与专家的操作灵活性评分也显著相关：DC（r = 0.38，P = 0.006）、SP（r = 0.53，P < 0.001）、PT（r = -0.34，P = 0.014）和MIT（r = -0.47，P < 0.001）。
该研究为基于人工智能的即时自动评估柔性支气管镜检查中的解剖和导航能力提供了初步的有效性证据。与传统的DC相比，SP与人类专家评分的相关性更强。
