Wang Mengfei, Wei Jianyong, Zeng Yao, Dai Lisong, Yan Bicong, Zhu Yueqi, Wei Xiaoer, Jin Yidong, Li Yuehua
School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, People's Republic of China.
Clinical Research Center, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People's Republic of China.
J Multidiscip Healthc. 2024 Nov 14;17:5163-5175. doi: 10.2147/JMDH.S486449. eCollection 2024.
Mechanical thrombectomy (MTB) is a critical procedure for acute ischemic stroke (AIS) patients. However, the free-text format of MTB surgical records limits the formulation of effective postoperative patient management and rehabilitation plans. This study compares the efficacy of large language models (LLMs) in structuring data from these free-text MTB surgical record.
This retrospective study collected a total of 382 MTB surgical records from a tertiary hospital. An initial analysis of 30 surgical record from these records provided a guiding prompt for LLMs, focusing on basic and advanced characteristics, such as occlusion locations, thrombectomy maneuvers, reperfusion status, and intraoperative complications. Six LLMs-ChatGPT, GPT-4, GeminiPro, ChatGLM4, Spark3, and QwenMax-were assessed against data extracted by neuroradiologists and a junior physician for comparison. The all 382 surgical records were used to test the performance of LLMs. The performance of the LLMs was quantified using Accuracy, Sensitivity, Specificity, AUC, and MSE as an additional metric for advanced characteristics.
All LLMs showed high performance in characteristic extraction, achieving an average accuracy of 95.09 ± 4.98% across 48 items, and 78.05 ± 4.2% overall. GLM4 and GPT-4 were most accurate in advanced characteristics extraction, with accuracies of 84.03% and 82.20%, respectively. The processing time for LLMs averaged 73.10 ± 10.86 seconds of six models, significantly faster than the 427.88 seconds for manual extraction by physicians.
LLMs, particularly GLM4 and GPT-4, efficiently and accurately structured both general and advanced characteristics from MTB surgical record, outperforming manual extraction methods and demonstrating potential for enhancing clinical data management in AIS treatment.
机械取栓术（MTB）是急性缺血性卒中（AIS）患者的关键治疗手段。然而，MTB手术记录的自由文本格式限制了有效的术后患者管理和康复计划的制定。本研究比较了大语言模型（LLMs）在构建这些自由文本MTB手术记录数据方面的效果。
这项回顾性研究共收集了一家三级医院的382份MTB手术记录。对其中30份手术记录的初步分析为LLMs提供了指导提示，重点关注基本和高级特征，如闭塞部位、取栓操作、再灌注状态和术中并发症。将六个LLMs——ChatGPT、GPT-4、GeminiPro、ChatGLM4、Spark3和QwenMax——与神经放射科医生和一名初级医生提取的数据进行评估比较。使用所有382份手术记录来测试LLMs的性能。通过准确率、灵敏度、特异性、AUC以及作为高级特征附加指标的MSE对LLMs的性能进行量化。
所有LLMs在特征提取方面表现出高性能，48项特征的平均准确率为95.09±4.98%，总体准确率为78.05±4.2%。ChatGLM4和GPT-4在高级特征提取方面最为准确，准确率分别为84.03%和82.20%。六个模型的LLMs平均处理时间为73.10±10.86秒，明显快于医生手动提取的427.88秒。
LLMs，尤其是ChatGLM4和GPT-4，能够高效、准确地构建MTB手术记录中的一般和高级特征，优于手动提取方法，并显示出在AIS治疗中加强临床数据管理的潜力。
