Bloom Ben, Haimovich Adrian, Pott Jason, Williams Sophie L, Cheetham Michael, Langsted Sandra, Skene Imogen, Astin-Chamberlain Raine, Thomas Stephen H
Emergency Department, Royal London Hospital, Barts Health NHS Trust, London, UK
Queen Mary University of London Blizard Institute, London, UK.
BMJ Health Care Inform. 2025 Jul 25;32(1):e101433. doi: 10.1136/bmjhci-2025-101433.
Identifying whether there is a traumatic intracranial bleed (ICB+) on head CT is critical for clinical care and research. Free text CT reports are unstructured and therefore must undergo time-consuming manual review. Existing artificial intelligence classification schemes are not optimised for the emergency department endpoint of classification of ICB+ or ICB-. We sought to assess three methods for classifying CT reports: a text classification (TC) programme, a commercial natural language processing programme (Clinithink) and a generative pretrained transformer large language model (Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting (DECIPHER)-LLM).
Primary objective: determine the diagnostic classification performance of the dichotomous categorisation of each of the three approaches.
determine whether the LLM could achieve a substantial reduction in CT report review workload while maintaining 100% sensitivity.Anonymised radiology reports of head CT scans performed for trauma were manually labelled as ICB+/-. Training and validation sets were randomly created to train the TC and natural language processing models. Prompts were written to train the LLM.
898 reports were manually labelled. Sensitivity and specificity (95% CI)) of TC, Clinithink and DECIPHER-LLM (with probability of ICB set at 10%) were respectively 87.9% (76.7% to 95.0%) and 98.2% (96.3% to 99.3%), 75.9% (62.8% to 86.1%) and 96.2% (93.8% to 97.8%) and 100% (93.8% to 100%) and 97.4% (95.3% to 98.8%).With DECIPHER-LLM probability of ICB+ threshold of 10% set to identify CT reports requiring manual evaluation, CT reports requiring manual classification reduced by an estimated 385/449 cases (85.7% (95% CI 82.1% to 88.9%)) while maintaining 100% sensitivity.
DECIPHER-LLM outperformed other tested free-text classification methods.
识别头部CT上是否存在创伤性颅内出血（ICB+）对临床护理和研究至关重要。CT报告的自由文本是非结构化的，因此必须经过耗时的人工审核。现有的人工智能分类方案并未针对ICB+或ICB-分类的急诊科终点进行优化。我们试图评估三种对CT报告进行分类的方法：一种文本分类（TC）程序、一种商业自然语言处理程序（Clinithink）和一种生成式预训练变换器大语言模型（用于阳性出血评估报告的数字化英语CT解读（DECIPHER）-LLM）。
主要目标：确定三种方法中每种方法的二分法分类的诊断分类性能。
确定大语言模型是否能在保持100%敏感性的同时大幅减少CT报告审核工作量。对因创伤进行的头部CT扫描的匿名放射学报告进行人工标记为ICB+/ -。随机创建训练集和验证集来训练TC和自然语言处理模型。编写提示来训练大语言模型。
898份报告被人工标记。TC、Clinithink和DECIPHER-LLM（将ICB的概率设定为10%）的敏感性和特异性（95%CI）分别为87.9%（76.7%至95.0%）和98.2%（96.3%至99.3%）、75.9%（62.8%至86.1%）和96.2%（93.8%至97.8%）以及100%（93.8%至100%）和97.4%（95.3%至98.8%）。将DECIPHER-LLM的ICB+概率阈值设定为10%以识别需要人工评估的CT报告时，需要人工分类的CT报告估计减少了385/449例（85.
