Zhang Yufeng, Kohne Joseph G, Webster Katherine, Vartanian Rebecca, Wittrup Emily, Najarian Kayvan
Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, MI 48105, United States.
Department of Pediatrics, University of Michigan, Ann Arbor, MI 48105, United States.
JAMIA Open. 2025 Feb 10;8(1):ooaf008. doi: 10.1093/jamiaopen/ooaf008. eCollection 2025 Feb.
The lack of a publicly accessible abdominal X-ray (AXR) dataset has hindered necrotizing enterocolitis (NEC) research. While significant strides have been made in applying natural language processing (NLP) to radiology reports, most efforts have focused on chest radiology. Development of an accurate NLP model to identify features of NEC on abdominal radiograph can support efforts to improve diagnostic accuracy for this and other rare pediatric conditions.
This study aims to develop privacy-preserving large language models (LLMs) and their distilled version to efficiently annotate pediatric AXR reports.
Utilizing pediatric AXR reports collected from C.S. Mott Children's Hospital, we introduced AXpert in 2 formats: one based on the instruction-fine-tuned 7-B Gemma model, and a distilled version employing a BERT-based model derived from the fine-tuned model to improve inference and fine-tuning efficiency. AXpert aims to detect NEC presence and classify its subtypes-pneumatosis, portal venous gas, and free air.
Extensive testing shows that LLMs, including Axpert, outperforms baseline BERT models on all metrics. Specifically, Gemma-7B (F1 score: 0.9 ± 0.015) improves upon BlueBERT by 132% in F1 score for detecting NEC positive samples. The distilled BERT model matches the performance of the LLM labelers and surpasses expert-trained baseline BERT models.
Our findings highlight the potential of using LLMs for clinical NLP tasks. With minimal expert knowledge injections, LLMs can achieve human-like performance, greatly reducing manual labor. Privacy concerns are alleviated as all models are trained and deployed locally.
AXpert demonstrates potential to reduce human labeling efforts while maintaining high accuracy in automating NEC diagnosis with AXR, offering precise image labeling capabilities.
缺乏公开可用的腹部X光（AXR）数据集阻碍了坏死性小肠结肠炎（NEC）的研究。虽然在将自然语言处理（NLP）应用于放射学报告方面已经取得了重大进展，但大多数努力都集中在胸部放射学上。开发一种准确的NLP模型以识别腹部X光片上NEC的特征，可以支持提高对此病及其他罕见儿科疾病诊断准确性的努力。
本研究旨在开发隐私保护大语言模型（LLMs）及其精简版本，以有效地注释儿科AXR报告。
利用从C.S. 莫特儿童医院收集的儿科AXR报告，我们以两种格式引入了AXpert：一种基于指令微调的7-B Gemma模型，另一种精简版本采用从微调模型派生的基于BERT的模型，以提高推理和微调效率。AXpert旨在检测NEC的存在并对其亚型——积气、门静脉积气和游离气体进行分类。
广泛测试表明，包括Axpert在内的大语言模型在所有指标上均优于基线BERT模型。具体而言，Gemma-7B（F1分数：0.9±0.015）在检测NEC阳性样本的F1分数方面比BlueBERT提高了132%。精简后的BERT模型与大语言模型标注器的性能相当，并且超过了专家训练的基线BERT模型。
我们的研究结果突出了使用大语言模型进行临床NLP任务的潜力。只需极少的专家知识注入，大语言模型就能达到类似人类的性能，大大减少了人工劳动。由于所有模型都是在本地进行训练和部署的，因此减轻了隐私方面的担忧。
AXpert展示了在使用AXR自动诊断NEC时减少人工标注工作量同时保持高精度的潜力，提供了精确的图像标注能力。
