Hasan Mohammad Junayed, Rahman Fuad, Mohammed Nabeel
Apurba NSU R&D Lab, Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh.
Apurba Technologies Ltd., Dhaka, Bangladesh.
Int J Med Inform. 2025 Mar;195:105764. doi: 10.1016/j.ijmedinf.2024.105764. Epub 2024 Dec 18.
Clinical Language Models (CLMs) possess the potential to reform traditional healthcare systems by aiding in clinical decision making and optimal resource utilization. They can enhance patient outcomes and help healthcare management through predictive clinical tasks. However, their real-world deployment is limited due to high computational cost at inference, in terms of both time and space complexity.
This study aims to develop and optimize an efficient framework that compresses CLMs without significant performance loss, reducing inference time and disk-space, and enabling real-world clinical applications.
We introduce OptimCLM, a framework for optimizing CLMs with ensemble learning, knowledge distillation (KD), pruning and quantization. Based on domain-knowledge and performance, we select and combine domain-adaptive CLMs DischargeBERT and COReBERT as the teacher ensemble model. We transfer the teacher's knowledge to two smaller generalist models, BERT-PKD and TinyBERT, and apply black-box KD, post-training unstructured pruning and post-training 8-bit model quantization to them. In an admission-to-discharge setting, we evaluate the framework on four clinical outcome prediction tasks (length of stay prediction, mortality prediction, diagnosis prediction and procedure prediction) using admission notes from the MIMIC-III clinical database.
The OptimCLM framework achieved up to 22.88× compression ratio and 28.7× inference speedup, with less than 5% and 2% loss in macro-averaged AUROC for TinyBERT and BERT-PKD, respectively. The teacher model outperformed five state-of-the-art models on all tasks. The optimized BERT-PKD model also outperformed them in most tasks.
Our findings suggest that domain-specific fine-tuning with ensemble learning and KD is more effective than domain-specific pre-training for domain-knowledge transfer and text classification tasks. Thus, this work demonstrates the feasibility and potential of deploying optimized CLMs in healthcare settings and developing them with less computational resources.
临床语言模型（CLMs）有潜力通过辅助临床决策和优化资源利用来改革传统医疗系统。它们可以改善患者预后，并通过预测性临床任务帮助医疗管理。然而，由于推理时在时间和空间复杂度方面的高计算成本，其在现实世界中的部署受到限制。
本研究旨在开发和优化一个高效框架，该框架能在不显著损失性能的情况下压缩CLMs，减少推理时间和磁盘空间，并实现现实世界的临床应用。
我们引入了OptimCLM，这是一个通过集成学习、知识蒸馏（KD）、剪枝和量化来优化CLMs的框架。基于领域知识和性能，我们选择并组合领域自适应CLMs DischargeBERT和COReBERT作为教师集成模型。我们将教师的知识转移到两个较小的通用模型BERT-PKD和TinyBERT，并对它们应用黑箱KD、训练后非结构化剪枝和训练后8位模型量化。在入院到出院的场景中，我们使用MIMIC-III临床数据库中的入院记录，在四个临床结局预测任务（住院时间预测、死亡率预测、诊断预测和手术预测）上评估该框架。
OptimCLM框架实现了高达22.88倍的压缩率和28.7倍的推理加速，TinyBERT和BERT-PKD的宏平均AUROC损失分别小于5%和2%。教师模型在所有任务上均优于五个最先进的模型。优化后的BERT-PKD模型在大多数任务上也优于它们。
我们的研究结果表明，对于领域知识转移和文本分类任务，使用集成学习和KD进行特定领域的微调比特定领域的预训练更有效。因此，这项工作证明了在医疗环境中部署优化后的CLMs并以更少的计算资源开发它们的可行性和潜力。
