IEEE J Biomed Health Inform. 2023 Oct;27(10):5110-5121. doi: 10.1109/JBHI.2023.3236661. Epub 2023 Oct 5.
Automatic generation of medical reports can provide diagnostic assistance to doctors and reduce their workload. To improve the quality of the generated medical reports, injecting auxiliary information through knowledge graphs or templates into the model is widely adopted in previous methods. However, they suffer from two problems: 1) The injected external information is limited in amount and difficult to adequately meet the information needs of medical report generation in content. 2) The injected external information increases the complexity of model and is hard to be reasonably integrated into the generation process of medical reports. Therefore, we propose an Information Calibrated Transformer (ICT) to address the above issues. First, we design a Precursor-information Enhancement Module (PEM), which can effectively extract numerous inter-intra report features from the datasets as the auxiliary information without external injection. And the auxiliary information can be dynamically updated with the training process. Secondly, a combination mode, which consists of PEM and our proposed Information Calibration Attention Module (ICA), is designed and embedded into ICT. In this method, the auxiliary information extracted from PEM is flexibly injected into ICT and the increment of model parameters is small. The comprehensive evaluations validate that the ICT is not only superior to previous methods in the X-Ray datasets, IU-X-Ray and MIMIC-CXR, but also successfully be extended to a CT COVID-19 dataset COV-CTR.
自动生成医学报告可以为医生提供诊断辅助，并减轻他们的工作量。为了提高生成的医学报告的质量，在之前的方法中广泛采用通过知识图谱或模板向模型中注入辅助信息的方法。然而，它们存在两个问题：1）注入的外部信息数量有限，难以充分满足医学报告生成内容方面的信息需求。2）注入的外部信息增加了模型的复杂性，难以合理地整合到医学报告的生成过程中。因此，我们提出了一种信息校准的 Transformer（ICT）来解决上述问题。首先，我们设计了一个前兆信息增强模块（PEM），它可以从数据集中有效地提取大量的报告内和报告间特征作为辅助信息，而无需外部注入。并且辅助信息可以随着训练过程动态更新。其次，我们设计并嵌入了一个由 PEM 和我们提出的信息校准注意力模块（ICA）组成的组合模式到 ICT 中。在这种方法中，PEM 中提取的辅助信息可以灵活地注入到 ICT 中，并且模型参数的增加很小。综合评估验证了 ICT 不仅在 X 射线数据集、IU-X-Ray 和 MIMIC-CXR 上优于以前的方法，而且还成功地扩展到了 CT COVID-19 数据集 COV-CTR。
