Wang Xu Wen, Zhang Yu, Guo Zhen, Li Jiao
Institute of Medical Information and Library, Chinese Academy of Medical Sciences/Peking Union Medical College, Beijing 100020, China.
Math Biosci Eng. 2019 Mar 8;16(4):1978-1991. doi: 10.3934/mbe.2019097.
Automatically identifying semantic concepts from medical images provides multimodal insights for clinical research. To study the effectiveness of concept detection on large scale medical images, we reconstructed over 230,000 medical image-concepts pairs collected from the ImageCLEFcaption 2018 evaluation task. A transfer learning-based multi-label classification model was used to predict multiple high-frequency concepts for medical images. Semantically relevant concepts of visually similar medical images were identified by the image retrieval-based topic model. The results showed that the transfer learning method achieved F1 score of 0.1298, which was comparable with the state of art methods in the ImageCLEFcaption tasks. The image retrieval-based method contributed to the recall performance but reduced the overall F1 score, since the retrieval results of the search engine introduced irrelevant concepts. Although our proposed method achieved second-best performance in the concept detection subtask of ImageCLEFcaption 2018, there will be plenty of further work to improve the concept detection with better understanding the medical images.
从医学图像中自动识别语义概念可为临床研究提供多模态见解。为了研究概念检测在大规模医学图像上的有效性，我们重建了从2018年ImageCLEFcaption评估任务中收集的超过230,000个医学图像 - 概念对。使用基于迁移学习的多标签分类模型来预测医学图像的多个高频概念。通过基于图像检索的主题模型识别视觉上相似的医学图像的语义相关概念。结果表明，迁移学习方法的F1分数达到0.1298，与ImageCLEFcaption任务中的现有方法相当。基于图像检索的方法有助于召回性能，但降低了整体F1分数，因为搜索引擎的检索结果引入了不相关的概念。尽管我们提出的方法在2018年ImageCLEFcaption的概念检测子任务中取得了第二好的性能，但仍有大量进一步的工作需要通过更好地理解医学图像来改进概念检测。
