Classifying Biomedical Figures by Modality via Multi-Label Learning.

The figures found in biomedical literature are a vital part of biomedical research, education, and clinical decision. The multitude of their modalities and the lack of corresponding metadata constitute search and information, retrieval a difficult task. In this paper, we introduce novel multi-label modality classification approaches for biomedical figures without segmenting the compound figures. In particular, we investigate using both simple and compound figures for training a multi-label model to be used for annotating either all figures or only those predicted as compound by a compound figure detection model. Using data from the medical task of ImageCLEF 2016, we train our approaches with visual features and compare them with the approach involving compound figure separation into sub-figures. Furthermore, we study how multimodal learning, from both visual and textual features affects the tasks of classifying biomedical figures by modality and detecting compound figures. Finally, we present a web application for medical figure retrieval, which is based on one of our classification approaches and allows users to search for figures of PubMed Central from any device and provide feedback about the modality of a figure classified by the system.