Medical Image Labelling and Semantic Understanding for Clinical Applications

Abstract

Semantic concept detection contributes to machine understanding and learning from medical images; it also plays an important role in image reading and image-assisted diagnosis. In this study, the problem of detecting high-frequency concepts from medical images was transformed into a multi-label classification task. The transfer learning method based on convolutional neural networks (CNNs) was used to recognize high-frequency medical concepts. The image retrieval-based topic modelling method was used to obtain the semantically related concepts from images similar to the given medical images. Our group participated in the concept detection subtasks that were launched by ImageCLEFcaption 2018 and ImageCLEFmed Caption 2019. In the 2018 task, the CNN-based transfer learning method achieved an F1 score of 0.0928, while the retrieval-based topic model achieved an F1 score of 0.0907. Although the latter method recalled some low-frequency concepts, it heavily depended on the image retrieval results. For the latter 2019 task, we proposed body part-based pre-classification strategies and achieved an F1 score of 0.2235. The results indicated that the transfer learning-based multi-label classification method was more robust in high-frequency concept detection across different data sets, but there is still much room for improvement in large-scale open semantic concept detection research.