Multi-channel content based image retrieval method for skin diseases using similarity network fusion and deep community analysis

Abstract

Content-based image retrieval (CBIR) could be an efficient diagnostic tool. Physicians could consult a CBIR system before making a diagnosis for a clinical case by retrieving a set of images with similar appearance and pathological diagnosis from a data archive. With access to various imaging modalities, physicians may want to match more than one image modality and non-image information. How to make full use of this diverse information is an important research question. In this paper, we propose a CBIR framework for skin diseases that incorporates multi-sourced information including dermoscopic images, clinical images, and meta information. The proposed framework fuses the multi-sourced features in mutual similarity level; thus, solving severe dimensional bias problems for image and non-image information. We then utilize a graph-based community analysis on similarity networks where similar images are strongly connected and help retrieve similar images with improved performance. Evaluations were carried out using two well-known skin datasets EDRA and ISIC 2019. The carefully designed framework demonstrates a substantial improvement in finding similar cases for different skin diseases with an average precision of 0.836, which is the state-of-the-art performance for retrieving skin disease types. In addition, the proposed framework is also applicable to scenarios with a single typed feature with improved performance. By integrating multi-sourced information from the same patient, the proposed CBIR system could be potentially used in complex clinical scenarios with a trustable performance benefitting from both abundant information and advanced community search technique.