Content-based Medical Image Retrieval Research Articles

A novel image-based retrieval system for characterization of maxillofacial lesions in cone beam CT images.

The objective of medical content-based image retrieval (CBIR) is to assist clinicians in decision making by retrieving the most similar cases to a given query image from a large database. Herein, a new method for content-based image retrieval of cone beam CT (CBCT) scans is presented. The introduced framework consists of two main phases: training database construction and querying. The goal of the training phase is database construction, which consists of three main steps. First, automatic segmentation of lesions using 3D symmetry analysis is performed. Embedding the prior shape knowledge of the 3D symmetry characteristics of the healthy human head structure increases the accuracy of automatic segmentation. Then, spatial pyramid matching is used for feature extraction, and the relative importance of each feature is learned using classifiers. The method was applied to a dataset of 1145 volumetric CBCT images with four classes of maxillofacial lesions. A symmetry-based analysis model for automatic lesion segmentation was evaluated using similarity measures. Mean Dice coefficients of 0.89, 0.85, 0.92, and 0.87 were achieved for maxillary sinus perforation, radiolucent lesion, unerupted tooth, and root fracture classes, respectively. Moreover, the execution time of automatic segmentation was reduced to 3min per case. The performance of the proposed search engine was evaluated using mean average precision and normalized discounted cumulative gain. A mean average retrieval accuracy and normalized discounted cumulative gain of 0.90 and 0.92, respectively, were achieved. Quantitative results show that the proposed approach is more effective than previous methods in the literature, and it can facilitate the introduction of CBIR in clinical CBCT applications.

A rotation and translation invariant method for 3D organ image classification using deep convolutional neural networks.

Three-dimensional (3D) medical image classification is useful in applications such as disease diagnosis and content-based medical image retrieval. It is a challenging task due to several reasons. First, image intensity values are vastly different depending on the image modality. Second, intensity values within the same image modality may vary depending on the imaging machine and artifacts may also be introduced in the imaging process. Third, processing 3D data requires high computational power. In recent years, significant research has been conducted in the field of 3D medical image classification. However, most of these make assumptions about patient orientation and imaging direction to simplify the problem and/or work with the full 3D images. As such, they perform poorly when these assumptions are not met. In this paper, we propose a method of classification for 3D organ images that is rotation and translation invariant. To this end, we extract a representative two-dimensional (2D) slice along the plane of best symmetry from the 3D image. We then use this slice to represent the 3D image and use a 20-layer deep convolutional neural network (DCNN) to perform the classification task. We show experimentally, using multi-modal data, that our method is comparable to existing methods when the assumptions of patient orientation and viewing direction are met. Notably, it shows similarly high accuracy even when these assumptions are violated, where other methods fail. We also explore how this method can be used with other DCNN models as well as conventional classification approaches.

Integration of global and local features based on hybrid similarity matching scheme for medical image retrieval system

Similarity measure is a challenging task in Content-Based Medical Image Retrieval (CBMIR) systems and the matching scheme is designed to improve the retrieval performance. However, there are several major shortcomings with conventional approaches for a matching scheme which can extensively affect their application of Medical Image Retrieval (MIR). To overcome the issues, in this paper a Multi-Level Matching (MLM) method for MIR using hybrid feature similarity is proposed. Here, images are represented by multi-level features including local level and global level. The Colour and Edge Directivity Descriptor (CEDD) is used as a colour and edge-based descriptor. Speeded-Up Robust Features (SURF) and Local Binary Pattern (LBP) are used as a local descriptor. The hybrid of both global and local features yields enhanced retrieval accuracy, which is analysed over collected image databases. From the experiment, the proposed method achieves better accuracy value about 92%, which is higher than other methods.

Medical Image Retrieval Based on Convolutional Neural Network and Supervised Hashing

In recent years, with extensive application in image retrieval and other tasks, a convolutional neural network (CNN) has achieved outstanding performance. In this paper, a new content-based medical image retrieval (CBMIR) framework using CNN and hash coding is proposed. The new framework adopts a Siamese network in which pairs of images are used as inputs, and a model is learned to make images belonging to the same class have similar features by using weight sharing and a contrastive loss function. In each branch of the network, CNN is adapted to extract features, followed by hash mapping, which is used to reduce the dimensionality of feature vectors. In the training process, a new loss function is designed to make the feature vectors more distinguishable, and a regularization term is added to encourage the real value outputs to approximate the desired binary values. In the retrieval phase, the compact binary hash code of the query image is achieved from the trained network and is subsequently compared with the hash codes of the database images. We experimented on two medical image datasets: the cancer imaging archive-computed tomography (TCIA-CT) and the vision and image analysis group/international early lung cancer action program (VIA/I-ELCAP). The results indicate that our method is superior to existing hash methods and CNN methods. Compared with the traditional hashing method, feature extraction based on CNN has advantages. The proposed algorithm combining a Siamese network with the hash method is superior to the classical CNN-based methods. The application of a new loss function can effectively improve retrieval accuracy.

Densely-Connected Multi-Magnification Hashing for Histopathological Image Retrieval.

Content-based medical image retrieval is an important computer-aided diagnosis technique providing the clinicians with interpretative references based on visual similarity. In this paper, we focus on the tasks of histopathological image retrieval for breast cancer diagnosis. The densely-connected multi-magnification (DCMMH) framework is proposed to generate the discriminative binary codes by exploiting the histopathological images with multiple magnification factors. The low-magnification images are boosted by the accumulated similarity based on local patches that also regularize the feature learning of high-magnification images. In order to fully utilize the information across different magnification levels, a densely-connected architecture is finally deployed for high-low magnification pairs of datasets. Experiments on BreakHis dataset demonstrate that, DCMMH outperforms the previous hashing methods on histopathological image retrieval.

A Novel Hybrid Framework for Medical Image Retrieval

A new hybrid framework for Content-Based Medical Image Retrieval (MCBIR) is proposed in this paper to deals with the accuracy issues related with the existing MCBIR. The proposed hybrid framework initially divides the images into number of non-overlapping rectangular regions. Subsequently, statistical based color autocorrelogram (CA) and texture autocorrelogram (TA) is extracted for each region respectively. Then the geometric based chordiogram descriptor (CD) is extracted for each region. Both the statistical based and geometric based descriptors are combined to create a feature vector. The corresponding image regionsor patches in the query and target medical images are compared using the Canberra distance measure. The proposed hybrid framework is evaluated using the benchmark database and it is confirmed that it significantly outperforms the state-of-the-art system in terms precision, recall and G-measure.

Content-based medical image retrieval by spatial matching of visual words

Content-Based Image Retrieval (CBIR) systems have recently emerged as one of the most promising and best image retrieval paradigms. To pacify the semantic gap associated with CBIR systems, the Bag of Visual Words (BoVW) techniques are now increasingly used. However, existing BoVW techniques fail to capture the location information of visual words effectively. This paper proposes an unsupervised Content-Based Medical Image Retrieval (CBMIR) framework based on the spatial matching of the visual words. The proposed method efficiently computes the spatial similarity of visual words using a novel similarity measure called the Skip Similarity Index. Experiments on three large medical datasets reveal promising results. The location-based correlation of visual words assists in more accurate and efficient retrieval of anatomically diverse and multimodal medical images than the state-of-the-art CBMIR systems.

Content-based medical image retrieval system based on gradient orientation and edge information

Retrieval of biomedical pictures is a significant side of computer based diagnosis. It helps the radiologist and restorative authority to spot and analyze the particular disease. This paper proposes a Content Based Medical Image Retrieval (CBMIR) approach for retrieving similar biomedical images. The extraction of retrieving features is based on histogram of oriented gradients (HOG) and canny edge detection. To reduce the dimensionality, principal component analysis(PCA) is employed over the feature vector. The experiments are conducted on high-resolution computed tomography medical images of lungs. With the average retrieval rate (ARR) and average retrieval precision (ARP), the performance of the proposed approach is analyzed and compared with other existing methods viz. Local Binary Pattern (LBP), LBP with uniform patterns (LBPu2), Local Mesh Pattern with uniform patterns (LMePu2) and LMeP with gabor transform (GLMeP).

Transferred Semantic Scores for Scalable Retrieval of Histopathological Breast Cancer Images

Content-based medical image retrieval (CBMIR) is an active field of research and a complementary decision support tool for the diagnosis of breast cancer. Current CBMIR systems employ hand-engineered image descriptors which are not effective enough at retrieval phase. Besides this drawback, the so-called semantic gap in the CBMIR is not still addressed leaving the room for further improvements. To fill in the two mentioned existing gaps, we proposed a new retrieval method which exploited a deep pre-trained convolutional neural network model to extract class-specific and patient-specific tumorous descriptor to firstly train a binary breast cancer classifier and then a multi-patient classifier aiming for reducing dimensions of the raw deeply transferred features and obtaining semantic scores which significantly enhanced the performance in terms of mean average precision. We evaluated the method on scalable BreakHis dataset of histopathological breast cancer images. After conducting five sets of experiments, results demonstrated the superior effectiveness of the proposed semantic-driven retrieval methods by means of increased mean average precision and decreased dimensionality and retrieval time. In overall, an improvement of 29.03% was obtained by the proposed class-driven semantic retrieval method.

An Efficient Medical Content Based Image Retrieval with Lenient Relevance Feedback

Content Based Image Retrieval (CBIR) for medical imageries is still in its early stage. There are many challenging research issues. Retrieve similar images only is the current problem in medical CBIR. One idea to solve this difficult is minimizing the gap among two descriptions i.e. low level extracted features of image and high level human perception of image. There are various Relevance Feedback (WF) methods have been considered to minimize the semantic gap in medical CBIR system. But most of them were deals with hard Feedback. In Hard Feedback system user can interact with the system in one query session. We recommend to aid the usage of lenient relevance response to better capture the intention of users. The meta-knowledge mined from multiple user’s experience be able to increase the precision of subsequent image recovery results. Here we suggest an algorithm to mine lenient association rules from the group of suggestion i.e. image weight value given by the user. To reduce the amount of strong rules we offer two rule lessening techniques related to redundancy detection and confidence quantization. Best first search and Binary search methods are similarly applied to advance the procedure of weight interface. The effectiveness of the offered system is assessed regarding precision and average retrieval time. The experimental results on medical images display that the proposed method is able to improve the accuracy of medical CBIR system and reduces the retrieval time than other usual methods.

Overview on subjective similarity of images for content-based medical image retrieval.

Computer-aided diagnosis systems for assisting the classification of various diseases have the potential to improve radiologists' diagnostic accuracy and efficiency, as reported in several studies. Conventional systems generally provide the probabilities of disease types in terms of numerical values, a method that may not be efficient for radiologists who are trained by reading a large number of images. Presentation of reference images similar to those of a new case being diagnosed can supplement the probability outputs based on computerized analysis as an intuitive guide, and it can assist radiologists in their diagnosis, reporting, and treatment planning. Many studies on content-based medical image retrievals have been reported on. For retrieval of perceptually similar and diagnostically relevant images, incorporation of perceptual similarity data by radiologists has been suggested. In this paper, studies on image retrieval methods are reviewed with a special focus on quantification, utilization, and the evaluation of subjective similarities between pairs of images.

Convolutional neural networks and hash learning for feature extraction and of fast retrieval of pulmonary nodules

With a widespread use of digital imaging data in hospitals, the size of medical image repositories is increasing rapidly. This causes difficulty in managing and querying these large databases leading to the need of content based medical image retrieval (CBMIR) systems. A major challenge in CBMIR systems is the ?semantic gap? that exists between the low level visual information captured by imaging devices and high level semantic information perceived by the human. Using deep convolution neural network (CNN) to construct the CBMIR system can fully characterize the high level semantic features information for medical image retrieval. The existing network mostly used for the natural images can?t produce a good result directly applied to medical image. This paper used UNet method to preprocessing under the guidance of medical knowledge. Then, multi-scale receiving field convolution module is used to extract features of the segmented images with different sizes. Finally, encoded the features and used a coarse to fine search strategy with an average search accuracy of 0.73.

Retrieval of head–neck medical images using Gabor filter based on power-law transformation method and rank BHMT

This research aims to work on the specific medical domain. In this work, retrieval of the head–neck medical images from a database is discussed. Content-based medical image retrieval system (CBMIR) is used for retrieving the head–neck images. CBMIR is automatic and more efficient compared with the text-based approach. Shape and texture features are used for constructing feature vector. Texture feature is extracted using a modified Gabor filter based on power-law transformation method. Shape feature is extracted using rank BHMT (rank-order blur hit or miss transformation) method. Shape and texture features are combined to form a single feature vector. Threshold value very near to zero is used to retrieve images from the database. The proposed method is compared with log-Gabor filters and rank BHMT method. Combinations of modified Gabor filter with rank BHMT gave better performance than other methods.

A fully automatic end-to-end method for content-based image retrieval of CT scans with similar liver lesion annotations.

The goal of medical content-based image retrieval (M-CBIR) is to assist radiologists in the decision-making process by retrieving medical cases similar to a given image. One of the key interests of radiologists is lesions and their annotations, since the patient treatment depends on the lesion diagnosis. Therefore, a key feature of M-CBIR systems is the retrieval of scans with the most similar lesion annotations. To be of value, M-CBIR systems should be fully automatic to handle large case databases. We present a fully automatic end-to-end method for the retrieval of CT scans with similar liver lesion annotations. The input is a database of abdominal CT scans labeled with liver lesions, a query CT scan, and optionally one radiologist-specified lesion annotation of interest. The output is an ordered list of the database CT scans with the most similar liver lesion annotations. The method starts by automatically segmenting the liver in the scan. It then extracts a histogram-based features vector from the segmented region, learns the features' relative importance, and ranks the database scans according to the relative importance measure. The main advantages of our method are that it fully automates the end-to-end querying process, that it uses simple and efficient techniques that are scalable to large datasets, and that it produces quality retrieval results using an unannotated CT scan. Our experimental results on 9 CT queries on a dataset of 41 volumetric CT scans from the 2014 Image CLEF Liver Annotation Task yield an average retrieval accuracy (Normalized Discounted Cumulative Gain index) of 0.77 and 0.84 without/with annotation, respectively. Fully automatic end-to-end retrieval of similar cases based on image information alone, rather that on disease diagnosis, may help radiologists to better diagnose liver lesions.

An overview of approaches for content-based medical image retrieval

Medical imaging performs a vital role in the medical field as it provides important information on the internal body parts for the clinical analysis and medical intervention which enables physicians to diagnose and treat a variety of diseases. Nowadays the medical diagnosis is increasing at a very high rate, which results in the formation of a huge medical image database, and retrieving similar medical images from such a huge database is a very difficult task. A literature review of various methods for biomedical image indexing and retrieval is presented here. Over 140 contributions are included from the literature in this survey. And it is mainly concentrated on the methodology based on the visual representation of the medical images as content-based medical image retrieval (CBMIR) approaches retrieve similar medical images more efficiently as compared to text-based biomedical image retrieval approaches. It also delineates how various ideas were adopted from different computer science methodologies for developing CBMIR systems.

Content-based Medical Image Tetrieval for Liver CT Annotation

The increase number of medical image stored and saved every day presents a unique opportunity for content-based medical image retrieval (CBMIR) systems. In this paper, we propose content-based medical image retrieval for annotating liver CT scans images in order to generate a structured report. For that, we have used the Bidimentional Empirical Mode Decomposition (BEMD), and then we have applied Gabor wavelet transform to extract the mean and the standard deviation as features descriptors. Finally, a proposed similarity distance was employed to retrieve the most similar training images to the image query, and a majority voting scheme was used to select the answers for an unannotated image. We have used the IMAGECLEF 2015 annotation dataset and the obtained score was 88.9%.

A medical image retrieval method based on texture block coding tree

Content-based medical image retrieval (CBMIR) has been widely studied for computer aided diagnosis. Accurate and comprehensive retrieval results are effective to facilitate diagnosis and treatment. Texture is one of the most important features used in CBMIR. Most of existing methods utilize the distances between matching point pairs for texture similarity measurement. However, the distance based similarity measurements are of low tolerance to slight texture shifts, which result in an excessive sensitivity. Furthermore, with the increase of the number of texture points, their time complexity is in explosive growth. In this paper, a new medical image retrieval model is presented based on an iterative texture block coding tree. The corresponding methods for coarse-grained and fine-grained similarity matching are also proposed. Moreover, a multi-level index structure is designed to enhance the retrieval efficiency. Experimental results show that, our methods are of high efficiency and appropriate tolerance on slight shifts, and achieve a relative better retrieval performance in comparison of other existing methods.

Medical image retrieval using deep convolutional neural network

With a widespread use of digital imaging data in hospitals, the size of medical image repositories is increasing rapidly. This causes difficulty in managing and querying these large databases leading to the need of content based medical image retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low level visual information captured by imaging devices and high level semantic information perceived by human. The efficacy of such systems is more crucial in terms of feature representations that can characterize the high-level information completely. In this paper, we propose a framework of deep learning for CBMIR system by using deep convolutional neural network (CNN) that is trained for classification of medical images. An intermodal dataset that contains twenty-four classes and five modalities is used to train the network. The learned features and the classification results are used to retrieve medical images. For retrieval, best results are achieved when class based predictions are used. An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task. The proposed method is best suited to retrieve multimodal medical images for different body organs.

A HYBRID METHOD FOR MEDICAL IMAGES INDEXING USING PRINCIPAL COMPONENT ANALYSIS AND CDF BIORTHOGONAL WAVELET BASED ON LIFTING SCHEME

Hospitals and clinics produce a great number of medical images that are stored in large databases. Content-based medical image retrieval systems (CBMIRs) is one of the solutions to access rapidly and efficiently to these databases using their visual content. In this paper, we propose a new algorithm for medical image indexing. It consists on a combination of bio-orthogonal CDF wavelet transform (WT) based on lifting scheme and principal component analysis (PCA). We use this WT to decompose images, then we apply the PCA method to reduce the number of features and select the pertinent components which represent image signature. Finally, Euclidean distance is used to retrieve most similar image from databases when query image is submitted. We have tested our algorithm on the retinal image database. The results obtained by our algorithm are compared with several published methods cited in the literature and shows an efficiency of 80%, which is significantly higher and much faster than recent methods in CBMIRs domain.

Sparse Codebook Model of Local Structures for Retrieval of Focal Liver Lesions Using Multiphase Medical Images.

Characterization and individual trait analysis of the focal liver lesions (FLL) is a challenging task in medical image processing and clinical site. The character analysis of a unconfirmed FLL case would be expected to benefit greatly from the accumulated FLL cases with experts' analysis, which can be achieved by content-based medical image retrieval (CBMIR). CBMIR mainly includes discriminated feature extraction and similarity calculation procedures. Bag-of-Visual-Words (BoVW) (codebook-based model) has been proven to be effective for different classification and retrieval tasks. This study investigates an improved codebook model for the fined-grained medical image representation with the following three advantages: (1) instead of SIFT, we exploit the local patch (structure) as the local descriptor, which can retain all detailed information and is more suitable for the fine-grained medical image applications; (2) in order to more accurately approximate any local descriptor in coding procedure, the sparse coding method, instead of K-means algorithm, is employed for codebook learning and coded vector calculation; (3) we evaluate retrieval performance of focal liver lesions (FLL) using multiphase computed tomography (CT) scans, in which the proposed codebook model is separately learned for each phase. The effectiveness of the proposed method is confirmed by our experiments on FLL retrieval.