Content-based Medical Image Retrieval System Research Articles

Medical image retrieval using deep convolutional neural network

With a widespread use of digital imaging data in hospitals, the size of medical image repositories is increasing rapidly. This causes difficulty in managing and querying these large databases leading to the need of content based medical image retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low level visual information captured by imaging devices and high level semantic information perceived by human. The efficacy of such systems is more crucial in terms of feature representations that can characterize the high-level information completely. In this paper, we propose a framework of deep learning for CBMIR system by using deep convolutional neural network (CNN) that is trained for classification of medical images. An intermodal dataset that contains twenty-four classes and five modalities is used to train the network. The learned features and the classification results are used to retrieve medical images. For retrieval, best results are achieved when class based predictions are used. An average classification accuracy of 99.77% and a mean average precision of 0.69 is achieved for retrieval task. The proposed method is best suited to retrieve multimodal medical images for different body organs.

A HYBRID METHOD FOR MEDICAL IMAGES INDEXING USING PRINCIPAL COMPONENT ANALYSIS AND CDF BIORTHOGONAL WAVELET BASED ON LIFTING SCHEME

Hospitals and clinics produce a great number of medical images that are stored in large databases. Content-based medical image retrieval systems (CBMIRs) is one of the solutions to access rapidly and efficiently to these databases using their visual content. In this paper, we propose a new algorithm for medical image indexing. It consists on a combination of bio-orthogonal CDF wavelet transform (WT) based on lifting scheme and principal component analysis (PCA). We use this WT to decompose images, then we apply the PCA method to reduce the number of features and select the pertinent components which represent image signature. Finally, Euclidean distance is used to retrieve most similar image from databases when query image is submitted. We have tested our algorithm on the retinal image database. The results obtained by our algorithm are compared with several published methods cited in the literature and shows an efficiency of 80%, which is significantly higher and much faster than recent methods in CBMIRs domain.

Integration of global and local features based on Hybrid Similarity Matching Scheme for Medical Image Retrieval System

Similarity measure is a challenging task in Content-Based Medical Image Retrieval (CBMIR) systems and the matching scheme is designed to improve the retrieval performance. However, there are several major shortcomings with conventional approaches for a matching scheme which can extensively affect their application of Medical Image Retrieval (MIR). To overcome the issues, in this paper a Multi-Level Matching (MLM) method for MIR using hybrid feature similarity is proposed. Here, images are represented by multi-level features including local level and global level. The Colour and Edge Directivity Descriptor (CEDD) is used as a colour and edge-based descriptor. Speeded-Up Robust Features (SURF) and Local Binary Pattern (LBP) are used as a local descriptor. The hybrid of both global and local features yields enhanced retrieval accuracy, which is analysed over collected image databases. From the experiment, the proposed method achieves better accuracy value about 92%, which is higher than other methods.

An optimised multidimensional feature vector for automatic tumour detection and retrieving similar images for diagnosing clinically

High speed advancements in the medical technology are revolutionising the medication. The usage of Computer-Aided Diagnosis (CAD) systems plays a potential role in improving the quality of clinical radiotherapy care. The tumours are fatal in nature and may invade the other parts of the body. Precise diagnosis is essential for accurate treatment of the tumour. In that direction to make easy access in the future all medical images are registered in a medical image database. Content-Based Medical Image Retrieval (CBMIR) techniques attract greater interest to facilitate medical fields in retrieving the interested medical images. Consequently, based on this, we propose an intellectual three-step CBMIR system which retrieves corresponding images from a large medical database to support the radiologist in the diagnosis tumour images. The initial step detects the tumour in the image. Based on this information it retrieves the nearby similar images within the pre-classified database with the optimised futures. In the final step, the performance of the retrieval system is enhanced by adapting the subjective feedback method. The experimental results show that the proposed region-based multiple features and integrated with classifier and subjective feedback method yield better results than classical retrieval systems.

An optimised multidimensional feature vector for automatic tumour detection and retrieving similar images for diagnosing clinically

High speed advancements in the medical technology are revolutionising the medication. The usage of Computer-Aided Diagnosis (CAD) systems plays a potential role in improving the quality of clinical radiotherapy care. The tumours are fatal in nature and may invade the other parts of the body. Precise diagnosis is essential for accurate treatment of the tumour. In that direction to make easy access in the future all medical images are registered in a medical image database. Content-Based Medical Image Retrieval (CBMIR) techniques attract greater interest to facilitate medical fields in retrieving the interested medical images. Consequently, based on this, we propose an intellectual three-step CBMIR system which retrieves corresponding images from a large medical database to support the radiologist in the diagnosis tumour images. The initial step detects the tumour in the image. Based on this information it retrieves the nearby similar images within the pre-classified database with the optimised futures. In the final step, the performance of the retrieval system is enhanced by adapting the subjective feedback method. The experimental results show that the proposed region-based multiple features and integrated with classifier and subjective feedback method yield better results than classical retrieval systems.

Similarity measurement of lung masses for medical image retrieval using kernel based semisupervised distance metric.

To develop a new algorithm to measure the similarity between the query lung mass and reference lung mass data set for content-based medical image retrieval (CBMIR). A lung mass data set including 746 mass regions of interest (ROIs) was assembled. Among them, 375 ROIs depicted malignant lesions and 371 depicted benign lesions. Each mass ROI is represented by a vector of 26 texture features. A kernel function was employed to map the original data in input space to a feature space. In this space, a semisupervised distance metric was learned, which used differential scatter discriminant criterion to represent the semantic relevance, and the regularization term to represent the visual similarity. The learned distance metric can measure the similarity of the query mass and reference mass data set. The clustering accuracy is used to configure the parameters. The retrieval accuracy and classification accuracy are used as the performance assessment index. After configuring the parameters, a mean clustering accuracy of 0.87 can be achieved. For retrieval accuracy, our algorithm achieves better performance than other state-of-the-art retrieval algorithms when applying a leave-one-out validation method to the testing data set. For classification accuracy, the area under the ROC curve of our algorithm can be achieved as 0.941 ± 0.006. The running times of 346 query images with the proposed algorithm are 5.399 and 6.0 s, respectively. The study results demonstrated the proposed algorithm outperforms the compared algorithms, when taking the semantic relevant and visual similarity into account in kernel space. The algorithm can be used in a CBMIR system for a query mass to retrieve similarity masses, which can help doctors make better decisions.

An Efficient Content-Based Medical Image Retrieval Using Local Vector Pattern

This paper presents image feature extraction and matching framework for image retrieval used in medical image database management and disease diagnosis applications. Since the process of retrieval has facilitated one to find out the required medical image automatically, based on its content, the proposed system is called as Content-Based Medical Image Retrieval (CBMIR) system. The goal of CBMIR system is to enhance visual information analysis by increasing the overall search capabilities to physicians. The proposed system encompasses image feature extraction scheme using Local Vector Pattern (LVP) descriptor and image matching using histogram to retrieve relevant images from a MRI brain images database. The main characteristics of the MRI images are their poor contrast, sensitive to noise and large intensity variations. These challenges are tackled by implementing LVP which was proved as best facial feature extracting approach in the existing system with the first and second order derivatives to improve the recognition performance of the system. Ultimately, results were compared with existing local pattern descriptors such as Local Binary Pattern (LBP), Local Derivative Pattern (LDP) and Local Tetra Pattern (LTrP). We proved that an average recognition rate obtained for implementing LVP in medical image database is better than the other state-of-the-art local pattern descriptor techniques.

Risk Analysis for Pathological Changes in Pulmonary Parenchyma Based on Lung Computed Tomography Images.

The purpose of this study is to design a content-based medical image retrieval system, which helps excavate and assess pathological change of pulmonary parenchyma for risks analysis. A data set including lung computed tomography images obtained from 115 patients who experienced pathological changes in pulmonary parenchyma is used. Using morphological theory, images are preprocessed and decomposed into groups of pixel blocks (words), which construct vocabulary. A latent Dirichlet allocation (LDA) model is constructed to assess each image for risk analysis with the method of leave-one-out cross-validation. The precision and recall rate are used as the performance assessment criteria. The LDA model generates a relevance rank of retrieval results from high to low. From the top 50 images, precision of identical tissue is 0.76 ± 0.031 and precision of each attribute of pulmonary parenchyma range from 0.776 ± 0.043 to 0.984 ± 0.008. The study results demonstrate that the proposed LDA model is conductive to lung computed tomography image retrieval and has reliable efficacy on risk analysis about pathological changes of pulmonary parenchyma.

Endowing a Content-Based Medical Image Retrieval System with Perceptual Similarity Using Ensemble Strategy.

Content-based medical image retrieval (CBMIR) is a powerful resource to improve differential computer-aided diagnosis. The major problem with CBMIR applications is the semantic gap, a situation in which the system does not follow the users' sense of similarity. This gap can be bridged by the adequate modeling of similarity queries, which ultimately depends on the combination of feature extractor methods and distance functions. In this study, such combinations are referred to as perceptual parameters, as they impact on how images are compared. In a CBMIR, the perceptual parameters must be manually set by the users, which imposes a heavy burden on the specialists; otherwise, the system will follow a predefined sense of similarity. This paper presents a novel approach to endow a CBMIR with a proper sense of similarity, in which the system defines the perceptual parameter depending on the query element. The method employs ensemble strategy, where an extreme learning machine acts as a meta-learner and identifies the most suitable perceptual parameter according to a given query image. This parameter defines the search space for the similarity query that retrieves the most similar images. An instance-based learning classifier labels the query image following the query result set. As the concept implementation, we integrated the approach into a mammogram CBMIR. For each query image, the resulting tool provided a complete second opinion, including lesion class, system certainty degree, and set of most similar images. Extensive experiments on a large mammogram dataset showed that our proposal achieved a hit ratio up to 10% higher than the traditional CBMIR approach without requiring external parameters from the users. Our database-driven solution was also up to 25% faster than content retrieval traditional approaches.

User-oriented evaluation of a medical image retrieval system for radiologists

PurposeThis article reports the user-oriented evaluation of a text- and content-based medical image retrieval system. User tests with radiologists using a search system for images in the medical literature are presented. The goal of the tests is to assess the usability of the system, identify system and interface aspects that need improvement and useful additions. Another objective is to investigate the system's added value to radiology information retrieval. The study provides an insight into required specifications and potential shortcomings of medical image retrieval systems through a concrete methodology for conducting user tests. MethodsUser tests with a working image retrieval system of images from the biomedical literature were performed in an iterative manner, where each iteration had the participants perform radiology information seeking tasks and then refining the system as well as the user study design itself. During these tasks the interaction of the users with the system was monitored, usability aspects were measured, retrieval success rates recorded and feedback was collected through survey forms. ResultsIn total, 16 radiologists participated in the user tests. The success rates in finding relevant information were on average 87% and 78% for image and case retrieval tasks, respectively. The average time for a successful search was below 3min in both cases. Users felt quickly comfortable with the novel techniques and tools (after 5 to 15min), such as content-based image retrieval and relevance feedback. User satisfaction measures show a very positive attitude toward the system's functionalities while the user feedback helped identifying the system's weak points. The participants proposed several potentially useful new functionalities, such as filtering by imaging modality and search for articles using image examples. ConclusionThe iterative character of the evaluation helped to obtain diverse and detailed feedback on all system aspects. Radiologists are quickly familiar with the functionalities but have several comments on desired functionalities. The analysis of the results can potentially assist system refinement for future medical information retrieval systems. Moreover, the methodology presented as well as the discussion on the limitations and challenges of such studies can be useful for user-oriented medical image retrieval evaluation, as user-oriented evaluation of interactive system is still only rarely performed. Such interactive evaluations can be limited in effort if done iteratively and can give many insights for developing better systems.

Lung segmentation on standard and mobile chest radiographs using oriented Gaussian derivatives filter.

BackgroundUnsupervised lung segmentation method is one of the mandatory processes in order to develop a Content Based Medical Image Retrieval System (CBMIRS) of CXR. The purpose of the study is to present a robust solution for lung segmentation of standard and mobile chest radiographs using fully automated unsupervised method.MethodsThe novel method is based on oriented Gaussian derivatives filter with seven orientations, combined with Fuzzy C-Means (FCM) clustering and thresholding to refine the lung region. In addition, a new algorithm to automatically generate a threshold value for each Gaussian response is also proposed. The algorithms are applied to both PA and AP chest radiographs from both public JSRT dataset and our private datasets from collaborative hospital. Two pre-processing blocks are introduced to standardize the images from different machines. Comparisons with the previous works found in the literature on JSRT dataset shows that our method gives a reasonably good result. We also compare our algorithm with other unsupervised methods to provide fairly comparative measures on the performances for all datasets.ResultsPerformance measures (accuracy, F-score, precision, sensitivity and specificity) for the segmentation of lung in public JSRT dataset are above 0.90 except for the overlap measure is 0.87. The standard deviations for all measures are very low, from 0.01 to 0.06. The overlap measure for the private image database is 0.81 (images from standard machine) and 0.69 (images from two mobile machines). The algorithm is fully automated and fast, with the average execution time of 12.5 s for 512 by 512 pixels resolution.ConclusionsOur proposed method is fully automated, unsupervised, with no training or learning stage is necessary to segment the lungs taken using both a standard machine and two different mobile machines. The proposed pre-processing blocks are significantly useful to standardize the radiographs from mobile machines. The algorithm gives good performance measures, robust, and fast for the application of the CBMIRS.

Design and development of a content-based medical image retrieval system for spine vertebrae irregularity.

BackgroundContent-based medical image retrieval (CBMIR) system enables medical practitioners to perform fast diagnosis through quantitative assessment of the visual information of various modalities.MethodsIn this paper, a more robust CBMIR system that deals with both cervical and lumbar vertebrae irregularity is afforded. It comprises three main phases, namely modelling, indexing and retrieval of the vertebrae image. The main tasks in the modelling phase are to improve and enhance the visibility of the x-ray image for better segmentation results using active shape model (ASM). The segmented vertebral fractures are then characterized in the indexing phase using region-based fracture characterization (RB-FC) and contour-based fracture characterization (CB-FC). Upon a query, the characterized features are compared to the query image. Effectiveness of the retrieval phase is determined by its retrieval, thus, we propose an integration of the predictor model based cross validation neural network (PMCVNN) and similarity matching (SM) in this stage. The PMCVNN task is to identify the correct vertebral irregularity class through classification allowing the SM process to be more efficient. Retrieval performance between the proposed and the standard retrieval architectures are then compared using retrieval precision (Pr@M) and average group score (AGS) measures.ResultsExperimental results show that the new integrated retrieval architecture performs better than those of the standard CBMIR architecture with retrieval results of cervical (AGS > 87%) and lumbar (AGS > 82%) datasets.ConclusionsThe proposed CBMIR architecture shows encouraging results with high Pr@M accuracy. As a result, images from the same visualization class are returned for further used by the medical personnel.

Optimal query-based relevance feedback in medical image retrieval using score fusion-based classification.

In this paper, a new content-based medical image retrieval (CBMIR) framework using an effective classification method and a novel relevance feedback (RF) approach are proposed. For a large-scale database with diverse collection of different modalities, query image classification is inevitable due to firstly, reducing the computational complexity and secondly, increasing influence of data fusion by removing unimportant data and focus on the more valuable information. Hence, we find probability distribution of classes in the database using Gaussian mixture model (GMM) for each feature descriptor and then using the fusion of obtained scores from the dependency probabilities, the most relevant clusters are identified for a given query. Afterwards, visual similarity of query image and images in relevant clusters are calculated. This method is performed separately on all feature descriptors, and then the results are fused together using feature similarity ranking level fusion algorithm. In the RF level, we propose a new approach to find the optimal queries based on relevant images. The main idea is based on density function estimation of positive images and strategy of moving toward the aggregation of estimated density function. The proposed framework has been evaluated on ImageCLEF 2005 database consisting of 10,000 medical X-ray images of 57 semantic classes. The experimental results show that compared with the existing CBMIR systems, our framework obtains the acceptable performance both in the image classification and in the image retrieval by RF.

Facilitating medical information search using Google Glass connected to a content-based medical image retrieval system.

Wearable computing devices are starting to change the way users interact with computers and the Internet. Among them, Google Glass includes a small screen located in front of the right eye, a camera filming in front of the user and a small computing unit. Google Glass has the advantage to provide online services while allowing the user to perform tasks with his/her hands. These augmented glasses uncover many useful applications, also in the medical domain. For example, Google Glass can easily provide video conference between medical doctors to discuss a live case. Using these glasses can also facilitate medical information search by allowing the access of a large amount of annotated medical cases during a consultation in a non-disruptive fashion for medical staff. In this paper, we developed a Google Glass application able to take a photo and send it to a medical image retrieval system along with keywords in order to retrieve similar cases. As a preliminary assessment of the usability of the application, we tested the application under three conditions (images of the skin; printed CT scans and MRI images; and CT and MRI images acquired directly from an LCD screen) to explore whether using Google Glass affects the accuracy of the results returned by the medical image retrieval system. The preliminary results show that despite minor problems due to the relative stability of the Google Glass, images can be sent to and processed by the medical image retrieval system and similar images are returned to the user, potentially helping in the decision making process.

Content-based image retrieval in medical domains: An overview of methods, issues and systems

Content-based image retrieval (CBIR) is one of the most interesting research areas of computer vision in recent years. CBIR uses feature comparison to search, browse, and retrieve images from large databases. It is considered as the main research topic in medical fields because of the daily increase in the number of recorded medical images. CBIR can help physicians diagnose various types of diseases as well as medical references, teaching purposes , training and research. This paper introduces the main concepts, stages, and methods of CBIR systems; explores the main types of features that can be extracted from images; and identifies the techniques to evaluate the systems. It also reviews and evaluates the advantages and disadvantages of several existing commercial and academic content-based medical image retrieval systems. Finally, the paper introduces suggestions for the future of medical domains to solve the discussed disadvantages. It proposes the technique of automatic medical image annotation/classification to enhance the accuracy of CBIR systems.

Medical Image Retrieval: A Multimodal Approach.

Medical imaging is becoming a vital component of war on cancer. Tremendous amounts of medical image data are captured and recorded in a digital format during cancer care and cancer research. Facing such an unprecedented volume of image data with heterogeneous image modalities, it is necessary to develop effective and efficient content-based medical image retrieval systems for cancer clinical practice and research. While substantial progress has been made in different areas of content-based image retrieval (CBIR) research, direct applications of existing CBIR techniques to the medical images produced unsatisfactory results, because of the unique characteristics of medical images. In this paper, we develop a new multimodal medical image retrieval approach based on the recent advances in the statistical graphic model and deep learning. Specifically, we first investigate a new extended probabilistic Latent Semantic Analysis model to integrate the visual and textual information from medical images to bridge the semantic gap. We then develop a new deep Boltzmann machine-based multimodal learning model to learn the joint density model from multimodal information in order to derive the missing modality. Experimental results with large volume of real-world medical images have shown that our new approach is a promising solution for the next-generation medical imaging indexing and retrieval system.

Designing user interfaces to enhance human interpretation of medical content-based image retrieval: application to PET-CT images

Content-based image retrieval (CBIR) in medicine has been demonstrated to improve evidence-based diagnosis, education, and teaching. However, the low clinical adoption of CBIR is partially because the focus of most studies has been the development of feature extraction and similarity measurement algorithms with limited work on facilitating better understanding of the similarity between complex volumetric and multi-modality medical images. In this paper, we present a method for defining user interfaces (UIs) that enable effective human user interpretation of retrieved images. We derived a set of visualisation and interaction requirements based on the characteristics of modern volumetric medical images. We implemented a UI that visualised multiple views of a single image, displayed abstractions of image data, and provided access to supplementary non-image data. We also defined interactions for refining the search and visually indicating the similarities between images. We applied the UI for the retrieval of multi-modality positron emission tomography and computed tomography (PET-CT) images. We conducted a user survey to evaluate the capabilities of our UI. Our proposed method obtained a high rating ( ≥ 4 out of 5) in the majority of survey questions. In particular, the survey responses indicated the UI presented all the information necessary to understand the retrieved images, and did so in an intuitive manner. Our proposed UI design improved the ability of users to interpret and understand the similarity between retrieved PET-CT images. The implementation of CBIR UIs designed to assist human interpretation could facilitate wider adoption of medical CBIR systems.

Medical X-ray Image Hierarchical Classification Using a Merging and Splitting Scheme in Feature Space

Due to the daily mass production and the widespread variation of medical X-ray images, it is necessary to classify these for searching and retrieving proposes, especially for content-based medical image retrieval systems. In this paper, a medical X-ray image hierarchical classification structure based on a novel merging and splitting scheme and using shape and texture features is proposed. In the first level of the proposed structure, to improve the classification performance, similar classes with regard to shape contents are grouped based on merging measures and shape features into the general overlapped classes. In the next levels of this structure, the overlapped classes split in smaller classes based on the classification performance of combination of shape and texture features or texture features only. Ultimately, in the last levels, this procedure is also continued forming all the classes, separately. Moreover, to optimize the feature vector in the proposed structure, we use orthogonal forward selection algorithm according to Mahalanobis class separability measure as a feature selection and reduction algorithm. In other words, according to the complexity and inter-class distance of each class, a sub-space of the feature space is selected in each level and then a supervised merging and splitting scheme is applied to form the hierarchical classification. The proposed structure is evaluated on a database consisting of 2158 medical X-ray images of 18 classes (IMAGECLEF 2005 database) and accuracy rate of 93.6% in the last level of the hierarchical structure for an 18-class classification problem is obtained.

An Approach of Multiscale Complexity in Texture Analysis of Lymphomas

Complexity is considered with high interest to texture analysis, such as the application of lymphoma inspection. Previously, multiscale entropy was proposed to account for the multiple time scales inherent in complex biological signals, and complexity texture descriptors were considered for content-based medical image retrieval system. However, in the past, these complexity texture descriptors were derived by linear algorithms of Fourier and wavelet transforms. In this study, we introduce a new approach of multiscale complexity (MSC) for texture analysis of lymphomas based on a nonlinear parameter of entropy. An extended algorithm of 2-D entropy was used in the calculations of complexities for the coarse-grained images of different scale factors. Furthermore, five parameters are defined to characterize 58 images of reactive lymphadenopahty and five categories of lymphomas using complexity on three scales and the changes of complexity between two successive scales. MSC analysis performed well to figure out the statistical differences of paired comparisons picked from reactive lymphadenopahty and five categories of lymphomas.

Ontology of Gaps in Content-Based Image Retrieval

Content-based image retrieval (CBIR) is a promising technology to enrich the core functionality of picture archiving and communication systems (PACS). CBIR has a potential for making a strong impact in diagnostics, research, and education. Research as reported in the scientific literature, however, has not made significant inroads as medical CBIR applications incorporated into routine clinical medicine or medical research. The cause is often attributed (without supporting analysis) to the inability of these applications in overcoming the "semantic gap." The semantic gap divides the high-level scene understanding and interpretation available with human cognitive capabilities from the low-level pixel analysis of computers, based on mathematical processing and artificial intelligence methods. In this paper, we suggest a more systematic and comprehensive view of the concept of "gaps" in medical CBIR research. In particular, we define an ontology of 14 gaps that addresses the image content and features, as well as system performance and usability. In addition to these gaps, we identify seven system characteristics that impact CBIR applicability and performance. The framework we have created can be used a posteriori to compare medical CBIR systems and approaches for specific biomedical image domains and goals and a priori during the design phase of a medical CBIR application, as the systematic analysis of gaps provides detailed insight in system comparison and helps to direct future research.