Large-scale Image Database Research Articles

Deep Convolutional Hashing for Low-Dimensional Binary Embedding of Histopathological Images.

Compact binary representations of histopa-thology images using hashing methods provide efficient approximate nearest neighbor search for direct visual query in large-scale databases. They can be utilized to measure the probability of the abnormality of the query image based on the retrieved similar cases, thereby providing support for medical diagnosis. They also allow for efficient managing of large-scale image databases because of a low storage requirement. However, the effectiveness of binary representations heavily relies on the visual descriptors that represent the semantic information in the histopathological images. Traditional approaches with hand-crafted visual descriptors might fail due to significant variations in image appearance. Recently, deep learning architectures provide promising solutions to address this problem using effective semantic representations. In this paper, we propose a deep convolutional hashing method that can be trained "point-wise" to simultaneously learn both semantic and binary representations of histopathological images. Specifically, we propose a convolutional neural network that introduces a latent binary encoding (LBE) layer for low-dimensional feature embedding to learn binary codes. We design a joint optimization objective function that encourages the network to learn discriminative representations from the label information, and reduce the gap between the real-valued low-dimensional embedded features and desired binary values. The binary encoding for new images can be obtained by forward propagating through the network and quantizing the output of the LBE layer. Experimental results on a large-scale histopathological image dataset demonstrate the effectiveness of the proposed method.

Selective deep ensemble for instance retrieval

In public security systems, visual instance retrieval has an explosive growing requirement, especially for large-scale image or video databases. Due to its wide range of applications in surveillance scenario, this paper aims at the retrieval tasks centered around ‘vehicle’ and ‘pedestrian’ targets. Many previous CNN-based methods have not exploited the ensemble abilities of different models, which achieve limited accuracy since a certain kind of deep architecture is not comprehensive. On the other hand, some features in the original deep representation are useless for retrieval tasks, while the attention-aware compact representation will be much more efficient and effective. To address the above problems, we propose a Selective Deep Ensemble (SDE) framework to combine various models and features in a complementary way, inspired by the attention mechanism. It is demonstrated that a large improvement can be acquired with slight increase on computation cost. Finally, we evaluate the performance on three public instance-retrieval datasets, VehicleID, VeRi and Market-1501, outperforming state-of-the-art methods by a large margin.

TOP-SIFT: the selected SIFT descriptor based on dictionary learning

The large amount of SIFT descriptors in an image and the high dimensionality of SIFT descriptor have made problems for the large-scale image database in terms of speed and scalability. In this paper, we present a descriptor selection algorithm based on dictionary learning to remove the redundant features and reserve only a small set of features, which we refer to as TOP-SIFTs. During the experiment, we discovered the inner relativity between the problem of descriptor selection and dictionary learning in sparse representation, and then turned our problem into dictionary learning. We designed a new dictionary learning method to adapt our problem and employed the simulated annealing algorithm to obtain the optimal solution. During the process of learning, we added the sparsity constraint and spatial distribution characteristic of SIFT points. And lastly selected the small representative feature set with good spatial distribution. Compared with the earlier methods, our method is neither relying on the database nor losing important information, and the experiments have shown that our algorithm can save memory space a lot and increase time efficiency while maintaining the accuracy as well.

Vision-Based Parking-Slot Detection: A Benchmark and A Learning-Based Approach

Recent years have witnessed a growing interest in developing automatic parking systems in the field of intelligent vehicles. However, how to effectively and efficiently locating parking-slots using a vision-based system is still an unresolved issue. Even more seriously, there is no publicly available labeled benchmark dataset for tuning and testing parking-slot detection algorithms. In this paper, we attempt to fill the above-mentioned research gaps to some extent and our contributions are twofold. Firstly, to facilitate the study of vision-based parking-slot detection, a large-scale parking-slot image database is established. This database comprises 8600 surround-view images collected from typical indoor and outdoor parking sites. For each image in this database, the marking-points and parking-slots are carefully labeled. Such a database can serve as a benchmark to design and validate parking-slot detection algorithms. Secondly, a learning-based parking-slot detection approach, namely P S D L , is proposed. Using P S D L , given a surround-view image, the marking-points will be detected first and then the valid parking-slots can be inferred. The efficacy and efficiency of P S D L have been corroborated on our database. It is expected that P S D L can serve as a baseline when the other researchers develop more sophisticated methods.

Reduced-Reference Image Quality Assessment in Free-Energy Principle and Sparse Representation

The free-energy principle in recent studies of brain theory and neuroscience models the perception and understanding of the outside scene as an active inference process, in which the brain tries to account for the visual scene with an internal generative model. Specifically, with the internal generative model, the brain yields corresponding predictions for its encountered visual scenes. Then, the discrepancy between the visual input and its brain prediction should be closely related to the quality of perceptions. On the other hand, sparse representation has been evidenced to resemble the strategy of the primary visual cortex in the brain for representing natural images. With the strong neurobiological support for sparse representation, in this paper, we approximate the internal generative model with sparse representation and propose an image quality metric accordingly, which is named FSI (free-energy principle and sparse representation-based index for image quality assessment). In FSI, the reference and distorted images are, respectively, predicted by the sparse representation at first. Then, the difference between the entropies of the prediction discrepancies is defined to measure the image quality. Experimental results on four large-scale image databases confirm the effectiveness of the FSI and its superiority over representative image quality assessment methods. The FSI belongs to reduced-reference methods, and it only needs a single number from the reference image for quality estimation.

CrowdOLR: Toward Object Location Recognition With Crowdsourced Fingerprints Using Smartphones

Recognizing object location by taking a photo with smartphones is useful for many location-based services. However, start-of-the-art technologies for both localization and location recognition have difficulty in achieving satisfactory performance. Moreover, it is a challenging issue to construct and maintain a large-scale image database for existing visual-based location recognition systems. To cope with these issues, we introduce CrowdOLR, a crowdsourcing based object location recognition system, which collects one location image together with various rich sensory data (GPS coordinates, azimuth angle, tilt angle, etc.) as a fingerprint of a location query and matches it to a fingerprint database crowdsourced from users’ smartphones. We designed a simple and efficient user action mode and proposed a series of fingerprint extracting, searching, and matching methods, so that CrowdOLR satisfies five desirable properties: high recognition accuracy, user friendliness, quick response, no/little site survey, and timely update. We implemented CrowdOLR and collected 8100 location fingerprints of 162 objects for performance evaluation. Extensive experiments demonstrate that CrowdOLR achieves promising results in various complicated and realistic scenarios.

Large-scale semantic web image retrieval using bimodal deep learning techniques

Semantic web image retrieval is useful to end-users for semantic image searches over the Internet. This paper aims to develop image retrieval techniques for large-scale web image databases. An advanced retrieval system, termed Multi-concept Retrieval using Bimodal Deep Learning (MRBDL), is proposed and implemented using Convolutional Neural Networks (CNNs) which can effectively capture semantic correlations between a visual image and its free contextual tags. Different from existing approaches using multiple and independent concepts in a query, MRBDL considers multiple concepts as a holistic scene for retrieval model learning. In particular, we first use a bimodal CNN to train a holistic scene classifier in two modalities, and then semantic correlations of the sub-concepts included in the images are leveraged to boost holistic scene recognition. The predicted semantic scores obtained from holistic scene classifier are combined with complementary information on web images to improve the retrieval performance. Experiments have been carried out over two publicly available web image databases. The results show that our proposed approach performs favorably compared with several other state-of-the-art methods.

SSDH: Semi-Supervised Deep Hashing for Large Scale Image Retrieval

Hashing methods have been widely used for efficient similarity retrieval on large scale image database. Traditional hashing methods learn hash functions to generate binary codes from hand-crafted features, which achieve limited accuracy since the hand-crafted features cannot optimally represent the image content and preserve the semantic similarity. Recently, several deep hashing methods have shown better performance because the deep architectures generate more discriminative feature representations. However, these deep hashing methods are mainly designed for supervised scenarios, which only exploit the semantic similarity information, but ignore the underlying data structures. In this paper, we propose the semi-supervised deep hashing (SSDH) approach, to perform more effective hash function learning by simultaneously preserving semantic similarity and underlying data structures. The main contributions are as follows: (1) We propose a semi-supervised loss to jointly minimize the empirical error on labeled data, as well as the embedding error on both labeled and unlabeled data, which can preserve the semantic similarity and capture the meaningful neighbors on the underlying data structures for effective hashing. (2) A semi-supervised deep hashing network is designed to extensively exploit both labeled and unlabeled data, in which we propose an online graph construction method to benefit from the evolving deep features during training to better capture semantic neighbors. To the best of our knowledge, the proposed deep network is the first deep hashing method that can perform hash code learning and feature learning simultaneously in a semi-supervised fashion. Experimental results on 5 widely-used datasets show that our proposed approach outperforms the state-of-the-art hashing methods.

Deep convolutional neural network with transfer learning for rectum toxicity prediction in cervical cancer radiotherapy: a feasibility study

Better understanding of the dose-toxicity relationship is critical for safe dose escalation to improve local control in late-stage cervical cancer radiotherapy. In this study, we introduced a convolutional neural network (CNN) model to analyze rectum dose distribution and predict rectum toxicity. Forty-two cervical cancer patients treated with combined external beam radiotherapy (EBRT) and brachytherapy (BT) were retrospectively collected, including twelve toxicity patients and thirty non-toxicity patients. We adopted a transfer learning strategy to overcome the limited patient data issue. A 16-layers CNN developed by the visual geometry group (VGG-16) of the University of Oxford was pre-trained on a large-scale natural image database, ImageNet, and fine-tuned with patient rectum surface dose maps (RSDMs), which were accumulated EBRT + BT doses on the unfolded rectum surface. We used the adaptive synthetic sampling approach and the data augmentation method to address the two challenges, data imbalance and data scarcity. The gradient-weighted class activation maps (Grad-CAM) were also generated to highlight the discriminative regions on the RSDM along with the prediction model. We compare different CNN coefficients fine-tuning strategies, and compare the predictive performance using the traditional dose volume parameters, e.g. D0.1/1/2cc, and the texture features extracted from the RSDM. Satisfactory prediction performance was achieved with the proposed scheme, and we found that the mean Grad-CAM over the toxicity patient group has geometric consistence of distribution with the statistical analysis result, which indicates possible rectum toxicity location. The evaluation results have demonstrated the feasibility of building a CNN-based rectum dose-toxicity prediction model with transfer learning for cervical cancer radiotherapy.

Multiple kernel scale invariant feature transform and cross indexing for image search and retrieval

ABSTRACTImage retrieval on large-scale image databases has attained more attention, in which mapping features into binary codes are showing great advancement. This paper proposes a new approach for image retrieval, Multiple Kernel SIFT (MKSIFT) that extracts the features from the pre-processed input image. It utilizes the steps of SIFT to extract the feature points. MKSIFT computes the keypoint descriptor with the introduction of exponential and tangential kernels, in which the weights assigned to the kernels are selected by Particle Swarm-Fractional Bacterial foraging optimization (PS-FBFO) algorithm. Moreover, it performs a cross-indexed image search by converting the feature points of MKSIFT into binary codes. The performance of MKSIFT+ Cross indexing is compared with that of SIFT, BSIFT, BSIFT+ Cross indexing, in which the proposed MKSIFT+ Cross indexing shows maximum performance. The experimental results evaluated the parameter precision, recall and F-measure provided maximum mean precision of 0.89793, recall of 0.8625, and F-measure of 0.87716.

LOCAL BINARY PATTERN WITH SUPPORT VECTOR MACHINE TO ENHANCE IMAGE RETRIEVAL

Image search techniques were not generally based on visual features but on the textual annotation of images. Images were firstly annotated with text and then searched using a text-based approach from traditional database management systems which is time consuming and difficult to manage. To overcome this problem, content based image retrieval is introduced which is becoming the hottest research area these days due to vast range of real time applications such as crime prevention, photograph archives, medical diagnosis, geographical information and remote sensing system etc. The CBIR system consist of various phases to extract and match the features and search the images from the large scale image databases on the basis of visual contents such as color, shape and texture according to the user's interest. During retrieval, features and descriptors of the query image are compared to those of the images in the database in order to rank each image according to its distance to the query. In our research work, a hybrid combination of SVM (support vector machine) and LBP (Local Binary Pattern) is applied to retrieve the images from the database. A Support Vector Machine (SVM) is a discriminative classifier formally defined by a separating hyper plane. Local Binary Pattern (LBP) is one of the techniques used in image classification based on texture. This operator is simple and very effective, which labels the image pixels based on their neighboring and consider the result being a binary number.

Location-Based Parallel Tag Completion for Geo-Tagged Social Image Retrieval

Having benefited from tremendous growth of user-generated content, social annotated tags get higher importance in the organization and retrieval of large-scale image databases on Online Sharing Websites (OSW). To obtain high-quality tags from existing community contributed tags with missing information and noise, tag-based annotation or recommendation methods have been proposed for performance promotion of tag prediction. While images from OSW contain rich social attributes, they have not taken full advantage of rich social attributes and auxiliary information associated with social images to construct global information completion models. In this article, beyond the image-tag relation, we take full advantage of the ubiquitous GPS locations and image-user relationship to enhance the accuracy of tag prediction and improve the computational efficiency. For GPS locations, we define the popular geo-locations where people tend to take more images as Points of Interests (POI), which are discovered by mean shift approach. For image-user relationship, we integrate a localized prior constraint, expecting the completed tag sub-matrix in each POI to maintain consistency with users’ tagging behaviors. Based on these two key issues, we propose a unified tag matrix completion framework, which learns the image-tag relation within each POI. To solve the optimization problem, an efficient proximal sub-gradient descent algorithm is designed. The model optimization can be easily parallelized and distributed to learn the tag sub-matrix for each POI. Extensive experimental results reveal that the learned tag sub-matrix of each POI reflects the major trend of users’ tagging results with respect to different POIs and users, and the parallel learning process provides strong support for processing large-scale online image databases. To fit the response time requirement and storage limitations of Tag-based Image Retrieval (TBIR) on mobile devices, we introduce Asymmetric Locality Sensitive Hashing (ALSH) to reduce the time cost and meanwhile improve the efficiency of retrieval.

No-Reference Quality Assessment of Tone-Mapped HDR Pictures.

Being able to automatically predict digital picture quality, as perceived by human observers, has become important in many applications where humans are the ultimate consumers of displayed visual information. Standard dynamic range (SDR) images provide 8 b/color/pixel. High dynamic range (HDR) images, which are usually created from multiple exposures of the same scene, can provide 16 or 32 b/color/pixel, but must be tonemapped to SDR for display on standard monitors. Multi-exposure fusion techniques bypass HDR creation, by fusing the exposure stack directly to SDR format while aiming for aesthetically pleasing luminance and color distributions. Here, we describe a new no-reference image quality assessment (NR IQA) model for HDR pictures that is based on standard measurements of the bandpass and on newly conceived differential natural scene statistics (NSS) of HDR pictures. We derive an algorithm from the model which we call the HDR IMAGE GRADient-based Evaluator. NSS models have previously been used to devise NR IQA models that effectively predict the subjective quality of SDR images, but they perform significantly worse on tonemapped HDR content. Toward ameliorating this we make here the following contributions: 1) we design HDR picture NR IQA models and algorithms using both standard space-domain NSS features as well as novel HDR-specific gradient-based features that significantly elevate prediction performance; 2) we validate the proposed models on a large-scale crowdsourced HDR image database; and 3) we demonstrate that the proposed models also perform well on legacy natural SDR images. The software is available at: http://live.ece.utexas.edu/research/Quality/higradeRelease.zip.

Visual instance mining from the graph perspective

In this paper, we address the problem of visual instance mining, which is to automatically discover frequently appearing visual instances from a large collection of images. We propose a scalable mining method by leveraging the graph structure with images as vertices. Different from most existing approaches that focus on either instance-level similarities or image-level context properties, our method captures both information. In the proposed framework, the instance-level information is integrated during the construction of a sparse instance graph based on the similarity between augmented local features, while the image-level context is explored with a greedy breadth-first search algorithm to discover clusters of visual instances from the graph. This framework can tackle the challenges brought by small visual instances, diverse intra-class variations, as well as noise in large-scale image databases. To further improve the robustness, we integrate two techniques into the basic framework. First, to better cope with the increasing noise of large databases, weak geometric consistency is adopted to efficiently combine the geometric information of local matches into the construction of the instance graph. Second, we propose the layout embedding algorithm, which leverages the algorithm originally designed for graph visualization to fully explore the image database structure. The proposed method was evaluated on four annotated data sets with different characteristics, and experimental results showed the superiority over state-of-the-art algorithms on all data sets. We also applied our framework on a one-million Flickr data set and proved its scalability.

Scene Categorization Through Using Objects Represented by Deep Features

Objects in scenes are thought to be important for scene recognition. In this paper, we propose to utilize scene-specific objects represented by deep features for scene categorization. Our approach combines benefits of deep learning and Latent Support Vector Machine (LSVM) to train a set of scene-specific object models for each scene category. Specifically, we first use deep Convolutional Neural Networks (CNNs) pre-trained on the large-scale object-centric image database ImageNet to learn rich object features and a large number of general object concepts. Then, the pre-trained CNNs is adopted to extract features from images in the target dataset, and initialize the learning of scene-specific object models for each scene category. After initialization, the scene-specific object models are obtained by alternating between searching over the most representative and discriminative regions of images in the target dataset and training linear SVM classifiers based on obtained region features. As a result, for each scene category a set of object models that are representative and discriminative can be acquired. We use them to perform scene categorization. In addition, to utilize global structure information of scenes, we use another CNNs pre-trained on the large-scale scene-centric database Places to capture structure information of scene images. By combining objects and structure information for scene categorization, we show superior performances to state-of-the-art approaches on three public datasets, i.e. MIT-indoor, UIUC-sports and SUN. Experiment results demonstrated the effectiveness of the proposed method.

Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features.

Dreaming is generally thought to be generated by spontaneous brain activity during sleep with patterns common to waking experience. This view is supported by a recent study demonstrating that dreamed objects can be predicted from brain activity during sleep using statistical decoders trained with stimulus-induced brain activity. However, it remains unclear whether and how visual image features associated with dreamed objects are represented in the brain. In this study, we used a deep neural network (DNN) model for object recognition as a proxy for hierarchical visual feature representation, and DNN features for dreamed objects were analyzed with brain decoding of fMRI data collected during dreaming. The decoders were first trained with stimulus-induced brain activity labeled with the feature values of the stimulus image from multiple DNN layers. The decoders were then used to decode DNN features from the dream fMRI data, and the decoded features were compared with the averaged features of each object category calculated from a large-scale image database. We found that the feature values decoded from the dream fMRI data positively correlated with those associated with dreamed object categories at mid- to high-level DNN layers. Using the decoded features, the dreamed object category could be identified at above-chance levels by matching them to the averaged features for candidate categories. The results suggest that dreaming recruits hierarchical visual feature representations associated with objects, which may support phenomenal aspects of dream experience.

Large scale image aided navigation

Currently, most airborne platforms use an Inertial Measurement Unit (IMU) to provide estimates of aircraft position, velocity, and attitude. Errors in the IMU result in a position solution that drifts over time. In order to counteract IMU drift, platforms typically integrate measurements from Global Navigation Satellite Systems (GNSS). However, many military and civilian scenarios encounter situations where GNSS may be unavailable due to intentional jamming, or when operating in challenging environments. Many solutions have been proposed that use images from an airborne camera to match features against a reference image to compute global position. These techniques require a relatively accurate prior estimate of position to converge. This leads to a situation where platforms being denied GNSS pass a threshold of position error due to IMU drift, from which current image-aided navigation system cannot recover. This paper presents a novel localization algorithm inspired by fusing state of the art Content Based Image Retrieval techniques with nonparametric Bayesian estimation strategies. We outline the framework for performing a search of a large-scale image database in order to provide a rough position estimate to an airborne platform. These techniques are analogous to solving the “kidnapped-robot” problem, applied to airborne platforms. This framework is evaluated using actual data collected from an airborne camera and navigation system.

Object segmentation by saliency-seeded and spatial-weighted region merging

In this paper, we present a region merging-based method for object segmentation in natural images. The method consists of three separate steps: (1) initial over-segmentation such that pixels in each region are as homogeneous as possible and therefore likely to be from the same object; (2) saliency-seeded interaction to provide proper prior input to guide the segmentation; (3) region merging by an introduced maximal spatially weighted similarity (MSWS) criterion. Saliency-seeded interaction can well reflect the human intention but does not require any manual user editing, which makes our method applicable to increasingly large-scale image databases. The MSWS criterion takes into account both the color similarity and spatial distance of the candidate regions for merging, which allows the region merging-based method to achieve better performance. Extensive experiments show that our method can reliably and automatically segment the objects from a great variety of natural images.

Growing random forest on deep convolutional neural networks for scene categorization

Breakthrough performances have been achieved in computer vision by utilizing deep neural networks. In this paper we propose to use random forest to classify image representations obtained by concatenating multiple layers of learned features of deep convolutional neural networks for scene classification. Specifically, we first use deep convolutional neural networks pre-trained on the large-scale image database Places to extract features from scene images. Then, we concatenate multiple layers of features of the deep neural networks as image representations. After that, we use random forest as the classifier for scene classification. Moreover, to reduce feature redundancy in image representations we derived a novel feature selection method for selecting features that are suitable for random forest classification. Extensive experiments are conducted on two benchmark datasets, i.e. MIT-Indoor and UIUC-Sports. Obtained results demonstrated the effectiveness of the proposed method. The contributions of the paper are as follows. First, by extracting multiple layers of deep neural networks, we can explore more information of image contents for determining their categories. Second, we proposed a novel feature selection method that can be used to reduce redundancy in features obtained by deep neural networks for classification based on random forest. In particular, since deep learning methods can be used to augment expert systems by having the systems essentially training themselves, and the proposed framework is general, which can be easily extended to other intelligent systems that utilize deep learning methods, the proposed method provide a potential way for improving performances of other expert and intelligent systems.

DEVELOPMENT OF CONTENT BASED IMAGE RETRIEVAL SYSTEM USING NEURAL NETWORK & MULTI-RESOLUTION ANALYSIS

INTERNATIONAL JOURNAL OF ENGINEERING SCIENCES & RESEARCH TECHNOLOGY DEVELOPMENT OF CONTENT BASED IMAGE RETRIEVAL SYSTEM USING NEURAL NETWORK & MULTI-RESOLUTION ANALYSIS Jitendra Singh, Prof. Kailash Patidar, Mr. Gaurav Saxena M.Tech Scholar, Dept. of SE, SSSUTMS, Sehore, M.P., India Professor& Head, Dept. of CSE, SSSUTMS, Sehore, M.P., India Assistant Professor, Dept. of CSE, SSSUTMS, Sehore, M.P., India DOI: 10.5281/zenodo.61481 ABSTRACT In a content-based image retrieval system, the image features that effectively represent the image contents in a database. CBIR is the technique which uses visual contents to search images from large scale image databases according to the user’s interest. In the term content refers to shape, texture color, that can be derived from the image. In this paper an image retrieval technique using neural network (NN) in MATLAB with the help of Wavelet transform and Gabor filter features is contemplated. In the proposed system, mean and standard deviation of the images are calculated later to the filtering process of the images using Gabor filter.