High-level Image Representation Research Articles

Multi-View Graph Fusion for Semi-Supervised Learning: Application to Image-Based Face Beauty Prediction

Facial Beauty Prediction (FBP) is an important visual recognition problem to evaluate the attractiveness of faces according to human perception. Most existing FBP methods are based on supervised solutions using geometric or deep features. Semi-supervised learning for FBP is an almost unexplored research area. In this work, we propose a graph-based semi-supervised method in which multiple graphs are constructed to find the appropriate graph representation of the face images (with and without scores). The proposed method combines both geometric and deep feature-based graphs to produce a high-level representation of face images instead of using a single face descriptor and also improves the discriminative ability of graph-based score propagation methods. In addition to the data graph, our proposed approach fuses an additional graph adaptively built on the predicted beauty values. Experimental results on the SCUTFBP-5500 facial beauty dataset demonstrate the superiority of the proposed algorithm compared to other state-of-the-art methods.

A Deep Convolutional Neural Network Framework for Lung Cancer Detection

Exact identification of pulmonary nodules with high sensitivity and specificity is basic for programmed lung malignancy analysis from CT scans. In fact, many deep learning-based algorithms gain incredible ground for improving the exactness of nodule recognition; the high false positive rate is yet a difficult issue which restricted the programmed determination. We propose a novel customized Deep Convolutional Neural Network (DCNN) architecture for learning high-level image representation to achieve high classification accuracy with low variance in medical image binary classification tasks. Moreover, a High Sensitivity and Specificity system is introduced to eliminate the erroneously recognized nodule competitors by following the appearance changes in consistent CT slices of every nodule. The proposed structure is assessed on the open Kaggle Data Science Bowl (KDSB17) challenge dataset. Our strategy can precisely distinguish lung nodules at high sensitivity and specificity and accomplishes 95 % sensitivity.

Multilabel Annotation of Multispectral Remote Sensing Images using Error-Correcting Output Codes and Most Ambiguous Examples

This paper presents a novel framework for multilabel classification of multispectral remote sensing images using error-correcting output codes. Starting with a set of primary class labels, the proposed framework consists in transforming the multiclass problem into multiple binary learning subtasks. The distributed output representations of these binary learners are then transformed into primary class labels. In order to train robust binary classifiers on a reduced annotated dataset, the learning process is iterative and involves determining most ambiguous examples, which are included in the training set at each iteration. As part of the semantic image recognition process, two categories of high-level image representations are proposed for the feature extraction part. First, deep convolutional neural networks are used to form high-level representations of the images. Second, we test our classification framework with a bag-of-visual words model based on the scale invariant feature transform, used in combination with color descriptors. In the first case, we propose the usage of pretrained state-of-the-art deep learning models that cancel the need to estimate model parameters of complex architectures, whereas, in the second case, a dictionary of visual words must be determined from the training set. Experiments are conducted on GeoEye-1 and Sentinel-2 images and the results show the effectiveness of the proposed approach toward a multilabel classification, when compared to other methods.

Supervised and Unsupervised End-to-End Deep Learning for Gene Ontology Classification of Neural In Situ Hybridization Images.

In recent years, large datasets of high-resolution mammalian neural images have become available, which has prompted active research on the analysis of gene expression data. Traditional image processing methods are typically applied for learning functional representations of genes, based on their expressions in these brain images. In this paper, we describe a novel end-to-end deep learning-based method for generating compact representations of in situ hybridization (ISH) images, which are invariant-to-translation. In contrast to traditional image processing methods, our method relies, instead, on deep convolutional denoising autoencoders (CDAE) for processing raw pixel inputs, and generating the desired compact image representations. We provide an in-depth description of our deep learning-based approach, and present extensive experimental results, demonstrating that representations extracted by CDAE can help learn features of functional gene ontology categories for their classification in a highly accurate manner. Our methods improve the previous state-of-the-art classification rate (Liscovitch, et al.) from an average AUC of 0.92 to 0.997, i.e., it achieves 96% reduction in error rate. Furthermore, the representation vectors generated due to our method are more compact in comparison to previous state-of-the-art methods, allowing for a more efficient high-level representation of images. These results are obtained with significantly downsampled images in comparison to the original high-resolution ones, further underscoring the robustness of our proposed method.

An Attribute-Based High-Level Image Representation for Scene Classification

Scene classification is increasingly popular due to its extensive usage in many real-world applications such as object detection, image retrieval, and so on. Traditionally, the low-level hand-crafted image representations are adopted to describe the scene images. However, they usually fail to detect semantic features of visual concepts, especially in handling complex scenes. In this paper, we propose a novel high-level image representation which utilizes image attributes as features for scene classification. More specifically, the attributes of each image are firstly extracted by a deep convolution neural network (CNN), which is trained to be a multi-label classifier by minimizing an element-wise logistic loss function. The process of generating attributes can reduce the “semantic gap” between the low-level feature representation and the high level scene meaning. Based on the attributes, we then build a system to discover semantically meaningful descriptions of the scene classes. Extensive experiments on four large-scale scene classification datasets show that our proposed algorithm considerably outperforms other state-of-the-art methods.

An Emotion-Embedded Visual Attention Model for Dimensional Emotion Context Learning

Dimensional emotion recognition has attracted more and more researchers' attention from various fields including psychology, cognition, and computer science. In this paper, we propose an emotion-embedded visual attention model (EVAM) to learn emotion context information for predicting affective dimension values from video sequences. First, deep CNN is used to generate a high-level representation of the raw face images. Second, a visual attention model based on the gated recurrent unit (GRU) is employed to learn the context information of the feature sequences from face features. Third, the k-means algorithm is adapted to embed previous emotion into attention model to produce more robust time series predictions, which emphasize the influence of previous emotion on current effective prediction. In this paper, all experiments are carried out on database AVEC 2016 and AVEC 2017. The experimental results validate the efficiency of our method, and competitive results are obtained.

Computer-Assisted Decision Support System in Pulmonary Cancer detection and stage classification on CT images

Pulmonary cancer is considered as one of the major causes of death worldwide. For the detection of lung cancer, computer-assisted diagnosis (CADx) systems have been designed. Internet-of-Things (IoT) has enabled ubiquitous internet access to biomedical datasets and techniques; in result, the progress in CADx is significant. Unlike the conventional CADx, deep learning techniques have the basic advantage of an automatic exploitation feature as they have the ability to learn mid and high level image representations. We proposed a Computer-Assisted Decision Support System in Pulmonary Cancer by using the novel deep learning based model and metastasis information obtained from MBAN (Medical Body Area Network). The proposed model, DFCNet, is based on the deep fully convolutional neural network (FCNN) which is used for classification of each detected pulmonary nodule into four lung cancer stages. The performance of proposed work is evaluated on different datasets with varying scan conditions. Comparison of proposed classifier is done with the existing CNN techniques. Overall accuracy of CNN and DFCNet was 77.6% and 84.58%, respectively. Experimental results illustrate the effectiveness of proposed method for the detection and classification of lung cancer nodules. These results demonstrate the potential for the proposed technique in helping the radiologists in improving nodule detection accuracy with efficiency.

A new image classification method using CNN transfer learning and web data augmentation

Since Convolutional Neural Network (CNN) won the image classification competition 202 (ILSVRC12), a lot of attention has been paid to deep layer CNN study. The success of CNN is attributed to its superior multi-scale high-level image representations as opposed to hand-engineering low-level features. However, estimating millions of parameters of a deep CNN requires a large number of annotated samples, which currently prevents many superior deep CNNs (such as AlexNet, VGG, ResNet) being applied to problems with limited training data. To address this problem, a novel two-phase method combining CNN transfer learning and web data augmentation is proposed. With our method, the useful feature presentation of pre-trained network can be efficiently transferred to target task, and the original dataset can be augmented with the most valuable Internet images for classification. Our method not only greatly reduces the requirement of a large training data, but also effectively expand the training dataset. Both of method features contribute to the considerable over-fitting reduction of deep CNNs on small dataset. In addition, we successfully apply Bayesian optimization to solve the tuff problem, hyper-parameter tuning, in network fine-tuning. Our solution is applied to six public small datasets. Extensive experiments show that, comparing to traditional methods, our solution can assist the popular deep CNNs to achieve better performance. Particularly, ResNet can outperform all the state-of-the-art models on six small datasets. The experiment results prove that the proposed solution will be the great tool for dealing with practice problems which are related to use deep CNNs on small dataset.

Exploring Web Images to Enhance Skin Disease Analysis Under A Computer Vision Framework.

To benefit the skin care, this paper aims to design an automatic and effective visual analysis framework, with the expectation of recognizing the skin disease from a given image conveying the disease affected surface. This task is nontrivial, since it is hard to collect sufficient well-labeled samples. To address such problem, we present a novel transfer learning model, which is able to incorporate external knowledge obtained from the rich and relevant Web images contributed by grassroots. In particular, we first construct a target domain by crawling a small set of images from vertical and professional dermatological websites. We then construct a source domain by collecting a large set of skin disease related images from commercial search engines. To reinforce the learning performance in the target domain, we initially build a learning model in the target domain, and then seamlessly leverage the training samples in the source domain to enhance this learning model. The distribution gap between these two domains are bridged by a linear combination of Gaussian kernels. Instead of training models with low-level features, we resort to deep models to learn the succinct, invariant, and high-level image representations. Different from previous efforts that focus on a few types of skin diseases with a small and confidential set of images generated from hospitals, this paper targets at thousands of commonly seen skin diseases with publicly accessible Web images. Hence the proposed model is easily repeatable by other researchers and extendable to other disease types. Extensive experiments on a real-world dataset have demonstrated the superiority of our proposed method over the state-of-the-art competitors.

A pooled Object Bank descriptor for image scene classification

Object Bank (OB) is a high-level image representation encoding semantic and spacial information, and has superior performance in scene classification tasks. However, the dimensionality of OB feature is high, which demands massive computation. Existing dimensionality reduction methods for OB are incapable of achieving both high classification accuracy and substantial dimensionality reduction simultaneously. In order to solve this problem, we propose a threshold value filter pooling method to avoid noise accumulation in histogram-pooling and represent more useful information than max-pooling. We also propose a Matthew effect normalization method to highlight the useful information, and thus boost the performance of OB-based image scene classification. Finally, we apply these two methods in a dimensionality reduction framework to simplify OB representation and construct more proper descriptors, and thus achieve both dimensionality reduction and classification accuracy increase. We evaluated our framework on three real-world datasets, namely, event dataset UIUC-Sports, natural scene dataset LabelMe, and mixture dataset 15-Scenes. The classification results demonstrate that our framework not only obtains accuracies similar to or higher than the original OB representation, but also reduces the dimensionality significantly. The computational complexity analysis shows that it can reduce the time complexity of classification. Therefore, our framework can improve OB-based image scene classification through both computational complexity reduction and accuracy increase.

Beyond Object Proposals: Random Crop Pooling for Multi-Label Image Recognition.

Learning high-level image representations using object proposals has achieved remarkable success in multi-label image recognition. However, most object proposals provide merely coarse information about the objects, and only carefully selected proposals can be helpful for boosting the performance of multi-label image recognition. In this paper, we propose an object-proposal-free framework for multi-label image recognition: random crop pooling (RCP). Basically, RCP performs stochastic scaling and cropping over images before feeding them to a standard convolutional neural network, which works quite well with a max-pooling operation for recognizing the complex contents of multi-label images. To better fit the multi-label image recognition task, we further develop a new loss function-the dynamic weighted Euclidean loss-for the training of the deep network. Our RCP approach is amazingly simple yet effective. It can achieve significantly better image recognition performance than the approaches using object proposals. Moreover, our adapted network can be easily trained in an end-to-end manner. Extensive experiments are conducted on two representative multi-label image recognition data sets (i.e., PASCAL VOC 2007 and PASCAL VOC 2012), and the results clearly demonstrate the superiority of our approach.

Change detection: the role of low-level versus high-level image representations

Change detection: the role of low-level versus high-level image representations

A deep bag-of-features model for the classification of melanomas in dermoscopy images.

Deep learning and unsupervised feature learning have received great attention in past years for their ability to transform input data into high level representations using machine learning techniques. Such interest has been growing steadily in the field of medical image diagnosis, particularly in melanoma classification. In this paper, a novel application of deep learning (stacked sparse auto-encoders) is presented for skin lesion classification task. The stacked sparse auto-encoder discovers latent information features in input images (pixel intensities). These high-level features are subsequently fed into a classifier for classifying dermoscopy images. In addition, we proposed a new deep neural network architecture based on bag-of-features (BoF) model, which learns high-level image representation and maps images into BoF space. Then, we examine how using this deep representation of BoF, compared with pixel intensities of images, can improve the classification accuracy. The proposed method is evaluated on a test set of 244 skin images. To test the performance of the proposed method, the area under the receiver operating characteristics curve (AUC) is utilized. The proposed method is found to achieve 95% accuracy.

Chinese Herbal Medicine Image Recognition and Retrieval by Convolutional Neural Network.

Chinese herbal medicine image recognition and retrieval have great potential of practical applications. Several previous studies have focused on the recognition with hand-crafted image features, but there are two limitations in them. Firstly, most of these hand-crafted features are low-level image representation, which is easily affected by noise and background. Secondly, the medicine images are very clean without any backgrounds, which makes it difficult to use in practical applications. Therefore, designing high-level image representation for recognition and retrieval in real world medicine images is facing a great challenge. Inspired by the recent progress of deep learning in computer vision, we realize that deep learning methods may provide robust medicine image representation. In this paper, we propose to use the Convolutional Neural Network (CNN) for Chinese herbal medicine image recognition and retrieval. For the recognition problem, we use the softmax loss to optimize the recognition network; then for the retrieval problem, we fine-tune the recognition network by adding a triplet loss to search for the most similar medicine images. To evaluate our method, we construct a public database of herbal medicine images with cluttered backgrounds, which has in total 5523 images with 95 popular Chinese medicine categories. Experimental results show that our method can achieve the average recognition precision of 71% and the average retrieval precision of 53% over all the 95 medicine categories, which are quite promising given the fact that the real world images have multiple pieces of occluded herbal and cluttered backgrounds. Besides, our proposed method achieves the state-of-the-art performance by improving previous studies with a large margin.

A context-aware semantic modeling framework for efficient image retrieval

In recent years, high-level image representation is gaining popularity in image classification and retrieval tasks. This paper proposes an efficient scheme known as semantic context model to derive high-level image descriptors well suited for the retrieval operation. Semantic context model uses an undirected graphical model based formulation which jointly exploits low-level visual features and contextual information for classifying local image blocks into some predefined concept classes. Contextual information involves concept co-occurrences and their spatial correlation statistics. More expressive potential functions are introduced to capture the structural dependencies among various semantic concepts. The proposed framework proceeds in three steps. Initially, optimal values of model parameters that impose spatial consistency of concept labels among local image blocks are learned from the training data. Then, the semantics associated with the constituent blocks of an unseen image are inferred using an improved message-passing algorithm. Finally, a compact but discriminative image signature is derived by integrating the frequency of occurrence of various regional semantics. Experimental results on various benchmark datasets show that semantic context model can effectively resolve local ambiguities and consequently improve concept recognition performance in complex images. Moreover, the retrieval efficiency of the new semantics based image feature is found to be much better than state-of-the-art approaches.

Object Knowledge Shapes Properties of Early Feature-Detectors by Top-Down Modulation

Classic models of visual perception assume that early stages of information-processing are carried out by arrays of specialized and static feature-detectors that operate independently of high-level representations of a stimulus. Current developments in computational vision research have challenged this notion, hypothesizing that feedback connections from higher- to lower-level stages of processing shape the properties of feature detectors to optimize their performance. Evidence to support this hypothesis is difficult to obtain since any experimental design trying to test it has to ensure that sensory stimulation of early detectors remains identical while high-level representations of the stimulus are manipulated. Here, we used two-tone images of objects in a psychophysical procedure to achieve this and demonstrate a significant influence of high-level object knowledge on response properties of early edge-detectors. Two-tone images feature stimulus regions that are physically homogenous but are experienced as an object contour (somewhat similar to illusory contours in Kanizsa figures) once the observer is provided with relevant image information. We embedded edge-elements in these areas that were consistent or inconsistent with the high-level contour representation and measured absolute contrast detection-thresholds for these elements before and after providing relevant object knowledge. Results indicate that prior knowledge facilitates absolute contrast-threshold for edge elements that are consistent with the high-level representation of a stimulus but has no effects on contour-inconsistent edge-elements. The experimental design of the study ensured that this pattern of results is unlikely to be due to uncertainty reduction about target orientation, a notion supported by a simulation based on a neurophysiologically-plausible population-code. Together, these findings provide clear evidence that early feature-detectors do not operate independently of visual cognition. Rather response properties of early information-processing units are shaped by top-down modulation from high-level image representations. Meeting abstract presented at VSS 2015.

Image classification and annotation based on robust regularized coding

In recent years, sparse coding has been widely applied to construct high-level image representation in computer vision applications. However, one of major deficiencies of sparse coding is that it fails to capture spatial context in the data. Similar descriptors may be quantized into different visual words during feature quantization process. In this paper, we propose a novel coding scheme called robust regularized coding (RRC), which fully exploits the geometrical information among local descriptors to significantly boost the discriminating capability of the resultant features. More specifically, both locality constraint and smoothness constraint terms with respect to RRC codes are incorporated into the objective function to preserve the local invariance of RRC codes. Besides, to scale up to larger databases, a novel online learning algorithm with no hyperparameter tuning is proposed to incrementally update the codebook. The obtained RRC codes are then employed to represent images for classification and annotation tasks in our experiments. We also propose an effective reconstruction-based image annotation algorithm to propagate the labels of training images to test image by multi-label linear embedding. The experimental results extensively evaluated over several benchmarking datasets demonstrate our approach can achieve significant performance improvements with respect to the state of the arts.

Learning object-to-class kernels for scene classification.

High-level image representations have drawn increasing attention in visual recognition, e.g., scene classification, since the invention of the object bank. The object bank represents an image as a response map of a large number of pretrained object detectors and has achieved superior performance for visual recognition. In this paper, based on the object bank representation, we propose the object-to-class (O2C) distances to model scene images. In particular, four variants of O2C distances are presented, and with the O2C distances, we can represent the images using the object bank by lower-dimensional but more discriminative spaces, called distance spaces, which are spanned by the O2C distances. Due to the explicit computation of O2C distances based on the object bank, the obtained representations can possess more semantic meanings. To combine the discriminant ability of the O2C distances to all scene classes, we further propose to kernalize the distance representation for the final classification. We have conducted extensive experiments on four benchmark data sets, UIUC-Sports, Scene-15, MIT Indoor, and Caltech-101, which demonstrate that the proposed approaches can significantly improve the original object bank approach and achieve the state-of-the-art performance.

Learning Object Groups for Scene Recognition

Scene recognition is an important task for many computer vision and robotics applications. Recent progress in high-level object-based image representation has shown superior performance on scene classification tasks. In this work, we make an observation that groups of objects tend to co-occur frequently in a scene. We therefore propose to a novel framework that automatically learns object groups, and use them to build an image representation for scene recognition tasks. We model each object group as a template that explicitly encodes the spatial configurations of objects. To encourage the informativeness and discriminability, we learn the object group templates in a sparse filtering framework. Experiment results show that our object group representation could achieve state-of-the-art performance for both scene discovery and scene classification tasks.

Online glocal transfer for automatic figure-ground segmentation.

This paper addresses the problem of automatic figure-ground segmentation, which aims at automatically segmenting out all foreground objects from background. The underlying idea of this approach is to transfer segmentation masks of globally and locally (glocally) similar exemplars into the query image. For this purpose, we propose a novel high-level image representation method named as object-oriented descriptor. Using this descriptor, a set of exemplar images glocally similar to the query image is retrieved. Then, using over-segmented regions of these retrieved exemplars, a discriminative classifier is learned on-the-fly and subsequently used to predict foreground probability for the query image. Finally, the optimal segmentation is obtained by combining the online prediction with typical energy optimization of Markov random field. The proposed approach has been extensively evaluated on three datasets, including Pascal VOC 2010, VOC 2011 segmentation challenges, and iCoseg dataset. Experiments show that the proposed approach outperforms state-of-the-art methods and has the potential to segment large-scale images containing unknown objects, which never appear in the exemplar images.