Hand-crafted Descriptors Research Articles

Investigation on deep learning for off-line handwritten Arabic character recognition

Handwritten character recognition is a system widely used in the modern world and it is still an important challenge. Traditional machine-learning techniques require careful engineering and considerable domain expertise to transform raw data into a feature vector from which the classifier could classify the input pattern. To cope with this problem, the popular Deep Convolutional Neural Networks (DCNN), introduced recently, have effectively replaced the hand-crafted descriptors with network features and have been shown to provide significantly better results than traditional methods. It is one of the fastest growing areas in machine learning, promising to reshape the future of artificial intelligence. However, the problem with deep learning is that it requires large datasets for training because of the huge number of parameters needed to be tuned by a learning algorithm. CNN model can be used in three different ways: (i) training the CNN from scratch; (ii) using the transfer learning strategy to leverage features from a pre-trained model on a larger dataset; and (iii) keeping the transfer learning strategy and fine-tune the weights of CNN architecture. In this work, we investigate the applicability of DCNN using transfer learning strategies on two datasets; a new expanded version of our recently proposed database for off-line isolated handwritten Arabic character, referred to as OIHACDB and AHCD. Our results showed satisfactory recognition accuracies and outperform all other prominent exiting methods in the field of Handwritten Arabic Character Recognition (HACR).

Centre symmetric quadruple pattern: A novel descriptor for facial image recognition and retrieval

Facial features are defined as the local relationships that exist amongst the pixels of a facial image. Hand-crafted descriptors identify the relationships of the pixels in the local neighborhood defined by the kernel. Kernel is a two dimensional matrix which is moved across the facial image. Distinctive information captured by the kernel with limited number of pixel achieves satisfactory recognition and retrieval accuracies on facial images taken under constrained environment (controlled variations in light, pose, expressions, and background). To achieve similar accuracies under unconstrained environment local neighborhood has to be increased, in order to encode more pixels. Increasing local neighborhood also increases the feature length of the descriptor. In this paper we propose a hand-crafted descriptor namely Centre Symmetric Quadruple Pattern (CSQP), which is structurally symmetric and encodes the facial asymmetry in quadruple space. The proposed descriptor efficiently encodes larger neighborhood with optimal number of binary bits. It has been shown using average entropy, computed over feature images encoded with the proposed descriptor, that the CSQP captures more meaningful information as compared to state of the art descriptors. The retrieval and recognition accuracies of the proposed descriptor has been compared with state of the art hand-crafted descriptors (CSLBP, CSLTP, LDP, LBP, SLBP and LDGP) on bench mark databases namely; LFW, Color-FERET, and CASIA-face-v5. Result analysis shows that the proposed descriptor performs well under controlled as well as uncontrolled variations in pose, illumination, background and expressions.

DeepCloud: Ground-Based Cloud Image Categorization Using Deep Convolutional Features

Accurate ground-based cloud image categorization is a critical but challenging task that has not been well addressed. One of the essential issues that affect the performance is to extract the representative visual features. Nearly all of the existing methods rely on the hand-crafted descriptors (e.g., local binary patterns, CENsus TRsansform hISTogram, and scale-invariant feature transform). Their limited discriminative power indeed leads to the unsatisfactory performance. To alleviate this, we propose “DeepCloud” as a novel cloud image feature extraction approach by resorting to the deep convolutional visual features. In the recent years, the deep convolutional neural network (CNN) has achieved the promising results in lots of computer vision and image understanding fields. Nevertheless, it has not been applied to cloud image classification yet. Thus, we actually pay the first effort to fill this blank. Since cloud image classification can be attributed to a multi-instance learning problem, simply employing the convolutional features within CNN cannot achieve the promising result. To address this, Fisher vector encoding is applied to executing the spatial feature aggregation and high-dimensional feature mapping on the raw deep convolutional features. Moreover, the hierarchical convolutional layers are used simultaneously to capture the fine textural characteristics and high-level semantic information in the unified manner. To further leverage the performance, a cloud pattern mining and selection method are also proposed. It targets at finding the discriminative local patterns to better distinguish the different kinds of clouds. The experiments on a challenging ground-based cloud image data set demonstrate the superiority of the proposition over the state-of-the-art methods.

HNIP: Compact Deep Invariant Representations for Video Matching, Localization, and Retrieval

With emerging demand for large-scale video analysis, MPEG initiated the compact descriptor for video analysis (CDVA) standardization in 2014. Beyond handcrafted descriptors adopted by the current MPEG-CDVA reference model, we study the problem of deep learned global descriptors for video matching, localization, and retrieval. First, inspired by a recent invariance theory, we propose a nested invariance pooling (NIP) method to derive compact deep global descriptors from convolutional neural networks (CNNs), by progressively encoding translation, scale, and rotation invariances into the pooled descriptors. Second, our empirical studies have shown that a sequence of well designed pooling moments (e.g., max or average) may drastically impact video matching performance, which motivates us to design hybrid pooling operations via NIP (HNIP). HNIP has further improved the discriminability of deep global descriptors. Third, the technical merits and performance improvements by combining deep and handcrafted descriptors are provided to better investigate the complementary effects. We evaluate the effectiveness of HNIP within the well-established MPEG-CDVA evaluation framework. The extensive experiments have demonstrated that HNIP outperforms the state-of-the-art deep and canonical handcrafted descriptors with significant mAP gains of 5.5% and 4.7%, respectively. In particular the combination of HNIP incorporated CNN descriptors and handcrafted global descriptors has significantly boosted the performance of CDVA core techniques with comparable descriptor size.

Aggregating Rich Hierarchical Features for Scene Classification in Remote Sensing Imagery

Scene classification is one of the most important issues in remote sensing image processing. To obtain a high discriminative feature representation for an image to be classified, traditional methods usually consider to densely accumulate hand-crafted low-level descriptors (e.g., scale-invariant feature transform) by feature encoding techniques. However, the performance is largely limited by the hand-crafted descriptors as they are not capable of describing the rich semantic information contained in various remote sensing images. To alleviate this problem, we propose a novel method to extract discriminative image features from the rich hierarchical information contained in convolutional neural networks (CNNs). Specifically, the low-level and middle-level intermediate convolutional features are, respectively, encoded by vector of locally aggregated descriptors (VLAD) and then reduced by principal component analysis to obtain hierarchical global features; meanwhile, the fully connected features are average pooled and subsequently normalized to form new global features. The proposed encoded mixed-resolution representation (EMR) is the concatenation of all the above-mentioned global features. Due to the usage of encoding strategies (VLAD and average pooling), our method can deal with images of different sizes. In addition, to reduce the computational consumption in the training stage, we directly extract EMR from VGG-VD and ResNet pretrained on the ImageNet dataset. We show in this paper that CNNs pretrained on the natural image dataset are more easily applied to the remote sensing dataset when the local structure similarity between two datasets is higher. Experimental evaluations on the UC-Merced and Brazilian Coffee Scenes datasets demonstrate that our method is superior to the state of the art.

Local quadruple pattern: A novel descriptor for facial image recognition and retrieval

In this paper a novel hand-crafted local quadruple pattern (LQPAT) is proposed for facial image recognition and retrieval. Most of the existing hand-crafted descriptors encode only a limited number of pixels in the local neighborhood. Under unconstrained environment the performance of these descriptors tends to degrade drastically. Major problem in increasing the local neighborhood is that, it also increases the feature length of the descriptor. The proposed descriptor tries to overcome these problems by defining an efficient encoding structure with optimal feature length. The proposed descriptor encodes relations amongst the neighbors in quadruple space. Two micro patterns are computed from the local relationships to form the descriptor. The retrieval and recognition accuracies of the proposed descriptor has been compared with state of the art hand crafted descriptors on bench mark databases namely; Caltech-face, LFW, Color-FERET, and CASIA-face-v5. Result analysis shows that the proposed descriptor performs well under uncontrolled variations.

Fine-Grained Recurrent Neural Networks for Automatic Prostate Segmentation in Ultrasound Images

Boundary incompleteness raises great challenges to automatic prostate segmentation in ultrasound images. Shape prior can provide strong guidance in estimating the missing boundary, but traditional shape models often suffer from hand-crafted descriptors and local information loss in the fitting procedure. In this paper, we attempt to address those issues with a novel framework. The proposed framework can seamlessly integrate feature extraction and shape prior exploring, and estimate the complete boundary with a sequential manner. Our framework is composed of three key modules. Firstly, we serialize the static 2D prostate ultrasound images into dynamic sequences and then predict prostate shapes by sequentially exploring shape priors. Intuitively, we propose to learn the shape prior with the biologically plausible Recurrent Neural Networks (RNNs). This module is corroborated to be effective in dealing with the boundary incompleteness. Secondly, to alleviate the bias caused by different serialization manners, we propose a multi-view fusion strategy to merge shape predictions obtained from different perspectives. Thirdly, we further implant the RNN core into a multiscale Auto-Context scheme to successively refine the details of the shape prediction map. With extensive validation on challenging prostate ultrasound images, our framework bridges severe boundary incompleteness and achieves the best performance in prostate boundary delineation when compared with several advanced methods. Additionally, our approach is general and can be extended to other medical image segmentation tasks, where boundary incompleteness is one of the main challenges.

A Comprehensive Study of Group Activity Recognition Methods in Video

Objectives: To provide comprehensive review of different group activity recognition methods, categorize them and provide path to new researcher in this domain. Methods/Statistical Analysis: Different methods of group activity recognition categorized and analyzed according to hand-crafted and learned feature descriptors. Pros and cons of each method are presented. Methods are analyzed in detailed by finding its local level features to global level feature descriptors used along with performance on benchmark dataset. Findings: Different models of group activity recognition are characterized as per the capabilities of the defined model considering individual pose of person, atomic activity of person, person-person interaction, person-group interaction, group-group interaction, uses of temporal information, and recognition of group activity frame wise or video wise. This comprehensive review provides brief information about group activity recognition methods and can be used as brief literature review to the researcher seeking the facts and findings in the field of computer vision in group activity recognition. Applications/Improvements: This reviews help in different applications of human activity analysis, mainly in group activity recognition and the models described here can be used in different applications such running or walking on pathways, waiting at public places, queuing in line in group and many more group activity applications for further enhancement.

Optimum Pipeline for Visual Terrain Classification Using Improved Bag of Visual Words and Fusion Methods

We propose an optimum pipeline and develop the hybrid representation to produce an effective and efficient visual terrain classification system. The bag of visual words (BOVW) framework has emerged as a promising approach and effective paradigm for visual terrain classification. The method includes four main steps: (1) feature extraction, (2) codebook generation, (3) feature coding, and (4) pooling and normalization. Recent researches have primarily focused on feature extraction in the development of new handcrafted descriptors that are specific to the visual terrain. However, the effects of other steps on visual terrain classification are still unknown. At the same time, fusion methods are often used to boost classification performance by exploring the complementarity of diverse features. We provide a comprehensive study of all steps in the BOVW framework and different fusion methods for visual terrain classification. Then, multiple approaches in each step and their effects are explored on the visual terrain dataset. Finally, the feature preprocessing technique, improved BOVW framework, and fusion method are used to construct an optimum pipeline for visual terrain classification. The hybrid representation developed by the optimum pipeline performs effectively and rapidly for visual terrain classification in the terrain dataset, outperforming those current methods. Furthermore, it is robust to diverse noises and illumination alterations.

Learning Multi-Instance Deep Discriminative Patterns for Image Classification.

Finding an effective and efficient representation is very important for image classification. The most common approach is to extract a set of local descriptors, and then aggregate them into a high-dimensional, more semantic feature vector, like unsupervised bag-of-features and weakly supervised part-based models. The latter one is usually more discriminative than the former due to the use of information from image labels. In this paper, we propose a weakly supervised strategy that using multi-instance learning (MIL) to learn discriminative patterns for image representation. Specially, we extend traditional multi-instance methods to explicitly learn more than one patterns in positive class, and find the "most positive" instance for each pattern. Furthermore, as the positiveness of instance is treated as a continuous variable, we can use stochastic gradient decent to maximize the margin between different patterns meanwhile considering MIL constraints. To make the learned patterns more discriminative, local descriptors extracted by deep convolutional neural networks are chosen instead of hand-crafted descriptors. Some experimental results are reported on several widely used benchmarks (Action 40, Caltech 101, Scene 15, MIT-indoor, SUN 397), showing that our method can achieve very remarkable performance.

Towards Reversal-Invariant Image Representation

State-of-the-art image classification approaches are mainly based on robust image representation, such as the bag-of-features (BoF) model or the convolutional neural network (CNN) architecture. In real applications, the orientation (left/right) of an image or an object might vary from sample to sample, whereas some handcrafted descriptors (e.g., SIFT) and network operations (e.g., convolution) are not reversal-invariant, leading to the unsatisfied stability of image features extracted from these models. To deal with, a popular solution is to augment the dataset by adding a left-right reversed copy for each image. This strategy improves the recognition accuracy to some extent, but also brings the price of almost doubled time and memory consumptions on both the training and testing stages. In this paper, we present an alternative solution based on designing reversal-invariant representation of local patterns, so that we can obtain the identical representation for an image and its left-right reversed copy. For the BoF model, we design a reversal-invariant version of SIFT descriptor named Max-SIFT, a generalized RIDE algorithm which can be applied to a large family of local descriptors. For the CNN architecture, we present a simple idea of generating reversal-invariant deep features (RI-Deep), and, inspired by which, design reversal-invariant convolution (RI-Conv) layers to increase the CNN capacity without increasing the model complexity. Experiments reveal consistent accuracy gain on various image classification tasks, including scene understanding, fine-grained object recognition, and large-scale visual recognition.

How could a subcellular image, or a painting by Van Gogh, be similar to a great white shark or to a pizza?

In this work, we propose an unorthodox approach for describing a given image. Each image is represented by a feature vector whose elements are the scores assigned to object classes by deep convolutional neural networks that were not related to those that built the given image classification problem. The deep neural networks are trained using 1000 classes; therefore, each image is described by 1000 scores, which are fed to a support vector machine. The proposed approach could be considered a transfer learning method, where, instead of repurposing the learned features to a second classification problem, we use the scores obtained by trained convolutional neural networks. Methods based on state of the art handcrafted descriptors, and the novel approach presented here are compared, together with selected ensembles of such methods. The fusion between a standard approach and the new unorthodox method boosts the performance of the standard approach. The Wilcoxon signed rank test is used to compare the different methods. The novel method is applied to 21 different datasets to demonstrate its generality. The MATLAB source code to replicate our experiments will be available at (https://www.dei.unipd.it/node/2357 +Pattern Recognition and Ensemble Classifiers).

Group-aware deep feature learning for facial age estimation

In this paper, we propose a group-aware deep feature learning (GA-DFL) approach for facial age estimation. Unlike most existing methods which utilize hand-crafted descriptors for face representation, our GA-DFL method learns a discriminative feature descriptor per image directly from raw pixels for face representation under the deep convolutional neural networks framework. Motivated by the fact that age labels are chronologically correlated and the facial aging datasets are usually lack of labeled data for each person in a long range of ages, we split ordinal ages into a set of discrete groups and learn deep feature transformations across age groups to project each face pair into the new feature space, where the intra-group variances of positive face pairs from the training set are minimized and the inter-group variances of negative face pairs are maximized, simultaneously. Moreover, we employ an overlapped coupled learning method to exploit the smoothness for adjacent age groups. To further enhance the discriminative capacity of face representation, we design a multi-path CNN approach to integrate the complementary information from multi-scale perspectives. Experimental results show that our approach achieves very competitive performance compared with most state-of-the-arts on three public face aging datasets that were captured under both controlled and uncontrolled environments.

Combining multiple features for color texture classification

The analysis of color and texture has a long history in image analysis and computer vision. These two properties are often considered as independent, even though they are strongly related in images of natural objects and materials. Correlation between color and texture information is especially relevant in the case of variable illumination, a condition that has a crucial impact on the effectiveness of most visual descriptors. We propose an ensemble of hand-crafted image descriptors designed to capture different aspects of color textures. We show that the use of these descriptors in a multiple classifiers framework makes it possible to achieve a very high classification accuracy in classifying texture images acquired under different lighting conditions. A powerful alternative to hand-crafted descriptors is represented by features obtained with deep learning methods. We also show how the proposed combining strategy hand-crafted and convolutional neural networks features can be used together to further improve the classification accuracy. Experimental results on a food database (raw food texture) demonstrate the effectiveness of the proposed strategy.

Mesh Convolutional Restricted Boltzmann Machines for Unsupervised Learning of Features With Structure Preservation on 3-D Meshes.

Discriminative features of 3-D meshes are significant to many 3-D shape analysis tasks. However, handcrafted descriptors and traditional unsupervised 3-D feature learning methods suffer from several significant weaknesses: 1) the extensive human intervention is involved; 2) the local and global structure information of 3-D meshes cannot be preserved, which is in fact an important source of discriminability; 3) the irregular vertex topology and arbitrary resolution of 3-D meshes do not allow the direct application of the popular deep learning models; 4) the orientation is ambiguous on the mesh surface; and 5) the effect of rigid and nonrigid transformations on 3-D meshes cannot be eliminated. As a remedy, we propose a deep learning model with a novel irregular model structure, called mesh convolutional restricted Boltzmann machines (MCRBMs). MCRBM aims to simultaneously learn structure-preserving local and global features from a novel raw representation, local function energy distribution. In addition, multiple MCRBMs can be stacked into a deeper model, called mesh convolutional deep belief networks (MCDBNs). MCDBN employs a novel local structure preserving convolution (LSPC) strategy to convolve the geometry and the local structure learned by the lower MCRBM to the upper MCRBM. LSPC facilitates resolving the challenging issue of the orientation ambiguity on the mesh surface in MCDBN. Experiments using the proposed MCRBM and MCDBN were conducted on three common aspects: global shape retrieval, partial shape retrieval, and shape correspondence. Results show that the features learned by the proposed methods outperform the other state-of-the-art 3-D shape features.

Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks.

Cerebral microbleeds (CMBs) are small haemorrhages nearby blood vessels. They have been recognized as important diagnostic biomarkers for many cerebrovascular diseases and cognitive dysfunctions. In current clinical routine, CMBs are manually labelled by radiologists but this procedure is laborious, time-consuming, and error prone. In this paper, we propose a novel automatic method to detect CMBs from magnetic resonance (MR) images by exploiting the 3D convolutional neural network (CNN). Compared with previous methods that employed either low-level hand-crafted descriptors or 2D CNNs, our method can take full advantage of spatial contextual information in MR volumes to extract more representative high-level features for CMBs, and hence achieve a much better detection accuracy. To further improve the detection performance while reducing the computational cost, we propose a cascaded framework under 3D CNNs for the task of CMB detection. We first exploit a 3D fully convolutional network (FCN) strategy to retrieve the candidates with high probabilities of being CMBs, and then apply a well-trained 3D CNN discrimination model to distinguish CMBs from hard mimics. Compared with traditional sliding window strategy, the proposed 3D FCN strategy can remove massive redundant computations and dramatically speed up the detection process. We constructed a large dataset with 320 volumetric MR scans and performed extensive experiments to validate the proposed method, which achieved a high sensitivity of 93.16% with an average number of 2.74 false positives per subject, outperforming previous methods using low-level descriptors or 2D CNNs by a significant margin. The proposed method, in principle, can be adapted to other biomarker detection tasks from volumetric medical data.

Query-Dependent Aesthetic Model With Deep Learning for Photo Quality Assessment

The automatic assessment of photo quality from an aesthetic perspective is a very challenging problem. Most existing research has predominantly focused on the learning of a universal aesthetic model based on hand-crafted visual descriptors . However, this research paradigm can achieve only limited success because 1) such hand-crafted descriptors cannot well preserve abstract aesthetic properties , and 2) such a universal model cannot always capture the full diversity of visual content. To address these challenges, we propose in this paper a novel query-dependent aesthetic model with deep learning for photo quality assessment. In our method, deep aesthetic abstractions are discovered from massive images , whereas the aesthetic assessment model is learned in a query- dependent manner. Our work addresses the first problem by learning mid-level aesthetic feature abstractions via powerful deep convolutional neural networks to automatically capture the underlying aesthetic characteristics of the massive training images . Regarding the second problem, because photographers tend to employ different rules of photography for capturing different images , the aesthetic model should also be query- dependent . Specifically, given an image to be assessed, we first identify which aesthetic model should be applied for this particular image. Then, we build a unique aesthetic model of this type to assess its aesthetic quality. We conducted extensive experiments on two large-scale datasets and demonstrated that the proposed query-dependent model equipped with learned deep aesthetic abstractions significantly and consistently outperforms state-of-the-art hand-crafted feature -based and universal model-based methods.

Automatic classification for field crop insects via multiple-task sparse representation and multiple-kernel learning

Classification of insect species of field crops such as corn, soybeans, wheat, and canola is more difficult than the generic object classification because of high appearance similarity among insect species. To improve the classification accuracy, we develop an insect recognition system using advanced multiple-task sparse representation and multiple-kernel learning (MKL) techniques. As different features of insect images contribute differently to the classification of insect species, the multiple-task sparse representation technique can combine multiple features of insect species to enhance the recognition performance. Instead of using hand-crafted descriptors, our idea of sparse-coding histograms is adopted to represent insect images so that raw features (e.g., color, shape, and texture) can be well quantified. Furthermore, the MKL method is proposed to fuse multiple features effectively. The proposed learning model can be optimized efficiently by jointly optimizing the kernel weights. Experimental results on 24 common pest species of field crops show that our proposed method performs well on the classification of insect species, and outperforms the state-of-the-art methods of the generic insect categorization.

Cost-Sensitive Local Binary Feature Learning for Facial Age Estimation.

In this paper, we propose a cost-sensitive local binary feature learning (CS-LBFL) method for facial age estimation. Unlike the conventional facial age estimation methods that employ hand-crafted descriptors or holistically learned descriptors for feature representation, our CS-LBFL method learns discriminative local features directly from raw pixels for face representation. Motivated by the fact that facial age estimation is a cost-sensitive computer vision problem and local binary features are more robust to illumination and expression variations than holistic features, we learn a series of hashing functions to project raw pixel values extracted from face patches into low-dimensional binary codes, where binary codes with similar chronological ages are projected as close as possible, and those with dissimilar chronological ages are projected as far as possible. Then, we pool and encode these local binary codes within each face image as a real-valued histogram feature for face representation. Moreover, we propose a cost-sensitive local binary multi-feature learning method to jointly learn multiple sets of hashing functions using face patches extracted from different scales to exploit complementary information. Our methods achieve competitive performance on four widely used face aging data sets.

A deep analysis on age estimation

The automatic estimation of age from face images is increasingly gaining attention, as it facilitates applications including advanced video surveillance, demographic statistics collection, customer profiling, or search optimization in large databases. Nevertheless, it becomes challenging to estimate age from uncontrollable environments, with insufficient and incomplete training data, dealing with strong person-specificity and high within-range variance. These difficulties have been recently addressed with complex and strongly hand-crafted descriptors, difficult to replicate and compare. This paper presents two novel approaches: first, a simple yet effective fusion of descriptors based on texture and local appearance; and second, a deep learning scheme for accurate age estimation. These methods have been evaluated under a diversity of settings, and the extensive experiments carried out on two large databases (MORPH and FRGC) demonstrate state-of-the-art results over previous work.