Popular Convolutional Neural Networks Research Articles

Driver Gaze Zone Estimation Using Convolutional Neural Networks: A General Framework and Ablative Analysis

Driver gaze has been shown to be an excellent surrogate for driver attention in intelligent vehicles. With the recent surge of highly autonomous vehicles, driver gaze can be useful for determining the handoff time to a human driver. While there has been significant improvement in personalized driver gaze zone estimation systems, a generalized system which is invariant to different subjects, perspectives, and scales is still lacking. We take a step toward this generalized system using convolutional neural networks (CNNs). We finetune four popular CNN architectures for this task, and provide extensive comparisons of their outputs. We additionally experiment with different input image patches, and also examine how the image size affects performance. For training and testing the networks, we collect a large naturalistic driving dataset comprising of 11 long drives, driven by ten subjects in two different cars. Our best performing model achieves an accuracy of 95.18% during cross-subject testing, outperforming current state-of-the-art techniques for this task. Finally, we evaluate our best performing model on the publicly available Columbia gaze dataset comprising of images from 56 subjects with varying head pose and gaze directions. Without any training, our model successfully encodes the different gaze directions on this diverse dataset, demonstrating good generalization capabilities.

Accelerating Convolutional Neural Network With FFT on Embedded Hardware

Fueled by ImageNet Large Scale Visual Recognition Challenge and Common Objects in Context competitions, the convolutional neural network (CNN) has become important in computer vision and natural language processing. However, state-of-the-art CNNs are computationally memory-intensive, thus energy-efficient implementation on the embedded platform is challenging. Recently, VGGNet and ResNet showed that deep neural networks with more convolution layers and a few fully connected layers can achieve lower error rates, thus reducing the complexity of convolution layers is of utmost importance. In this paper, we evaluate three variations of convolutions, including direct convolution (Direct-Conv), fast Fourier transform (FFT)-based convolution (FFT-Conv), and FFT overlap and add convolution (FFT-OVA-Conv) in terms of computation complexity and memory storage requirements for popular CNN networks in embedded hardware. We implemented these three techniques for ResNet-20 with the CIFAR-10 data set on a low-power domain-specific many-core architecture called power-efficient nanoclusters (PENCs), NVIDIA Jetson TX1 graphics processing unit (GPU), ARM Cortex A53 CPU, and SPARse Convolutional NETwork (SPARCNet) accelerator on Zynq 7020 FPGA to explore the tradeoff between software and hardware implementation, domain-specific logic and instructions, as well as various parallelism across different architectures. Results are evaluated and compared with respect to throughput per layer, energy consumption, and execution time for the three methods. SPARCNet deployed on Zynq FPGA achieved 42-ms runtime with 135-mJ energy consumption with a 10.8-MB/s throughput per layer using FFT-Conv for ResNet-20. Using built-in FFT instruction in PENC, the FFT-OVA-Conv performs $2.9\times $ and $1.65\times $ faster and achieves $6.8\times $ and $2.5\times $ higher throughput per watt than Direct-Conv and FFT-Conv. In ARM A53 CPU, FFT-OVA-Conv achieves $3.36\times $ and $1.38\times $ improvement in execution time and $2.72\times $ and $1.32\times $ higher throughput than Direct-Conv and FFT-Conv. In TX1 GPU, FFT-Conv is $1.9\times $ faster, $2.2\times $ more energy-efficient, and achieves $5.6\times $ higher throughput per layer than Direct-Conv. PENC is 10 $916\times $ and $1.8\times $ faster and $5053\times $ and $4.3\times $ more energy-efficient and achieves $7.5\times $ and $1.2\times $ higher throughput per layer than ARM A53 CPU and TX1 GPU, respectively.

Transfer learning for classification of cardiovascular tissues in histological images

Background and Objective: Automatic classification of healthy tissues and organs based on histology images is an open problem, mainly due to the lack of automated tools. Solutions in this regard have potential in educational medicine and medical practices. Some preliminary advances have been made using image processing techniques and classical supervised learning. Due to the breakthrough performance of deep learning in various areas, we present an approach to recognise and classify, automatically, fundamental tissues and organs using Convolutional Neural Networks (CNN).Methods: We adapt four popular CNNs architectures – ResNet, VGG19, VGG16 and Inception – to this problem through transfer learning. The resulting models are evaluated at three stages. Firstly, all the transferred networks are compared to each other. Secondly, the best resulting fine-tuned model is compared to an ad-hoc 2D multi-path model to outline the importance of transfer learning. Thirdly, the same model is evaluated against the state-of-the-art method, a cascade SVM using LBP-based descriptors, to contrast a traditional machine learning approach and a representation learning one. The evaluation task consists of separating six classes accurately: smooth muscle of the elastic artery, smooth muscle of the large vein, smooth muscle of the muscular artery, cardiac muscle, loose connective tissue, and light regions. The different networks are tuned on 6000 blocks of 100 × 100 pixels and tested on 7500.Results: Our proposal yields F-score values between 0.717 and 0.928. The highest and lowest performances are for cardiac muscle and smooth muscle of the large vein, respectively. The main issue leading to limited classification scores for the latter class is its similarity with the elastic artery. However, this confusion is evidenced during manual annotation as well. Our algorithm reached improvements in F-score between 0.080 and 0.220 compared to the state-of-the-art machine learning approach.Conclusions: We conclude that it is possible to classify healthy cardiovascular tissues and organs automatically using CNNs and that deep learning holds great promise to improve tissue and organs classification. We left our training and test sets, models and source code publicly available to the research community.

On the generalization of color texture-based face anti-spoofing

Despite the significant attention given to the problem of face spoofing, we still lack generalized presentation attack detection (PAD) methods performing robustly in practical face recognition systems. The existing face anti-spoofing techniques have indeed achieved impressive results when trained and evaluated on the same database (i.e. intra-test protocols). Cross-database experiments have, however, revealed that the performance of the state-of-the-art methods drops drastically as they fail to cope with new attacks scenarios and other operating conditions that have not been seen during training and development phases. So far, even the popular convolutional neural networks (CNN) have failed to derive well-generalizing features for face anti-spoofing. In this work, we explore the effect of different factors, such as acquisition conditions and presentation attack instrument (PAI) variation, on the generalization of color texture-based face anti-spoofing. Our extensive cross-database evaluation of seven color texture-based methods demonstrates that most of the methods are unable to generalize to unseen spoofing attack scenarios. More importantly, the experiments show that some facial color texture representations are more robust to particular PAIs than others. From this observation, we propose a face PAD solution of attack-specific countermeasures based solely on color texture analysis and investigate how well it generalizes under display and print attacks in different conditions. The evaluation of the method combining attack-specific detectors on three benchmark face anti-spoofing databases showed remarkable generalization ability against display attacks while print attacks require still further attention.

A Flexible and Energy-Efficient Convolutional Neural Network Acceleration With Dedicated ISA and Accelerator

State-of-the-art convolutional neural networks (CNNs) usually have a large number of layers and filter weights which bring huge computation and communication overheads. A general purpose instruction set architecture (ISA) is flexible but has low code density and high power consumption. The existing CNN-specific accelerators are much more efficient but usually are inflexible or require a complex controller to handle the computation and data transfer of different CNNs. In this brief, we propose a new CNN-specific ISA which embeds the parallel computation and data reuse parameters in the instructions. An instruction generator deploys the instruction parameters according to the feature of CNNs and hardware’s computation and storage resources. In addition, a reconfigurable accelerator with 225 multipliers and 24 adder trees is realized to obtain efficient parallel computation and data transfer. Compared with x86 processors, our design has 392 times better energy efficiency and 16 times higher code density. Compared with other state-of-the-art accelerators, our solution has a higher flexibility to support all popular CNNs and a higher energy efficiency.

Symmetric convolutional neural network for mandible segmentation

Mandible segmentation is an essential step in craniomaxillofacial surgery planning, which aims to segment mandible from multi-slice computed tomography (MSCT) images. One main disadvantage of most existing mandible segmentation methods is that they require numerous expert knowledge for semi-automatic segmentation. However, the high-quality expert knowledge is hard to achieve in practice due to the scarcity of experienced doctors and experts. To solve this problem, we propose an end-to-end trainable deep learning based method, which performs segmentation in an automatic, accurate and efficient manner. Different from the popular convolutional neural network (CNN), our proposed symmetric convolutional neural network (SCNN) enforces convolution and deconvolution computation to be symmetric as to achieve a good segmentation performance. Furthermore, benefiting from the manner of end-to-end, SCNN could automatically perform mandible segmentation from raw image data. Such advantages remarkably reduce the human effort and achieve competitive performance. To verify the effectiveness of our method, we build a multi-slice computer tomography mandible dataset which includes 93 cases. The experimental results show that the proposed SCNN is superior to several popular baselines in terms of the dice similarity coefficient (DSC).

Indexing of the CNN features for the large scale image search

The convolutional neural network (CNN) features can give a good description of image content, which usually represent images with unique global vectors. Although they are compact compared to local descriptors, they still cannot efficiently deal with large-scale image retrieval due to the cost of the linear incremental computation and storage. To address this issue, we build a simple but effective indexing framework based on inverted table, which significantly decreases both the search time and memory usage. In addition, several strategies are fully investigated under an indexing framework to adapt it to CNN features and compensate for quantization errors. First, we use multiple assignment for the query and database images to increase the probability of relevant images' co-existing in the same Voronoi cells obtained via the clustering algorithm. Then, we introduce embedding codes to further improve precision by removing false matches during a search. We demonstrate that by using hashing schemes to calculate the embedding codes and by changing the ranking rule, indexing framework speeds can be greatly improved. Extensive experiments conducted on several unsupervised and supervised benchmarks support these results and the superiority of the proposed indexing framework. We also provide a fair comparison between the popular CNN features.

Deep Max-Margin Discriminant Projection

In this paper, a unified Bayesian max-margin discriminant projection framework is proposed, which is able to jointly learn the discriminant feature space and the max-margin classifier with different relationships between the latent representations and observations. We assume that the latent representation follows a normal distribution whose sufficient statistics are functions of the observations. The function can be flexibly realized through either shallow or deep structures. The shallow structure includes linear, nonlinear kernel-based functions, and even the convolutional projection, which can be further trained layerwisely to build a multilayered convolutional feature learning model. To take the advantage of the deep neural networks, especially their highly expressive ability and efficient parameter learning, we integrate Bayesian modeling and the popular neural networks, for example, mltilayer perceptron and convolutional neural network, to build an end-to-end Bayesian deep discriminant projection under the proposed framework, which degenerated into the existing shallow linear or convolutional projection with the single-layer structure. Moreover, efficient scalable inferences for the realizations with different functions are derived to handle large-scale data via a stochastic gradient Markov chain Monte Carlo. Finally, we demonstrate the effectiveness and efficiency of the proposed models by the experiments on real-world data, including four image benchmarks (MNIST, CIFAR-10, STL-10, and SVHN) and one measured radar high-resolution range profile dataset, with the detailed analysis about the parameters and computational complexity.

Training CNNs With Normalized Kernels

Several methods of normalizing convolution kernels have been proposed in the literature to train convolutional neural networks (CNNs), and have shown some success. However, our understanding of these methods has lagged behind their success in application; there are a lot of open questions, such as why a certain type of kernel normalization is effective and what type of normalization should be employed for each (e.g., higher or lower) layer of a CNN. As the first step towards answering these questions, we propose a framework that enables us to use a variety of kernel normalization methods at any layer of a CNN. A naive integration of kernel normalization with a general optimization method, such as SGD, often entails instability while updating parameters. Thus, existing methods employ ad-hoc procedures to empirically assure convergence. In this study, we pose estimation of convolution kernels under normalization constraints as constraint-free optimization on kernel submanifolds that are identified by the employed constraints. Note that naive application of the established optimization methods for matrix manifolds to the aforementioned problems is not feasible because of the hierarchical nature of CNNs. To this end, we propose an algorithm for optimization on kernel manifolds in CNNs by appropriate scaling of the space of kernels based on structure of CNNs and statistics of data. We theoretically prove that the proposed algorithm has assurance of almost sure convergence to a solution at single minimum. Our experimental results show that the proposed method can successfully train popular CNN models using several different types of kernel normalization methods. Moreover, they show that the proposed method improves classification performance of baseline CNNs, and provides state-of-the-art performance for major image classification benchmarks.

Animal classification using facial images with score‐level fusion

A real‐world animal biometric system that detects and describes animal life in image and video data is an emerging subject in machine vision. These systems develop computer vision approaches for the classification of animals. A novel method for animal face classification based on score‐level fusion of recently popular convolutional neural network (CNN) features and appearance‐based descriptor features is presented. This method utilises a score‐level fusion of two different approaches; one uses CNN which can automatically extract features, learn and classify them; and the other one uses kernel Fisher analysis (KFA) for its feature extraction phase. The proposed method may also be used in other areas of image classification and object recognition. The experimental results show that automatic feature extraction in CNN is better than other simple feature extraction techniques (both local‐ and appearance‐based features), and additionally, appropriate score‐level combination of CNN and simple features can achieve even higher accuracy than applying CNN alone. The authors showed that the score‐level fusion of CNN extracted features and appearance‐based KFA method have a positive effect on classification accuracy. The proposed method achieves 95.31% classification rate on animal faces which is significantly better than the other state‐of‐the‐art methods.

Finger vein identification using Convolutional Neural Network and supervised discrete hashing

Automated personal identification using vascular biometrics, such as from the finger vein images, is highly desirable as it helps to protect the personal privacy and anonymity in automated personal identification. The Convolutional Neural Network (CNN) has shown remarkable capability for learning biometric features that can offer robust and accurate matching. This paper introduces a new approach for the finger vein authentication using the CNN and supervised discrete hashing. We also systematically investigate comparative performance using several popular CNN architectures in other domains, i.e., Light CNN, VGG-16, Siamese and the CNN with Bayesian inference based matching. The experimental results are presented using a publicly available two-session finger-vein database. Most accurate performance is achieved by incorporating supervised discrete hashing from a CNN trained using the triplet-based loss function. The proposed approach not only achieves outperforming results over other considered CNN architecture available in the literature but also offers significantly reduced template size as compared with those over the other finger-vein images matching methods available in the literature.

An Analysis Of Convolutional Neural Networks For Image Classification

This paper presents an empirical analysis of theperformance of popular convolutional neural networks (CNNs) for identifying objects in real time video feeds. The most popular convolution neural networks for object detection and object category classification from images are Alex Nets, GoogLeNet, and ResNet50. A variety of image data sets are available to test the performance of different types of CNN’s. The commonly found benchmark datasets for evaluating the performance of a convolutional neural network are anImageNet dataset, and CIFAR10, CIFAR100, and MNIST image data sets. This study focuses on analyzing the performance of three popular networks: Alex Net, GoogLeNet, and ResNet50. We have taken three most popular data sets ImageNet, CIFAR10, and CIFAR100 for our study, since, testing the performance of a network on a single data set does not reveal its true capability and limitations. It must be noted that videos are not used as a training dataset, they are used as testing datasets. Our analysis shows that GoogLeNet and ResNet50 are able to recognize objects with better precision compared to Alex Net. Moreover, theperformance of trained CNN’s vary substantially across different categories of objects and we, therefore, will discuss the possible reasons for this.

DM-L Based Feature Extraction and Classifier Ensemble for Object Recognition

Deep Learning is a powerful technique that is widely applied to Image Recognition and Natural Language Processing tasks amongst many other tasks. In this work, we propose an efficient technique to utilize pre-trained Convolutional Neural Network (CNN) architectures to extract powerful features from images for object recognition purposes. We have built on the existing concept of extending the learning from pre-trained CNNs to new databases through activations by proposing to consider multiple deep layers. We have exploited the progressive learning that happens at the various intermediate layers of the CNNs to construct Deep Multi-Layer (DM-L) based Feature Extraction vectors to achieve excellent object recognition performance. Two popular pre-trained CNN architecture models i.e. the VGG_16 and VGG_19 have been used in this work to extract the feature sets from 3 deep fully connected multiple layers namely “fc6”, “fc7” and “fc8” from inside the models for object recognition purposes. Using the Principal Component Analysis (PCA) technique, the Dimensionality of the DM-L feature vectors has been reduced to form powerful feature vectors that have been fed to an external Classifier Ensemble for classification instead of the Softmax based classification layers of the two original pre-trained CNN models. The proposed DM-L technique has been applied to the Benchmark Caltech-101 object recognition database. Conventional wisdom may suggest that feature extractions based on the deepest layer i.e. “fc8” compared to “fc6” will result in the best recognition performance but our results have proved it otherwise for the two considered models. Our experiments have revealed that for the two models under consideration, the “fc6” based feature vectors have achieved the best recognition performance. State-of-the-Art recognition performances of 91.17% and 91.35% have been achieved by utilizing the “fc6” based feature vectors for the VGG_16 and VGG_19 models respectively. The recognition performance has been achieved by considering 30 sample images per class whereas the proposed system is capable of achieving improved performance by considering all sample images per class. Our research shows that for feature extraction based on CNNs, multiple layers should be considered and then the best layer can be selected that maximizes the recognition performance.

Rotation Invariant Local Binary Convolution Neural Networks

Convolutional neural networks (CNNs) have achieved unprecedented successes in computer vision fields, but they remain challenged by the problem about how to effectively process the orientation transformation of objects with fewer parameters. In this paper, we propose a new convolutional module, local binary orientation module (LBoM), which takes advantages of both local binary convolutional and active rotating filters to effectively deal with the rotation variations with fewer parameters. LBoM can be naturally inserted to popular CNN models and upgrade them to be rotation invariant local binary CNNs (RI-LBCNNs). RI-LBCNNs can be learned with off-the-shelf optimization approaches in an end-to-end manner and fulfill image classification tasks. Extensive experiments on four benchmarks show that RI-LBCNNs can perform image classification with fewer network parameters and significantly outperform the baseline LBCNN when processing images with large rotation variations.

Improving Transfer Learning and Squeeze- and-Excitation Networks for Small-Scale Fine-Grained Fish Image Classification

Scientific studies on species composition and abundance distribution of fishes have considerable importance to the fishery industry, biodiversity protection, and marine ecosystem. In these studies, fish images are typically collected with the help of scuba divers or autonomous underwater vehicles. These images are then annotated manually by marine biologists. Such a process is certainly a tremendous waste of manpower and material resources. In recent years, the introduction of deep learning has helped making remarkable progress in this area. However, fish image classification can be considered as fine-grained problem, which is more challenging than common image classification, especially with low-quality and small-scale data. Meanwhile, well-known effective convolutional neural networks (CNNs) consistently require a large quantity of high-quality data. This paper presents a new method by improving transfer learning and squeeze-and-excitation networks for fine-grained fish image classification on low-quality and small-scale datasets. Our method enhances data augmentation through super-resolution reconstruction to enlarge the dataset with high-quality images, pre-pretrains, and pretrains to learn common and domain knowledge simultaneously while fine-tuning with professional skill. In addition, refined squeeze-and-excitation blocks are designed to improve bilinear CNNs for a fine-grained classification. Unlike well-known CNNs for image classification, our method can classify images with insufficient low-quality training data. Moreover, we compare the performance of our method with commonly used CNNs on small-scale fine-grained datasets, namely, Croatian and QUT fish datasets. The experimental results show that our method outperforms popular CNNs with higher fish classification accuracy, which indicates its potential applications in combination with other newly updated CNNs.

A Discriminatively Learned CNN Embedding for Person Reidentification

In this article, we revisit two popular convolutional neural networks in person re-identification (re-ID): verification and identification models. The two models have their respective advantages and limitations due to different loss functions. Here, we shed light on how to combine the two models to learn more discriminative pedestrian descriptors. Specifically, we propose a Siamese network that simultaneously computes the identification loss and verification loss. Given a pair of training images, the network predicts the identities of the two input images and whether they belong to the same identity. Our network learns a discriminative embedding and a similarity measurement at the same time, thus taking full usage of the re-ID annotations. Our method can be easily applied on different pretrained networks. Albeit simple, the learned embedding improves the state-of-the-art performance on two public person re-ID benchmarks. Further, we show that our architecture can also be applied to image retrieval. The code is available at https://github.com/layumi/2016_person_re-ID.

Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification.

Spiking neural networks (SNNs) can potentially offer an efficient way of doing inference because the neurons in the networks are sparsely activated and computations are event-driven. Previous work showed that simple continuous-valued deep Convolutional Neural Networks (CNNs) can be converted into accurate spiking equivalents. These networks did not include certain common operations such as max-pooling, softmax, batch-normalization and Inception-modules. This paper presents spiking equivalents of these operations therefore allowing conversion of nearly arbitrary CNN architectures. We show conversion of popular CNN architectures, including VGG-16 and Inception-v3, into SNNs that produce the best results reported to date on MNIST, CIFAR-10 and the challenging ImageNet dataset. SNNs can trade off classification error rate against the number of available operations whereas deep continuous-valued neural networks require a fixed number of operations to achieve their classification error rate. From the examples of LeNet for MNIST and BinaryNet for CIFAR-10, we show that with an increase in error rate of a few percentage points, the SNNs can achieve more than 2x reductions in operations compared to the original CNNs. This highlights the potential of SNNs in particular when deployed on power-efficient neuromorphic spiking neuron chips, for use in embedded applications.

Deep Neural Network Initialization Methods for Micro-Doppler Classification With Low Training Sample Support

Deep neural networks (DNNs) require large-scale labeled data sets to prevent overfitting while having good generalization. In radar applications, however, acquiring a measured data set of the order of thousands is challenging due to constraints on manpower, cost, and other resources. In this letter, the efficacy of two neural network initialization techniques—unsupervised pretraining and transfer learning—for dealing with training DNNs on small data sets is compared. Unsupervised pretraining is implemented through the design of a convolutional autoencoder (CAE), while transfer learning from two popular convolutional neural network architectures (VGGNet and GoogleNet) is used to augment measured RF data for training. A 12-class problem for discrimination of micro-Doppler signatures for indoor human activities is utilized to analyze activation maps, bottleneck features, class model, and classification accuracy with respect to training sample size. Results show that on meager data sets, transfer learning outperforms unsupervised pretraining and random initialization by 10% and 25%, respectively, but that when the sample size exceeds 650, unsupervised pretraining surpasses transfer learning and random initialization by 5% and 10%, respectively. Visualization of activation layers and learned models reveals how the CAE succeeds in representing the micro-Doppler signature.

A new image classification method using CNN transfer learning and web data augmentation

Since Convolutional Neural Network (CNN) won the image classification competition 202 (ILSVRC12), a lot of attention has been paid to deep layer CNN study. The success of CNN is attributed to its superior multi-scale high-level image representations as opposed to hand-engineering low-level features. However, estimating millions of parameters of a deep CNN requires a large number of annotated samples, which currently prevents many superior deep CNNs (such as AlexNet, VGG, ResNet) being applied to problems with limited training data. To address this problem, a novel two-phase method combining CNN transfer learning and web data augmentation is proposed. With our method, the useful feature presentation of pre-trained network can be efficiently transferred to target task, and the original dataset can be augmented with the most valuable Internet images for classification. Our method not only greatly reduces the requirement of a large training data, but also effectively expand the training dataset. Both of method features contribute to the considerable over-fitting reduction of deep CNNs on small dataset. In addition, we successfully apply Bayesian optimization to solve the tuff problem, hyper-parameter tuning, in network fine-tuning. Our solution is applied to six public small datasets. Extensive experiments show that, comparing to traditional methods, our solution can assist the popular deep CNNs to achieve better performance. Particularly, ResNet can outperform all the state-of-the-art models on six small datasets. The experiment results prove that the proposed solution will be the great tool for dealing with practice problems which are related to use deep CNNs on small dataset.

Maximizing CNN Accelerator Efficiency Through Resource Partitioning

Convolutional neural networks (CNNs) are revolutionizing machine learning, but they present significant computational challenges. Recently, many FPGA-based accelerators have been proposed to improve the performance and efficiency of CNNs. Current approaches construct a single processor that computes the CNN layers one at a time; the processor is optimized to maximize the throughput at which the collection of layers is computed. However, this approach leads to inefficient designs because the same processor structure is used to compute CNN layers of radically varying dimensions. We present a new CNN accelerator paradigm and an accompanying automated design methodology that partitions the available FPGA resources into multiple processors, each of which is tailored for a different subset of the CNN convolutional layers. Using the same FPGA resources as a single large processor, multiple smaller specialized processors increase computational efficiency and lead to a higher overall throughput. Our design methodology achieves 3.8x higher throughput than the state-of-the-art approach on evaluating the popular AlexNet CNN on a Xilinx Virtex-7 FPGA. For the more recent SqueezeNet and GoogLeNet, the speedups are 2.2x and 2.0x.