SVHN Datasets Research Articles

EncoDeep

This article proposes EncoDeep, an end-to-end framework that facilitates encoding, bitwidth customization, fine-tuning, and implementation of neural networks on FPGA platforms. EncoDeep incorporates nonlinear encoding to the computation flow of neural networks to save memory. The encoded features demand significantly lower storage compared to the raw full-precision activation values; therefore, the execution flow of EncoDeep hardware engine is completely performed within the FPGA using on-chip streaming buffers with no access to the off-chip DRAM. We further propose a fully automated optimization algorithm that determines the flexible encoding bitwidths across network layers. EncoDeep full-stack framework comprises a compiler that takes a high-level Python description of an arbitrary neural network. The compiler then instantiates the corresponding elements from EncoDeep Hardware library for FPGA implementation. Our evaluations on MNIST, SVHN, and CIFAR-10 datasets demonstrate an average of 4.65× throughput improvement compared to stand-alone weight encoding. We further compare EncoDeep with six FPGA accelerators on ImageNet, showing an average of 3.6× and 2.54× improvement in throughput and performance-per-watt, respectively.

Hierarchical Auxiliary Learning

Conventional application of convolutional neural networks (CNNs) for image classification and recognition is based on the assumption that all target classes are equal (i.e. no hierarchy) and exclusive of one another (i.e. no overlap). CNN-based image classifiers built on this assumption, therefore, cannot take into account an innate hierarchy among target classes (e.g. cats and dogs in animal image classification) or additional information that can be easily derived from the data (e.g. numbers larger than five in the recognition of handwritten digits), thereby resulting in scalability issues when the number of target classes is large. Combining two related but slightly different ideas of hierarchical classification and logical learning by auxiliary inputs, we propose a new learning framework called hierarchical auxiliary learning, which not only address the scalability issues with a large number of classes but also could further reduce the classification/recognition errors with a reasonable number of classes. In the hierarchical auxiliary learning, target classes are semantically or non-semantically grouped into superclasses, which turns the original problem of mapping between an image and its target class into a new problem of mapping between a pair of an image and its superclass and the target class. To take the advantage of a superclass as a hint during the learning phase, we introduce an auxiliary block into a neural network, which generates auxiliary scores used as additional information for final classification/recognition; in this paper, we add the auxiliary block between the last residual block and the fully-connected output layer of the ResNet. Experimental results show that the proposed hierarchical auxiliary learning reduces classification errors up to 0.56, 1.6 and 3.56 percent with MNIST, SVHN and CIFAR-10 datasets, respectively.

Deep network compression with teacher latent subspace learning and LASSO

Deep neural networks have been shown to excel in understanding multimedia by using latent representations to learn complex and useful abstractions. However, they remain unpractical for embedded devices due to memory constraints, high latency, and considerable power consumption at runtime. In this paper, we propose the compression of deep models based on learning lower dimensional subspaces from their latent representations while maintaining a minimal loss of performance. We leverage on the premise that deep convolutional neural networks extract many redundant features to learn new subspaces for feature representation. We construct a compressed model by reconstruction from representations captured by an already trained large model. As compared to state-of-the-art, the proposed approach does not rely on labeled data. Moreover, it allows the use of sparsity inducing LASSO parameter penalty to achieve better compression results than when used to train models from scratch. We perform extensive experiments using VGG-16 and wide ResNet models on CIFAR-10, CIFAR-100, MNIST and SVHN datasets. For instance, VGG-16 with 8.96M parameters trained on CIFAR-10 was pruned by 81.03 % with only 0.26 % generalization performance loss. Correspondingly, the size of the VGG-16 model is reduced from 35MB to 6.72MB to facilitate compact storage. Furthermore, the associated inference time for the same VGG-16 model is reduced from 1.1 secs to 0.6 secs so that inference is accelerated. Particularly, the proposed student models outperform state-of-the-art approaches and the same models trained from scratch.

Incremental Concept Learning via Online Generative Memory Recall.

The ability to learn more concepts from incrementally arriving data over time is essential for the development of a lifelong learning system. However, deep neural networks often suffer from forgetting previously learned concepts when continually learning new concepts, which is known as the catastrophic forgetting problem. The main reason for catastrophic forgetting is that past concept data are not available, and neural weights are changed during incrementally learning new concepts. In this article, we propose an incremental concept learning framework that includes two components, namely, ICLNet and RecallNet. ICLNet, which consists of a trainable feature extractor and a dynamic concept memory matrix, aims to learn new concepts incrementally. We propose a concept-contrastive loss to alleviate the magnitude of neural weight changes and mitigate the catastrophic forgetting problems. RecallNet aims to consolidate old concepts memory and recall pseudo samples, whereas ICLNet learns new concepts. We propose a balanced online memory recall strategy to reduce the information loss of old concept memory. We evaluate the proposed approach on the MNIST, Fashion-MNIST, and SVHN data sets and compare it with other pseudorehearsal-based approaches. Extensive experiments demonstrate the effectiveness of our approach.

Structured feature sparsity training for convolutional neural network compression

Convolutional neural networks (CNNs) with large model size and computing operations are difficult to be deployed on embedded systems, such as smartphones or AI cameras. In this paper, we propose a novel structured pruning method, termed the structured feature sparsity training (SFST), to speed up the inference process and reduce the memory usage of CNNs. Unlike other existing pruning methods, which require multiple iterations of pruning and retraining to ensure stable performance, SFST only needs to fine-tune the pretrained model with additional regularization on the less important features and then prune them, no multiple pruning and retraining needed. SFST can be deployed to a variety of modern CNN architectures including VGGNet, ResNet and MobileNetv2. Experimental results on CIFAR, SVHN, ImageNet and MSTAR benchmark dataset demonstrate the effectiveness of our scheme, which achieves superior performance over the state-of-the-art methods.

Highlight Every Step: Knowledge Distillation via Collaborative Teaching

High storage and computational costs obstruct deep neural networks to be deployed on resource-constrained devices. Knowledge distillation (KD) aims to train a compact student network by transferring knowledge from a larger pretrained teacher model. However, most existing methods on KD ignore the valuable information among the training process associated with training results. In this article, we provide a new collaborative teaching KD (CTKD) strategy which employs two special teachers. Specifically, one teacher trained from scratch (i.e., scratch teacher) assists the student step by step using its temporary outputs. It forces the student to approach the optimal path toward the final logits with high accuracy. The other pretrained teacher (i.e., expert teacher) guides the student to focus on a critical region that is more useful for the task. The combination of the knowledge from two special teachers can significantly improve the performance of the student network in KD. The results of experiments on CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, and ImageNet datasets verify that the proposed KD method is efficient and achieves state-of-the-art performance.

Controlling information capacity of binary neural network

Despite the growing popularity of deep learning technologies, high memory requirements and power consumption are essentially limiting their application in mobile and IoT areas. While binary convolutional networks can alleviate these problems, the limited bitwidth of weights is often leading to significant degradation of prediction accuracy. In this paper, we present a method for training binary networks that maintains a stable predefined level of their information capacity throughout the training process by applying Shannon entropy based penalty to convolutional filters. The results of experiments conducted on SVHN, CIFAR and ImageNet datasets demonstrate that the proposed approach can statistically significantly improve the accuracy of binary networks.

Sketch-then-Edit Generative Adversarial Network

Generative Adversarial Network (GAN) has been widely used to generate impressively plausible data. However, it is a non-trivial task to train the original GAN model in practice due to the vanishing gradient problem. This is because the JS divergence could be a constant (i.e., log2) when original data distribution and generated data distribution hold a negligible overlapping area. Under such a scenario, the gradient of generator is 0. Most efforts have been devoted to designing a more proper difference measure while few attentions have been paid to the former aspect of the issue.In this paper, we propose a new method to design a noise distribution having a guaranteed non-negligible overlapping area with raw data distribution. The key idea is to transform the noise from the randomized space into the raw data space. We propose to obtain the transformation as the basis matrix in non-negative matrix factorization because the basis matrix has the underlying features of the raw data. The proposed idea is instantiated as Sketch-then-Edit GAN (SEGAN) where sketches are the noises after transformation and are adopted as the name since they contains basic features of the raw data. Moreover, a new generator for editing the sketches into realistic-like data is designed. We mathematically prove that SEGAN solves the gradient vanishing problem, and conduct extensive experiments on the MNIST, CIFAR10, SVHN and Celeba datasets to demonstrate the effectiveness of SEGAN.

Progressive Learning of Low-Precision Networks for Image Classification

Recent years have witnessed a great advance of deep learning in a variety of vision tasks. Many state-of-the-art deep neural networks suffer from large size and high complexity, which makes them difficult to deploy in resource-limited platforms such as mobile devices. To this end, low-precision neural networks are widely studied that quantize weights or activations into the low-bit format. Although efficient, low-precision networks are usually difficult to train and encounter severe accuracy degradation. In this paper, we propose a new training strategy based on progressive learning for image classification. First, we equip each low-precision convolutional layer with an ancillary full-precision convolutional layer based on a low-precision network structure. Second, a decay method is introduced to reduce the output of the added full-precision convolution gradually, which keeps the resulting topology structure the same as the original low-precision convolution. Extensive experiments on SVHN, CIFAR and ILSVRC-2012 datasets reveal that the proposed method can bring faster convergence and higher accuracy for low-precision neural networks.

Utilizing Amari-Alpha Divergence to Stabilize the Training of Generative Adversarial Networks.

Generative Adversarial Nets (GANs) are one of the most popular architectures for image generation, which has achieved significant progress in generating high-resolution, diverse image samples. The normal GANs are supposed to minimize the Kullback–Leibler divergence between distributions of natural and generated images. In this paper, we propose the Alpha-divergence Generative Adversarial Net (Alpha-GAN) which adopts the alpha divergence as the minimization objective function of generators. The alpha divergence can be regarded as a generalization of the Kullback–Leibler divergence, Pearson divergence, Hellinger divergence, etc. Our Alpha-GAN employs the power function as the form of adversarial loss for the discriminator with two-order indexes. These hyper-parameters make our model more flexible to trade off between the generated and target distributions. We further give a theoretical analysis of how to select these hyper-parameters to balance the training stability and the quality of generated images. Extensive experiments of Alpha-GAN are performed on SVHN and CelebA datasets, and evaluation results show the stability of Alpha-GAN. The generated samples are also competitive compared with the state-of-the-art approaches.

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures.

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN, and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Optimized training and scalable implementation of Conditional Deep Neural Networks with early exits for Fog-supported IoT applications

The incoming IoT big data era requires efficient and resource-constrained mining of large sets of distributed data. This paper explores a possible approach to this end, combining the two emerging paradigms of Conditional Neural Networks with early exits and Fog Computing. Apart from describing the general framework, we provide four specific contributions. First, after reviewing the basic architectures of CDNNs with early exits and characterizing their computational capacity, we consider three basic algorithms for their supervised training (namely the End-to-End, Layer-Wise and Classifier-Wise training algorithms), and, then, formally characterize and compare the resulting tradeoffs in a Fog-supported implementation. Second, after presenting a reference architecture for the local classifiers equipping the considered CDNNs, we develop an optimized framework for the parallel and distributed setting of their decision thresholds. Third, we propose a greedy algorithm for placing the early exits efficiently on the considered CDNNs and prove its linear scaling complexity. Fourth, we analytically characterize in closed-form and analyze the energy performance of the optimal CDNN-onto-Fog mapping. Finally, extensive numerical tests are presented, in order to test and compare the energy-vs.-implementation complexity-vs.-accuracy performance of the resulting optimized CDNN-over-Fog platforms under the IoT-oriented SVHN and FER-2013 datasets.

TDP: Two-dimensional perceptron for image recognition

Convolutional neural network (CNN) is widely applied to different areas due to good recognition performance. However, convolution operation is a complex computation and consumes the bulk of processing time for CNN. It is still a hot problem how to develop a novel model with good recognition performance for deep learning. Here, we propose a novel model, namely, two-dimensional perceptron (TDP), to get direct input of two-dimensional data for further processing. A TDP has a new network architecture and an innovative computation process of hidden neurons. In cases with the same number of hidden neurons, compared with multilayer perceptron (MLP), TDP achieves good recognition performance with 1×-36× speedup and a decrease of parameters by exceeding 97% on MNIST and COIL-20 datasets. Meanwhile, TDP obtains 1%–32% improvement of recognition accuracy in comparison to CNN on CIFAR-10 and SVHN datasets. Furthermore, on INFUSE dataset, TDP has an increase of F1 score by up to almost 11% in comparison with MLP and CNN. The results indicate that TDP is a promising and novel model with excellent recognition performance.

Deterministic conversion rule for CNNs to efficient spiking convolutional neural networks

This paper proposes a general conversion theory to reveal the relations between convolutional neural network (CNN) and spiking convolutional neural network (spiking CNN) from structure to information processing. Based on the conversion theory and the statistical features of the activations distribution in CNN, we establish a deterministic conversion rule to convert CNNs into spiking CNNs with definite conversion procedure and the optimal setting of all parameters. Included in conversion rule, we propose a novel “n-scaling” weight mapping method to realize high-accuracy, low-latency and power efficient object classification on hardware. For the first time, the minimum dynamic range of spiking neurons membrane potential is studied to help to balance the trade-off between representation range and precise of the data type adopted by dedicated hardware when spiking CNNs run on it. The simulation results demonstrate that the converted spiking CNNs perform well on MNIST, SVHN and CIFAR-10 datasets. The accuracy loss over three datasets is no more than 0.4%. 39% of processing time is shortened at best, and less power consumption is benefited from lower latency achieved by our conversion rule. Furthermore, the results of noise robustness experiments indicate that spiking CNN inherits the robustness from its corresponding CNN.

Improved Highway Network Block for Training Very Deep Neural Networks

Very deep networks are successful in various tasks with reported results surpassing human performance. However, training such very deep networks is not trivial. Typically, the problems of learning the identity function and feature reuse can work together to plague optimization of very deep networks. In this paper, we propose a highway network with gate constraints that addresses the aforementioned problems, and thus alleviates the difficulty of training. Namely, we propose two variants of highway network, HWGC and HWCC , employing feature summation and concatenation respectively. The proposed highway networks, besides being more computationally efficient, are shown to have more interesting learning characteristics such as natural learning of hierarchical and robust representations due to a more effective usage of model depth, fewer gates for successful learning, better generalization capacity and faster convergence than the original highway network. Experimental results show that our models outperform the original highway network and many state-of-the-art models. Importantly, we observe that our second model with feature concatenation and compression consistently outperforms our model with feature summation of similar depth, the original highway network, many state-of-the-art models and even ResNets on four benchmarking datasets which are CIFAR-10, CIFAR-100, Fashion-MNIST, SVHN and imagenet-2012 (ILSVRC) datasets. Furthermore, the second proposed model is more computationally efficient than the state-of-the-art in view of training, inference time and GPU memory resource, which strongly supports real-time applications. Using a similar number of model parameters for the CIFAR-10, CIFAR-100, Fashion-MNIST and SVHN datasets, the significantly shallower proposed model can surpass the performance of ResNet-110 and ResNet-164 that are roughly 6 and 8 times deeper, respectively. Similarly, for the imagenet dataset, the proposed models surpass the performance of ResNet-101 and ResNet-152 that are roughly three times deeper.

An Incremental Self-Labeling Strategy for Semi-Supervised Deep Learning Based on Generative Adversarial Networks

The recent success of deep neural networks is attributed in part to large-scale well-labeled training data. However, with the ever-increasing size of modern datasets, combined with the difficulty of obtaining label information, semi-supervised learning (SSL) has become one of the most remarkable issues in data analysis. In this paper, we propose an Incremental Self-Labeling strategy for SSL based on Generative Adversarial Nets (ISL-GAN), which functions by constantly assigning unlabeled data with virtual labels for promoting the training process. Specifically, during the virtual labeling process, we introduce a temporal-based self-labeling strategy for safe and stable data labeling. Then, to dynamically assign more virtual labels to data during the training, we conduct a phased incremental label screening and updating strategy. Finally, to balance the contribution of samples with different loss during the training process, we further introduce the Balance factor Term (BT). The experimental results show that the proposed method gives rise to state-of-the-art semi-supervised learning results for the MNIST, CIFAR-10, and SVHN datasets. Particularly, our model performs well with fewer labeled conditions. With a dataset of only 1,000 labeled CIFAR-10 images with CONV-Large Net, a test error of 11.2% can be achieved, and nearly the same performance with a 3.5% test error can be achieved with both 500 and 1,000 image-labeled SVHN datasets.

PARAMETRIC FLATTEN-T SWISH: AN ADAPTIVE NONLINEAR ACTIVATION FUNCTION FOR DEEP LEARNING

Activation function is a key component in deep learning that performs non-linear mappings between the inputs and outputs. Rectified Linear Unit (ReLU) has been the most popular activation function across the deep learning community. However, ReLU contains several shortcomings that can result in inefficient training of the deep neural networks, these are: 1) the negative cancellation property of ReLU tends to treat negative inputs as unimportant information for the learning, resulting in performance degradation; 2) the inherent predefined nature of ReLU is unlikely to promote additional flexibility, expressivity, and robustness to the networks; 3) the mean activation of ReLU is highly positive and leads to bias shift effect in network layers; and 4) the multilinear structure of ReLU restricts the non-linear approximation power of the networks. To tackle these shortcomings, this paper introduced Parametric Flatten-T Swish (PFTS) as an alternative to ReLU. By taking ReLU as a baseline method, the experiments showed that PFTS improved classification accuracy on SVHN dataset by 0.31%, 0.98%, 2.16%, 17.72%, 1.35%, 0.97%, 39.99%, and 71.83% on DNN-3A, DNN-3B, DNN-4, DNN-5A, DNN-5B, DNN-5C, DNN-6, and DNN-7, respectively. Besides, PFTS also achieved the highest mean rank among the comparison methods. The proposed PFTS manifested higher non-linear approximation power during training and thereby improved the predictive performance of the networks.

Adversarial Dual Network Learning With Randomized Image Transform for Restoring Attacked Images

We develop a new method for defending deep neural networks against attacks using adversarial dual network learning with randomized nonlinear image transform. We introduce a randomized nonlinear transform to disturb and partially destroy the sophisticated pattern of attack noise. We then design a generative cleaning network to recover the original image content damaged by this nonlinear transform and remove residual attack noise. We also construct a detector network which serves as the dual network for the target classifier to be defended, being able to detect patterns of attack noise. The generative cleaning network and detector network are jointly trained using adversarial learning, fighting against each other to minimize both perceptual loss and adversarial loss. Our extensive experimental results demonstrate that our approach improves the state-of-art by large margins in both white-box and black-box attacks. It significantly improves the classification accuracy for white-box attacks upon the second best method by more than 30% on the SVHN dataset and more than 14% on the challenging CIFAR-10 dataset.

Group Feedback Capsule Network

In capsule networks (CapsNets), the capsule is made up of collections of neurons. Their adjacent capsule layers are connected using routing-by-agreement mechanisms in an unsupervised way. The routing-by-agreement mechanisms have two main drawbacks: a) too many parameters and high computation complexity; b) the cluster distribution assumptions of these routing mechanisms may not hold in some complex real-world data. In this paper, we propose a novel Group Feedback Capsule Network (GF-CapsNet) which adopts a supervised routing strategy called group-routing. Compared with the previous routing strategies which globally transform each capsule, Group-routing equally splits capsules into groups where capsules locally share the same transformation weights, reducing routing parameters. To address the second drawback, we devise a distance network to directly predict capsules in a supervised way without making distribution assumptions. Our proposed group-routing captures local information of low-level capsules by group-wise transformation and supervisedly predicts high-level ones in a feedback way to address two drawbacks respectively. We conduct experiments on CIFAR-10/100 and SVHN datasets and the results show that our method can perform better against state-of-the-arts.

RS-CapsNet: An Advanced Capsule Network

Capsule Network is a novel and promising neural network in the field of deep learning, which has shown good performance in image classification by encoding features into capsules and constructing the part-whole relationships. However, the original Capsule Network is not suitable for the images with complex background due to its weak ability of feature extraction, a large number of training parameters, and the characteristic of explaining everything in the image. To address the above issues, we propose an advanced Capsule Network named RS-CapsNet, which uses Res2Net block to extract multi-scale features, and Squeeze-and-Excitation (SE) block to highlight useful features and suppress useless features. Meanwhile, we adopt the method of linear combination between capsules, which enhances the representation ability of capsules for detected object and reduces the number of capsules. Moreover, we propose the method of firstly constructing intermediate capsules that can represent most of the detected object, and then using the intermediate capsules and primary capsules to construct the classification capsules together. The experimental results show that proposed RS-CapsNet has better performance on CIFAR10, CIFAR100, SVHN, FashionMNIST, and AffNIST datasets, it can also provide better translation equivariance, and the number of training parameters is reduced by 65.11%.