NORB Datasets Research Articles

Self-balancing Incremental Broad Learning System with privacy protection

Incremental learning algorithms have been developed as an efficient solution for fast remodeling in Broad Learning Systems (BLS) without a retraining process. Even though the structure and performance of broad learning are gradually showing superiority, private data leakage in broad learning systems is still a problem that needs to be solved. Recently, Multiparty Secure Broad Learning System (MSBLS) is proposed to allow two clients to participate training. However, privacy-preserving broad learning across multiple clients has received limited attention. In this paper, we propose a Self-Balancing Incremental Broad Learning System (SIBLS) with privacy protection by considering the effect of different data sample sizes from clients, which allows multiple clients to be involved in the incremental learning. Specifically, we design a client selection strategy to select two clients in each round by reducing the gap in the number of data samples in the incremental updating process. To ensure the security under the participation of multiple clients, we introduce a mediator in the data encryption and feature mapping process. Three classical datasets are used to validate the effectiveness of our proposed SIBLS, including MNIST, Fashion and NORB datasets. Experimental results show that our proposed SIBLS can have comparable performance with MSBLS while achieving better performance than federated learning in terms of accuracy and running time.

Stacked Broad Learning System: From Incremental Flatted Structure to Deep Model

The broad learning system (BLS) has been proved to be effective and efficient lately. In this article, several deep variants of BLS are reviewed, and a new adaptive incremental structure, Stacked BLS, is proposed. The proposed model is a novel incremental stacking of BLS. This invariant inherits the efficiency and effectiveness of BLS that the structure and weights of lower layers of BLS are fixed when the new blocks are added. The incremental stacking algorithm computes not only the connection weights between the newly stacking blocks but also the connection weights of the enhancement nodes within the BLS block. The Stacked BLS is considered as the increment of “layers” and “neurons” dynamically during the training for multilayer neural networks. The proposed architecture along with the training algorithms that utilizes the residual characteristic is very versatile in comparison with traditional fixed architecture. Finally, experimental results on UCI datasets, MNIST dataset, NORB dataset, CIFAR-10 dataset, SVHN dataset, and CIFAR-100 dataset indicate that the proposed method outperforms the selected state-of-the-art methods on both accuracy and training speed, such as deep residual networks. The results also imply that the proposed structure could highly reduce the number of nodes and the training time of the original BLS in the classification task of some datasets.

Densely Connected Deep Extreme Learning Machine Algorithm

As a single hidden layer feed-forward neural network, the extreme learning machine (ELM) has been extensively studied for its short training time and good generalization ability. Recently, with the deep learning algorithm becoming a research hotspot, some deep extreme learning machine algorithms such as multi-layer extreme learning machine (ML-ELM) and hierarchical extreme learning machine (H-ELM) have also been proposed. However, the deep ELM algorithm also has many shortcomings: (1) when the number of model layers is shallow, the random feature mapping makes the sample features cannot be fully learned and utilized; (2) when the number of model layers is deep, the validity of the sample features will decrease after continuous abstraction and generalization. In order to solve the above problems, this paper proposes a densely connected deep ELM algorithm: dense-HELM (D-HELM). Benchmark data sets of different sizes have been employed for the property of the D-HELM algorithm. Compared with the H-ELM algorithm on the benchmark dataset, the average test accuracy is increased by 5.34% and the average training time is decreased by 21.15%. On the NORB dataset, the proposed D-HELM algorithm still maintains the best classification results and the fastest training speed. The D-HELM algorithm can make full use of the features of hidden layer learning by using the densely connected network structure and effectively reduce the number of parameters. Compared with the H-ELM algorithm, the D-HELM algorithm significantly improves the recognition accuracy and accelerates the training speed of the algorithm.

Analysis and Variants of Broad Learning System

The broad learning system (BLS) is designed based on the technology of compressed sensing and pseudo-inverse theory, and consists of feature nodes and enhancement nodes, has been proposed recently. Compared with the popular deep learning structures, such as deep neural networks, BLS has the ability of rapid incremental learning and can remodel the system without the usual tedious retraining process. However, given that BLS is still in its infancy, it still needs analysis, improvements, and verification. In this article, we first analyze the principle of fast incremental learning ability of BLS in depth. Second, in order to provide an in-depth analysis of the BLS structure, according to the novel structure design concept of deep neural networks, we present four brand-new BLS variant networks and their incremental realizations. Third, based on our analysis of the effect of feature nodes and enhancement nodes, a new BLS structure with a semantic feature extraction layer has been proposed, which is called SFEBLS. The experimental results show that SFEBLS and its variants can increase the accuracy rate on the NORB dataset 6.18%, Fashion-MNIST dataset by 3.15%, ORL data by 5.00%, street view house number dataset by 12.88%, and CIFAR-10 dataset by 18.42%, respectively, and the four brand-new BLS variant networks also obviously outperform the original BLS.

Neural network with multiple connection weights

Biological studies have shown that the interaction between neurons are based on neurotransmitters, which transmit signals between neurons, and that one neuron sends information to another neuron by releasing a number of different neurotransmitters, which play different roles. Motivated by this biological discovery, a novel neural networks model is proposed by extending the dimension of connection weights from one to multiple, i.e. there are multiple not only one connections between each two units. The number of dimensions of connection weight represents the number of categories of neurotransmitters and different components of the weight correspond to different neurotransmitters. In order to make these neurotransmitters collaborate and compete appropriately, the input and output for each unit in our proposed model have been heuristically defined. From the biological perspective, the proposed neural network is much closer to biological neural network. From the viewpoint of new model structure, the characteristic that the activation of each hidden unit is based on several filters, can improve the interpretability of features learned by the proposed neural network. Experimental results on MNIST, NORB and several other data sets have demonstrated that the performances of traditional neural networks can be improved by extending the dimension of connection weight between units, and the idea of multiple connection weights provides a new paradigm for the design of neural networks.

A Novel Broad Learning Model-Based Semi-Supervised Image Classification Method

Broad learning system (BLS) is an effective and efficient incremental learning system without the deep architecture. It has strong feature extraction ability and high computational efficiency. However, it is greatly limited in the applicability of supervised learning. For the collected actual data, more data are unlabeled data and less data are labeled data. To overcome these problems, Fick's law assisted propagation (FLAP) is introduced into the BLS to propose a new semi-supervised classification algorithm, namely FLAP-BLS in this paper. In the FLAP-BLS, the FLAP has the labeled ability from the labeled examples to unlabeled examples, it is used to mark plenty of unlabeled samples by few labeled samples in order to obtain a large number of labeled samples and build the sample data matrix. Then an efficient incremental BLS without deep structure can effectively extract features from large-scale data, it is used to effectively classify the sample matrix. Finally, USPS, MNIST and NORB datasets are selected to validate the effectiveness of the FLAP-BLS. The experiment results show that the FLAP-BLS can effectively classify the few labeled samples and a large of unlabeled samples and obtain classification results with high accuracy, and it has faster classification speed, stronger generalization ability and better stability. The proposed method provides a new method for image classification.

Mutual information-based dropout: Learning deep relevant feature representation architectures

We propose a new regularization strategy called DropMI, which is a generalization of Dropout for the regularization of networks that introduces mutual information (MI) dynamic analysis. The standard Dropout randomly drops a certain proportion of neural units, according to the Bernoulli distribution, thereby resulting in the loss of some important hidden feature information. In DropMI, we first evaluate the importance of each neural unit in the feature representation of the hidden layer based on the MI between it and the target. We then construct a new binary mask matrix based on the sorting distribution of MI, thus developing a dynamic DropMI strategy that highlights the important neural units that are beneficial to the feature representation. The results from the MNIST, NORB, CIFAR-10, CIFAR-100, SVHN, and Multi-PIE datasets indicate that, relative to other state-of-the-art regularization methods based on the benchmark autoencoder and convolutional neural networks, our method has better feature representation performance and effectively reduces the overfitting of the model.

<inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math> </inline-formula>-Dropout: A Unified Dropout

Dropout is an effective regularization method for deep learning tasks. Several variants of dropout based on sampling with different distributions have been proposed individually and have shown good generalization performance on various learning tasks. Among these variants, the canonical Bernoulli dropout is a discrete method, while the uniform dropout and the Gaussian dropout are continuous dropout methods. When facing a new learning task, one must make a decision on which method is more suitable, which is somehow unnatural and inconvenient. In this paper, we attempt to change the selection problem to a parameter tuning problem by proposing a general form of dropout, $\beta $ -dropout, to unify the discrete dropout with continuous dropout. We show that by adjusting the shape parameter $\beta $ , the $\beta $ -dropout can yield the Bernoulli dropout, uniform dropout, and approximate Gaussian dropout. Furthermore, it can obtain continuous regularization strength, which paves the way for self-adaptive dropout regularization. As a first attempt, we propose a self-adaptive $\beta $ -dropout, in which the parameter $\beta $ is tuned automatically following a pre-designed strategy. The $\beta $ -dropout is tested extensively on the MNIST, CIFAR-10, SVHN, NORB, and ILSVRC-12 datasets to investigate its superior performance. The results show that the $\beta $ -dropout can conduct finer control of its regularization strength, therefore obtaining better performance.

Improved broad learning system: partial weights modification based on BP algorithm

Although breakthrough achievements of deep learning have been made in different areas, there is no good idea to prevent the time-consuming training process. Single-layer feedforward neural networks (e.g. BLS) are used to reduce the training time. However, with the decrease of training time, the accuracy degradation has emerged. In view of the limitation of random generation of connection parameters between feature nodes and enhancement nodes, this paper presents an algorithm (IBLS) based on BLS and backpropagation algorithm to learn the weights between feature nodes and enhancement nodes. Experiments over NORB and MNIST data sets show that the improved broad learning system achieves acceptable results.

Local receptive fields based extreme learning machine with hybrid filter kernels for image classification

In this paper, an innovative method called extreme learning machine with hybrid local receptive fields (ELM-HLRF) is presented for image classification. In this method, filters generated by Gabor functions and the randomly generated convolution filters are incorporated into the convolution filter kernels of local receptive fields based extreme learning machine (ELM-LRF). Extreme learning machine (ELM) is derived from single hidden layer feed-forward neural networks, and the parameters of its hidden layer can be generated randomly. As locally connected ELM, ELM-LRF directly processes information with strong correlations such as images and speech. In this paper, two main contributions are proposed to improve the classification performance of ELM-LRF. First, the Gabor functions are used as one kind of convolution filter kernels of ELM-HLRF to execute image classification. Second, we use a data augmentation method to preprocess training images to avoid overfitting. Experiments on the Outex texture dataset, the Yale face dataset, the ORL face database and the NORB dataset demonstrate that ELM-HLRF outperforms ELM-LRF, ELM and support vector machine in classification accuracy, and the presented data augmentation method improves the classification performance.

Empirical kernel map-based multilayer extreme learning machines for representation learning

Abstract Recently, multilayer extreme learning machine (ML-ELM) and hierarchical extreme learning machine (H-ELM) were developed for representation learning whose training time can be reduced from hours to seconds compared to traditional stacked autoencoder (SAE). However, there are three practical issues in ML-ELM and H-ELM: (1) the random projection in every layer leads to unstable and suboptimal performance; (2) the manual tuning of the number of hidden nodes in every layer is time-consuming; and (3) under large hidden layer, the training time becomes relatively slow and a large storage is necessary. More recently, issues (1) and (2) have been resolved by kernel method, namely, multilayer kernel ELM (ML-KELM), which encodes the hidden layer in form of a kernel matrix (computed by using kernel function on the input data), but the storage and computation issues for kernel matrix pose a big challenge in large-scale application. In this paper, we empirically show that these issues can be alleviated by encoding the hidden layer in the form of an approximate empirical kernel map (EKM) computed from low-rank approximation of the kernel matrix. This proposed method is called ML-EKM-ELM, whose contributions are: (1) stable and better performance is achieved under no random projection mechanism; (2) the exhaustive manual tuning on the number of hidden nodes in every layer is eliminated; (3) EKM is scalable and produces a much smaller hidden layer for fast training and low memory storage, thereby suitable for large-scale problems. Experimental results on benchmark datasets demonstrated the effectiveness of the proposed ML-EKM-ELM. As an illustrative example, on the NORB dataset, ML-EKM-ELM can be respectively up to 16 times and 37 times faster than ML-KELM for training and testing with a little loss of accuracy of 0.35%, while the memory storage can be reduced up to 1/9.

Appropriateness of Dropout Layers and Allocation of Their 0.5 Rates across Convolutional Neural Networks for CIFAR-10, EEACL26, and NORB Datasets

Abstract A technique of DropOut for preventing overfitting of convolutional neural networks for image classification is considered in the paper. The goal is to find a rule of rationally allocating DropOut layers of 0.5 rate to maximise performance. To achieve the goal, two common network architectures are used having either 4 or 5 convolutional layers. Benchmarking is fulfilled with CIFAR-10, EEACL26, and NORB datasets. Initially, series of all admissible versions for allocation of DropOut layers are generated. After the performance against the series is evaluated, normalized and averaged, the compromising rule is found. It consists in non-compactly inserting a few DropOut layers before the last convolutional layer. It is likely that the scheme with two or more DropOut layers fits networks of many convolutional layers for image classification problems with a plenty of features. Such a scheme shall also fit simple datasets prone to overfitting. In fact, the rule “prefers” a fewer number of DropOut layers. The exemplary gain of the rule application is roughly between 10 % and 50 %.

Appropriate Number of Standard 2 × 2 Max Pooling Layers and Their Allocation in Convolutional Neural Networks for Diverse and Heterogeneous Datasets

A problem of appropriately allocating pooling layers in convolutional neural networks is considered. The consideration is based on CIFAR-10, NORB, and EEACL26 datasets for preventing “overfitting” in a solution of the problem. For highly accurate image recognition within these datasets, the networks are used with the max pooling operation. The most common form of such operation, which is a 2 ´ 2 pooling layer, is applied with a stride of 2 without padding after convolutional layers. Based on performance against a series of the network architectures, a rule for the best allocation of max pooling layers is formulated. The rule is to insert a few pooling layers after the starting convolutional layers and to insert a one pooling layer after the last but one convolutional layer (“11...100...010”). For much simpler datasets, the best allocation is “11...100...0”.

Extreme learning machine with multi-scale local receptive fields for texture classification

Texture classification is a challenging task due to the wide range of natural texture types and large intra-class variations in texture images, such as different rotations, scales, positions and lighting conditions. Many existing methods for extracting texture features are designed carefully by user for specific applications. The extracted texture features are then used as input to various classification methods, such as support vector machines, to classify the textures. The system performance greatly depends on the feature extractor. Unfortunately, there is no systematic approach for feature extractor design. In this paper, we propose a method called extreme learning machine with multi-scale local receptive fields (ELM-MSLRF) to achieve feature learning and classification simultaneously for texture classification. In contrast to traditional methods, the proposed method learns the features by the network itself and can be applied to more general applications. Additionally, it is fast and requires few computations. Experiments on the ALOT texture dataset demonstrate the attractive performance of ELM-MSLRF even compared with the state-of-the-art algorithms. Moreover, the proposed ELM-MSLRF achieves the best performance on the NORB dataset.

Local Receptive Fields Based Extreme Learning Machine

Extreme learning machine (ELM), which was originally proposed for generalized single-hidden layer feedforward neural networks (SLFNs), provides efficient unified learning solutions for the applications of feature learning, clustering, regression and classification. Different from the common understanding and tenet that hidden neurons of neural networks need to be iteratively adjusted during training stage, ELM theories show that hidden neurons are important but need not be iteratively tuned. In fact, all the parameters of hidden nodes can be independent of training samples and randomly generated according to any continuous probability distribution. And the obtained ELM networks satisfy universal approximation and classification capability. The fully connected ELM architecture has been extensively studied. However, ELM with local connections has not attracted much research attention yet. This paper studies the general architecture of locally connected ELM, showing that: 1) ELM theories are naturally valid for local connections, thus introducing local receptive fields to the input layer; 2) each hidden node in ELM can be a combination of several hidden nodes (a subnetwork), which is also consistent with ELM theories. ELM theories may shed a light on the research of different local receptive fields including true biological receptive fields of which the exact shapes and formula may be unknown to human beings. As a specific example of such general architectures, random convolutional nodes and a pooling structure are implemented in this paper. Experimental results on the NORB dataset, a benchmark for object recognition, show that compared with conventional deep learning solutions, the proposed local receptive fields based ELM (ELM-LRF) reduces the error rate from 6.5% to 2.7% and increases the learning speed up to 200 times.

Generic Object Recognition with Local Receptive Fields Based Extreme Learning Machine

Abstract Generic object recognition is to classify the object to a generic category. Intra-class variabilities cause big troubles for this task. Traditional methods involve plenty of pre-processing steps, like model construction, feature extraction, etc. Moreover, these methods are only effective for some specific dataset. In this paper, we propose to use local receptive fields based extreme learning machine (ELM-LRF) as a general framework for object recognition. It is operated directly on the raw images and thus suitable for all different datasets. Additionally, the architecture is simple and only requires few computations, as most connection weights are randomly generated. Comparing to state-of-the-art results on NORB, ETH-80 and COIL datasets, it is on par with the best one on ETH-80 and sets the new records for NORB and COIL.

Expected energy-based restricted Boltzmann machine for classification

In classification tasks, restricted Boltzmann machines (RBMs) have predominantly been used in the first stage, either as feature extractors or to provide initialization of neural networks. In this study, we propose a discriminative learning approach to provide a self-contained RBM method for classification, inspired by free-energy based function approximation (FE-RBM), originally proposed for reinforcement learning. For classification, the FE-RBM method computes the output for an input vector and a class vector by the negative free energy of an RBM. Learning is achieved by stochastic gradient-descent using a mean-squared error training objective. In an earlier study, we demonstrated that the performance and the robustness of FE-RBM function approximation can be improved by scaling the free energy by a constant that is related to the size of network. In this study, we propose that the learning performance of RBM function approximation can be further improved by computing the output by the negative expected energy (EE-RBM), instead of the negative free energy. To create a deep learning architecture, we stack several RBMs on top of each other. We also connect the class nodes to all hidden layers to try to improve the performance even further. We validate the classification performance of EE-RBM using the MNIST data set and the NORB data set, achieving competitive performance compared with other classifiers such as standard neural networks, deep belief networks, classification RBMs, and support vector machines. The purpose of using the NORB data set is to demonstrate that EE-RBM with binary input nodes can achieve high performance in the continuous input domain.

Discriminative restricted Boltzmann machine for invariant pattern recognition with linear transformations

How to make a machine automatically achieve invariant pattern recognition like human brain is still very challenging in machine learning community. In this paper, we present a single hidden-layer network TIClassRBM for invariant pattern recognition by incorporating linear transformations into discriminative restricted Boltzmann machine. In our model, invariant feature extraction and pattern classification can be implemented simultaneously. The mapping from input features to class label is represented by two groups of weights: transformed weights that connect hidden units to data, and pooling weights that connect pooling units yielded by probabilistic max-pooling to class label. All weights play an important role in the invariant pattern recognition. Moreover, TIClassRBM can handle general transformations contained in images, such as translation, rotation and scaling. The experimental studies on the variations of MNIST and NORB datasets demonstrate that the proposed model yields the best performance among some comparative models.

An Efficient Learning Procedure for Deep Boltzmann Machines

We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer pretraining phase that initializes the weights sensibly. The pretraining also allows the variational inference to be initialized sensibly with a single bottom-up pass. We present results on the MNIST and NORB data sets showing that deep Boltzmann machines learn very good generative models of handwritten digits and 3D objects. We also show that the features discovered by deep Boltzmann machines are a very effective way to initialize the hidden layers of feedforward neural nets, which are then discriminatively fine-tuned.