Online RBM: Growing Restricted Boltzmann Machine on the fly for unsupervised representation

Abstract

In this work, we endeavor to investigate and propose a novel unsupervised online learning algorithm, namely the Online Restricted Boltzmann Machine (O-RBM). The O-RBM is able to construct and adapt the architecture of a Restricted Boltzmann Machine (RBM) artificial neural network, according to the statistics of the streaming input data. Specifically, for a training data that is not fully available at the onset of training, the proposed O-RBM begins with a single neuron in the hidden layer of the RBM, progressively adds and suitably adapts the network to account for the variations in streaming data distributions. Such an unsupervised learning helps to effectively model the probability distribution of the entire data stream, and generates robust features. We will demonstrate that such unsupervised representations can be used for discriminative classifications on a set of multi-category and binary classification problems for unstructured image and structured signal data sets, having varying degrees of class-imbalance. We first demonstrate the O-RBM algorithm and characterize the network evolution using the simple and conventional multi-class MNIST image dataset, aimed at recognizing hand-written digit. We then benchmark O-RBM performance to other machine learning, neural network and Class RBM techniques using a number of public non-stationary datasets. Finally, we study the performance of the O-RBM on a real-world problem involving predictive maintenance of an aircraft component using time series data. In all these studies, it is observed that the O-RBM converges to a stable, concise network architecture, wherein individual neurons are inherently discriminative to the class labels despite unsupervised training. It can be observed from the performance results that on an average O-RBM improves accuracy by 2.5%–3% over conventional offline batch learning techniques while requiring at least 24%–70% fewer neurons.