Fast Training Process Research Articles

Sparsity in optimal randomized classification trees

Decision trees are popular Classification and Regression tools and, when small-sized, easy to interpret. Traditionally, a greedy approach has been used to build the trees, yielding a very fast training process; however, controlling sparsity (a proxy for interpretability) is challenging. In recent studies, optimal decision trees, where all decisions are optimized simultaneously, have shown a better learning performance, especially when oblique cuts are implemented. In this paper, we propose a continuous optimization approach to build sparse optimal classification trees, based on oblique cuts, with the aim of using fewer predictor variables in the cuts as well as along the whole tree. Both types of sparsity, namely local and global, are modeled by means of regularizations with polyhedral norms. The computational experience reported supports the usefulness of our methodology. In all our data sets, local and global sparsity can be improved without harming classification accuracy. Unlike greedy approaches, our ability to easily trade in some of our classification accuracy for a gain in global sparsity is shown.

Tensorizing Restricted Boltzmann Machine

Restricted Boltzmann machine (RBM) is a famous model for feature extraction and can be used as an initializer for neural networks. When applying the classic RBM to multidimensional data such as 2D/3D tensors, one needs to vectorize such as high-order data. Vectorizing will result in dimensional disaster and valuable spatial information loss. As RBM is a model with fully connected layers, it requires a large amount of memory. Therefore, it is difficult to use RBM with high-order data on low-end devices. In this article, to utilize classic RBM on tensorial data directly, we propose a new tensorial RBM model parameterized by the tensor train format (TTRBM). In this model, both visible and hidden variables are in tensorial form, which are connected by a parameter matrix in tensor train format. The biggest advantage of the proposed model is that TTRBM can obtain comparable performance compared with the classic RBM with much fewer model parameters and faster training process. To demonstrate the advantages of TTRBM, we conduct three real-world applications, face reconstruction, handwritten digit recognition, and image super-resolution in the experiments.

Trainable Projected Gradient Detector for Massive Overloaded MIMO Channels: Data-Driven Tuning Approach

This paper presents a deep learning-aided iterative detection algorithm for massive overloaded multiple-input multiple-output (MIMO) systems where the number of transmit antennas $n$ is larger than that of receive antennas $m$. Since the proposed algorithm is based on the projected gradient descent method with trainable parameters, it is named the trainable projected gradient-detector (TPG-detector). The trainable internal parameters, such as the step-size parameter, can be optimized with standard deep learning techniques, i.e., the back propagation and stochastic gradient descent algorithms. This approach is referred to as data-driven tuning, and ensures fast convergence during parameter estimation in the proposed scheme. The TPG-detector mainly consists of matrix-vector product operations whose computational cost is proportional to $m n$ for each iteration. In addition, the number of trainable parameters in the TPG-detector is independent of the number of antennas. These features of the TPG-detector result in a fast and stable training process and reasonable scalability for large systems. Numerical simulations show that the proposed detector achieves a comparable detection performance to those of existing algorithms for massive overloaded MIMO channels, e.g., the state-of-the-art IW-SOAV detector, with a lower computation cost.

Neural Networks for Predicting Prices of Sugarcane Derivatives

Brazil is one of the largest sugarcane producers. It directly affects job creation and national gross domestic product, as well as bringing a large amount of foreign money to the country. Hence, this paper proposes investigating the application of neural networks—multilayer perceptron, extreme learning machines (ELM) and echo state networks (ESN)—for predicting the following cane derivatives prices: sugar, hydrous ethanol, and anhydrous ethanol. The main characteristic of ELM and ESN is their simple and fast training process, being based on the analytic calculation of the coefficients of a linear combiner. Their intermediate layer stands untuned, and the weights in this layer are randomly and independently defined. The computational results show that the application of the ELM achieved the best overall results, showing that they are viable candidates for these kinds of problems.

ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation

A user can be represented as what he/she does along the history. A common way to deal with the user modeling problem is to manually extract all kinds of aggregated features over the heterogeneous behaviors, which may fail to fully represent the data itself due to limited human instinct. Recent works usually use RNN-based methods to give an overall embedding of a behavior sequence, which then could be exploited by the downstream applications. However, this can only preserve very limited information, or aggregated memories of a person. When a downstream application requires to facilitate the modeled user features, it may lose the integrity of the specific highly correlated behavior of the user, and introduce noises derived from unrelated behaviors. This paper proposes an attention based user behavior modeling framework called ATRank, which we mainly use for recommendation tasks. Heterogeneous user behaviors are considered in our model that we project all types of behaviors into multiple latent semantic spaces, where influence can be made among the behaviors via self-attention. Downstream applications then can use the user behavior vectors via vanilla attention. Experiments show that ATRank can achieve better performance and faster training process. We further explore ATRank to use one unified model to predict different types of user behaviors at the same time, showing a comparable performance with the highly optimized individual models.

Semi-active control of magnetorheological elastomer base isolation system utilising learning-based inverse model

Magnetorheological elastomer (MRE) base isolations have attracted considerable attention over the last two decades thanks to its self-adaptability and high-authority controllability in semi-active control realm. Due to the inherent nonlinearity and hysteresis of the devices, it is challenging to obtain a reasonably complicated mathematical model to describe the inverse dynamics of MRE base isolators and hence to realise control synthesis of the MRE base isolation system. Two aims have been achieved in this paper: i) development of an inverse model for MRE base isolator based on optimal general regression neural network (GRNN); ii) numerical and experimental validation of a real-time semi-active controlled MRE base isolation system utilising LQR controller and GRNN inverse model. The superiority of GRNN inverse model lays in fewer input variables requirement, faster training process and prompt calculation response, which makes it suitable for online training and real-time control. The control system is integrated with a three-storey shear building model and control performance of the MRE base isolation system is compared with bare building, passive-on isolation system and passive-off isolation system. Testing results show that the proposed GRNN inverse model is able to reproduce desired control force accurately and the MRE base isolation system can effectively suppress the structural responses when compared to the passive isolation system.

A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture

A novel self-learning optimal control method for a class of discrete-time nonlinear systems is proposed based on iteration adaptive dynamic programming (ADP) algorithm. It is proven that the iteration costate functions converge to the optimal one, and a detailed convergence analysis of the iteration ADP algorithm is given. Furthermore, echo state network (ESN) architecture is used as the approximator of the costate function for each iteration. To ensure the reliability of the ESN approximator, the ESN mean square training error is constrained in the satisfactory range. Two simulation examples are given to demonstrate that the proposed control method has a fast response speed due to the special structure and the fast training process.

A probabilistic artificial neural network-based procedure for variance change point estimation

Control charts are useful tools of monitoring quality characteristics. One of the problems of employing a control chart is that the time it alarms is not synchronic with the time when assignable cause manifests itself in the process. This makes difficult to search and find assignable causes. Knowing the real time of manifestation of assignable cause (change point) helps to find assignable cause(s) sooner and eases corrective actions to be taken. In this paper, a probabilistic neural network (PNN)-based procedure was developed to estimate the variance change point of a normally distributed quality characteristic. The PNN was selected based on trial and error among different types of artificial neural networks and on the basis of its advantages such as fast training process, converging to optimal classifier and adding or removing samples without extensive retraining. In the proposed procedure, the signal is first received by an $$S^{2}$$ S 2 control chart and then based on the designed tests of hypothesis, which distinguish the size of shift in the variance, a suitable PNN is activated. The performance of the proposed procedure is evaluated through extensive simulation studies. In addition, the results of a comparison study with the maximum likelihood estimation (MLE) method show that the proposed procedure outperforms MLE in estimating the real time of the step change in variance of a normal quality characteristic. Finally, an illustrative example is presented to clarify the procedure step by step.

Using Radial Basis Function Networks for Function Approximation and Classification

The radial basis function (RBF) network has its foundation in the conventional approximation theory. It has the capability of universal approximation. The RBF network is a popular alternative to the well-known multilayer perceptron (MLP), since it has a simpler structure and a much faster training process. In this paper, we give a comprehensive survey on the RBF network and its learning. Many aspects associated with the RBF network, such as network structure, universal approimation capability, radial basis functions, RBF network learning, structure optimization, normalized RBF networks, application to dynamic system modeling, and nonlinear complex-valued signal processing, are described. We also compare the features and capability of the two models.

Application of echo state networks in short-term electric load forecasting

The paper presents the application of echo state network (ESN) to short-term load forecasting (STLF) problem in power systems for both 1-h and 24-h ahead predictions while using the least number of inputs: current-hour load, predicted target-hour temperature, and only for 24-h ahead forecasting, day-type index. The study is much attractive due to inclusion of weekends/holidays what makes STLF problem much more difficult. The main aim is to show the great capabilities of ESN as a stand-alone forecaster to learn complex dynamics of hourly electric load time series and forecast the near future loads with high accuracies. ESN as the state-of-the-art recurrent neural network (RNN) gains a reservoir of dynamics tapped by trained output units with a simple and fast single-stage training process. Furthermore, the application of ESN to predict the target-hour temperature needed by ESN-based load forecasters is examined. Since temperature prediction errors affect load forecasting accuracy, effects of such errors on ESN-based load forecasting are studied by both sensitivity analysis and applying noisy temperature series. Real hourly load and temperature data of a North-American electric utility is used as the data set. The results reflect that the ESN-based STLF method provides load forecasts with acceptable high accuracy.

An interpretable fuzzy rule-based classification methodology for medical diagnosis

The aim of this paper is to present a novel fuzzy classification framework for the automatic extraction of fuzzy rules from labeled numerical data, for the development of efficient medical diagnosis systems. The proposed methodology focuses on the accuracy and interpretability of the generated knowledge that is produced by an iterative, flexible and meaningful input partitioning mechanism. The generated hierarchical fuzzy rule structure is composed by linguistic; multiple consequent fuzzy rules that considerably affect the model comprehensibility. The performance of the proposed method is tested on three medical pattern classification problems and the obtained results are compared against other existing methods. It is shown that the proposed variable input partitioning leads to a flexible decision making framework and fairly accurate results with a small number of rules and a simple, fast and robust training process.

Radial Basis Function Networks GPU-Based Implementation

Neural networks (NNs) have been used in several areas, showing their potential but also their limitations. One of the main limitations is the long time required for the training process; this is not useful in the case of a fast training process being required to respond to changes in the application domain. A possible way to accelerate the learning process of an NN is to implement it in hardware, but due to the high cost and the reduced flexibility of the original central processing unit (CPU) implementation, this solution is often not chosen. Recently, the power of the graphic processing unit (GPU), on the market, has increased and it has started to be used in many applications. In particular, a kind of NN named radial basis function network (RBFN) has been used extensively, proving its power. However, their limiting time performances reduce their application in many areas. In this brief paper, we describe a GPU implementation of the entire learning process of an RBFN showing the ability to reduce the computational cost by about two orders of magnitude with respect to its CPU implementation.