Fast Training Algorithm Research Articles

A Space-Efficient One-Pass Online SVM Algorithm

In this paper we consider the problem of training a Support Vector Machine (SVM) online using a stream of data in random order. We provide a fast online training algorithm for general SVM on very large datasets. Based on the geometric interpretation of SVM known as the polytope distance, our algorithm uses a gradient descent procedure to solve the problem. With high probability our algorithm outputs an [Formula: see text]-approximation result in constant time and space, which is independent of the size of the dataset, where [Formula: see text]-approximation means that the separating margin of the classifier is almost optimal (with error [Formula: see text]), and the number of misclassified training points is very small (with error [Formula: see text]). Experimental results show that our algorithm outperforms most of existing online algorithms, especially in the space requirement aspect, while maintaining high accuracy.

Simulation Environment of Unmanned Group Reconnaissance System Based on RVIZ and GYM

This paper studies the simulation environment of the current cluster unmanned system, analyzes the shortcomings of the cluster unmanned system, and designs a cluster robot simulation system based on RVIZ and GYM, which has both the fast training algorithm and the physical attributes of the simulation environment, and uses DDS to transmit robot control instructions and state information. After the GYM fast training algorithm, Control the cluster robots to verify the algorithm and collect the state information of the robots in the physical simulation environment of RVIZ. After the whole simulation is passed, the algorithm can be directly used for the path planning of cluster robots. The results show that the simulation environment can effectively simulate the unmanned group reconnaissance system.

NARX Deep Convolutional Fuzzy System for Modelling Nonlinear Dynamic Processes

This paper presents a new approach for modelling nonlinear dynamic processes (NDP). It is based on a nonlinear autoregressive with exogenous (NARX) inputs model structure and a deep convolutional fuzzy system (DCFS). The DCFS is a hierarchical fuzzy structure, which can overcome the deficiency of general fuzzy systems when facing high dimensional data. For relieving the curse of dimensionality, as well as improving approximation performance of fuzzy models, we propose combining the NARX with the DCFS to provide a good approximation of the complex nonlinear dynamic behavior and a fast-training algorithm with ensured convergence. There are three NARX DCFS structures proposed, and the appropriate training algorithm is adapted. Evaluations were performed on a popular benchmark—Box and Jenkin’s gas furnace data set and the four nonlinear dynamic test systems. The experiments show that the proposed NARX DCFS method can be successfully used to identify nonlinear dynamic systems based on external dynamics structures and nonlinear static approximators.

FST-EfficientNetV2: Exceptional Image Classification for Remote Sensing

Recently, the semantic classification (SC) algorithm for remote sensing images (RSI) has been greatly improved by deep learning (DL) techniques, e.g., deep convolutional neural networks (CNNs). However, too many methods employ complex procedures (e.g., multi-stages), excessive hardware budgets (e.g., multi-models), and an extreme reliance on domain knowledge (e.g., handcrafted features) for the pure purpose of improving accuracy. It obviously goes against the superiority of DL, i.e., simplicity and automation. Meanwhile, these algorithms come with unnecessarily expensive overhead on parameters and hardware costs. As a solution, the author proposed a fast and simple training algorithm based on the smallest architecture of EfficientNet version 2, which is called FST-EfficientNet. The approach employs a routine transfer learning strategy and has fast training characteristics. It outperforms all the former methods by a 0.8%–2.7% increase in accuracy. It does, however, use a higher testing resolution of 5122 and 6002, which results in high consumption of graphics processing units (GPUs). As an upgrade option, the author proposes a novel and more efficient method named FST-EfficientNetV2 as the successor. The new algorithm still employs a routine transfer learning strategy and maintains fast training characteristics. But a set of crucial algorithmic tweaks and hyperparameter re-optimizations have been updated. As a result, it achieves a noticeable increase in accuracy of 0.3%–1.1% over its predecessor. More importantly, the algorithm’s GPU costs are reduced by 75%–81%, with a significant reduction in training time costs of 60%–80%. The results demonstrate that an efficient training optimization strategy can significantly boost the CNN algorithm's performance for RSI-SC. More crucially, the results prove that the distribution shift introduced by data augmentation (DA) techniques is vital to the method’s performance for RSI-SC, which has been ignored to date. These findings may help us gain a correct understanding of the CNN algorithm for RSI-SC.

A fast conjugate functional gain sequential minimal optimization training algorithm for LS-SVM model

A fast conjugate functional gain sequential minimal optimization training algorithm for LS-SVM model

Efficient Neuromorphic Hardware Through Spiking Temporal Online Local Learning

Local learning schemes have shown promising performance in spiking neural networks (SNNs) training and are considered a step toward more biologically plausible learning. Despite many efforts to design high-performance neuromorphic systems, a fast and efficient on-chip training algorithm is still missing, which limits the deployment of neuromorphic systems in many real-time applications. This work proposes a scalable, fast, and efficient spiking neuromorphic hardware system with on-chip local learning capability. We introduce an effective hardware-friendly local training algorithm compatible with sparse temporal input coding and binary random classification weights. The algorithm is demonstrated to deliver competitive accuracy in different tasks. The proposed digital system explores spike sparsity in communication, parallelism in vector–matrix operations and process-level dataflow, and locality of training errors, which leads to low cost and fast training speed. The system is optimized under various performance metrics. Taking into consideration energy, speed, resources, and accuracy, the proposed method shows around <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$10\times $ </tex-math></inline-formula> efficiency over a recent work with a direct feedback alignment (DFA) method and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.5\times $ </tex-math></inline-formula> efficiency over the spike-timing-dependent plasticity (STDP) method. Moreover, our hardware architecture can easily scale up with the network size at a linear rate. Thus, our method has demonstrated great potential for use in various applications, especially those demanding low latency.

A Novel Fast Feedforward Neural Networks Training Algorithm

Abstract In this paper1 a new neural networks training algorithm is presented. The algorithm originates from the Recursive Least Squares (RLS) method commonly used in adaptive filtering. It uses the QR decomposition in conjunction with the Givens rotations for solving a normal equation - resulting from minimization of the loss function. An important parameter in neural networks is training time. Many commonly used algorithms require a big number of iterations in order to achieve a satisfactory outcome while other algorithms are effective only for small neural networks. The proposed solution is characterized by a very short convergence time compared to the well-known backpropagation method and its variants. The paper contains a complete mathematical derivation of the proposed algorithm. There are presented extensive simulation results using various benchmarks including function approximation, classification, encoder, and parity problems. Obtained results show the advantages of the featured algorithm which outperforms commonly used recent state-of-the-art neural networks training algorithms, including the Adam optimizer and the Nesterov’s accelerated gradient.

Forecasting of COVID 19 Cases in Karnataka State using Artificial Neural Network (ANN)

COVID-19 is a pandemic that has caused lot of deaths and infections in the last 2 months and is showing an increasing trend not only in the number of infections and deaths, but also in the recovery rate. Accurate prediction models are very much essential to make proper forecasts and take necessary actions. This study demonstrates the capability of Multilayer Perceptron (MLP), an Artificial Neural network (ANN) model for forecasting the number of infected cases in the state of Karnataka in India. It is trained using a fast training algorithm namely, Extreme Learning machine to reduce the training time required. The parameters required for the forecasting model have been selected using partial autocorrelation function, which is a conventional method, and its performance has been compared with parameters selected using cuckoo search (CS) algorithm, which is a very popular metaheuristic optimization algorithm. The testing of the forecasting model has been done, and comparison between the two parameter selection methods as well as with MLP with conventional backpropagation has been carried out. Use of CS algorithm has resulted in a better forecasting performance based on mean absolute percentage error (MAPE), with a value of 6.62% on training data and 7.03% on the test data. Further to check the efficacy of the model, the data of COVID-19 cases of Hungary from 4 March to 19 April 2020 have been used, which resulted in a MAPE of 1.55%, thereby establishing the robustness of the proposed ANN model for forecasting COVID-19 cases for the state of Karnataka.

Fast Training of Adversarial Deep Fuzzy Classifier by Downsizing Fuzzy Rules With Gradient Guided Learning

While our recent deep fuzzy classifier DSA-FC, which stacks adversarial interpretable Takagi–Sugeno–Kang fuzzy subclassifiers, shares its promising classification, its training speed will become very slow and even intolerable for large-scale datasets, due to successive training on all training samples with their random gradient based updates along each layer of its stacked structure. In order to circumvent this bottleneck issue, a fast training algorithm FTA is developed in this study by downsizing fuzzy rules with the proposed gradient guided learning for each subclassifier at each layer of DSA-FC on large-scale datasets. The core of FTA is to assure fast training of each subclassifier at each layer of DSA-FC, which first generates first-order smooth gradient guided information by means of the proposed top- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> fuzzy rules selected from all fuzzy rules in each subclassifier, and then quickly updates the current inputs in terms of such information, which will be taken as the inputs of the subclassifier at the next layer. Our theoretical analysis reveals that the proposed gradient guided learning indeed enhances the generalization capability of a deep fuzzy classifier with or without adversarial attacks on outputs. Experimental results on large datasets demonstrate that FTA indeed trains the deep fuzzy classifier DSA-FC quickly with enhanced generalization capability.

A fast saddle-point dynamical system approach to robust deep learning

Recent focus on robustness to adversarial attacks for deep neural networks produced a large variety of algorithms for training robust models. Most of the effective algorithms involve solving the min–max optimization problem for training robust models (min step) under worst-case attacks (max step). However, they often suffer from high computational cost from running several inner maximization iterations (to find an optimal attack) inside every outer minimization iteration. Therefore, it becomes difficult to readily apply such algorithms for moderate to large size real world data sets. To alleviate this, we explore the effectiveness of iterative descent–ascent algorithms where the maximization and minimization steps are executed in an alternate fashion to simultaneously obtain the worst-case attack and the corresponding robust model. Specifically, we propose a novel discrete-time dynamical system-based algorithm that aims to find the saddle point of a min–max optimization problem in the presence of uncertainties. Under the assumptions that the cost function is convex and uncertainties enter concavely in the robust learning problem, we analytically show that our algorithm converges asymptotically to the robust optimal solution under a general adversarial budget constraints as induced by ℓp norm, for 1≤p≤∞. Based on our proposed analysis, we devise a fast robust training algorithm for deep neural networks. Although such training involves highly non-convex robust optimization problems, empirical results show that the algorithm can achieve significant robustness compared to other state-of-the-art robust models on benchmark data sets.

Two-Stage 3D Codebook Design and Fast Beam Search Algorithm for Millimeter-Wave Massive MIMO Systems

For 3D beam access in massive multi-input multi-output (MIMO) systems, this paper proposes a feasible two-stage 3D codebook design and corresponding fast beam training algorithm. The complete codebook consists of a primary codebook and an auxiliary codebook—the primary codebook produces a basic directional beam using a smaller number of phase shifts to maintain lower hardware complexity. The small auxiliary codebook creates finer beams centered on each main beam. In addition, a beam search scheme for two-level codebook design is proposed. The main beam search is performed using a modified binary search algorithm based on the signal-to-noise ratio threshold; with the refined auxiliary codebook, the auxiliary beam search is completed using a basic binary search. The simulation results verify the performance advantages of our proposed codebook in terms of beamforming gain and beam access accuracy in MIMO systems. In addition, the comparison of beam training complexity shows that our specially designed two-stage beam search scheme has significant advantages over other existing solutions.

Prediction of Daily Global Solar Radiation using Resilient-propagation Artificial Neural Network and Historical Data: A Case Study of Hail, Saudi Arabia

In this paper, several different Feed Forward Artificial Neural Networks (FFANNs) were used for forecasting the one-day-ahead Global Horizontal Irradiation (GHI) in Hail region, Saudi Arabia. The main motivation behind predicting GHI is that it is a critical parameter in sizing and planning photovoltaic water pumping systems. The novelty of the proposed approach is that it employs only the historical values of the GHI itself as explanatory variables and a fast training algorithm (resilient-propagation). In terms of performance metrics, the rp-trained FFANNs provided better results than Quasi-Newton (bfg) algorithm trained FFANNs for almost all the studied combinations of the FFANN structure. It has been also shown that increasing the number of neurons per layer didn’t improve the performance. Medium structures with fast training algorithms are recommended.

Fast Training Algorithms for Deep Convolutional Fuzzy Systems with Application to Stock Index Prediction

A deep convolutional fuzzy system (DCFS) on a high-dimensional input space is a multilayer connection of many low-dimensional fuzzy systems, where the input variables to the low-dimensional fuzzy systems are selected through a moving window across the input spaces of the layers. To design the DCFS based on an input–output data pairs, we propose a bottom–up layer-by-layer scheme. Specifically, by viewing each of the first-layer fuzzy systems as a weak estimator of the output based only on a very small portion of the input variables, we design these fuzzy systems using the Wang–Mendel method. After the first-layer fuzzy systems are designed, we pass the data through the first layer to form a new dataset and design the second-layer fuzzy systems based on this new dataset in the same way as designing the first-layer fuzzy systems. Repeating this process layer-by-layer, we design the whole DCFS. We also propose a DCFS with parameter sharing to save memory and computation. We apply the DCFS models to predict a synthetic chaotic plus random time-series and the real Hang Seng Index of the Hong Kong stock market.

Sparse low-rank separated representation models for learning from data.

We consider the problem of learning a multivariate function from a set of scattered observations using a sparse low-rank separated representation (SSR) model. The model structure considered here is promising for high-dimensional learning problems; however, existing training algorithms based on alternating least-squares (ALS) are known to have convergence difficulties, particularly when the rank of the model is greater than 1. In the present work, we supplement the model structure with sparsity constraints to ensure the well posedness of the approximation problem. We propose two fast training algorithms to estimate the model parameters: (i) a cyclic coordinate descent algorithm and (ii) a block coordinate descent (BCD) algorithm. While the first algorithm is not provably convergent owing to the non-convexity of the optimization problem, the BCD algorithm guarantees convergence to a Nash equilibrium point. The computational cost of the proposed algorithms is shown to scale linearly with respect to all of the parameters in contrast to methods based on ALS. Numerical studies on synthetic and real-world regression datasets indicate that the proposed SSR model structure holds significant potential for machine learning problems.

FANCFIS: Fast adaptive neuro-complex fuzzy inference system

Large-scale multivariate time series forecasting is an increasingly important problem, with sensor networks pouring out sampled data at unprecedented rates, trillions of dollar's worth of stock trades made every data, and zettabytes of traffic transmitted over the Internet every year. While they are only examples of the much larger domain of general data streams, the uniformly-sampled time series still remains a very large and important subdomain. Extensive research has shown that machine-learning algorithms can often be very effective forecasting models, but many of these algorithms do not scale well. The Adaptive Neuro-Complex-Fuzzy Inferential System is one such approach; built to leverage complex fuzzy sets, it is both an accurate and parsimonious forecasting algorithm. However, its scaling is poor due to a relatively slow training algorithm (gradient descent hybridized with chaotic simulated annealing). Before the algorithm can be used for large-scale forecasting, a fast training algorithm that preserves the system's accuracy and compactness must be developed.We propose the Fast Adaptive Neuro-Complex Fuzzy Inference System, which is designed for fast training of a compact, accurate forecasting model. We use the Fast Fourier Transform algorithm to identify the dominant frequencies in a time series, and then create complex fuzzy sets to match them as the antecedents of complex fuzzy rules. Consequent linear functions are then learned via recursive least-squares. We evaluate this algorithm on both univariate and multivariate time series, finding that this incremental-learning algorithm is as accurate and compact as its slower predecessor, and can be trained much more quickly.

Sensitivity analysis of parameters affecting scour depth around bridge piers based on the non-tuned, rapid extreme learning machine method

The extreme learning machine (ELM) is a new, non-tuned and fast training algorithm for feedforward neural networks (FFNN). It is highly precise and randomly produces the input weights of single-layer FFNN. In the current study, the scour depth around bridge piers is predicted by ELM as a powerful method of nonlinear system modeling. To predict scour depth, the effective dimensionless parameters are determined through dimensional analysis. Due to the complexity of scour mechanisms around bridges, different models with diverse input numbers are presented. In 5 categories, 31 different models were obtained for modeling and ELM analysis. Following the training and validation of each model presented, the optimum model was selected from each of the 5 categories and its relationship to the respective category was identified to help determine scour depth in practical engineering. For the best models presented in the different input modes, new explicit expressions were deduced. The results show that the most important parameters affecting relative scour depth (ds/y) include ratio of pier width to flow depth (D/y) and ratio of pier length to flow depth (L/y) (RMSE = 0.08; MARE = 0.0.35). The ELM performance was compared for a range of pier geometries with regression-based equations. The results confirm that ELM outperforms other methods.

Surrogate Modelling in (and of) Agent-Based Models: A Prospectus

A very timely issue for economic agent-based models (ABMs) is their empirical estimation. This paper describes a line of research that could resolve the issue by using machine learning techniques, using multi-layer artificial neural networks (ANNs), or so called Deep Nets. The seminal contribution by Hinton et al. (Neural Comput 18(7):1527–1554, 2006) introduced a fast and efficient training algorithm called Deep Learning, and there have been major breakthroughs in machine learning ever since. Economics has not yet benefited from these developments, and therefore we believe that now is the right time to apply multi-layered ANNs and Deep Learning to ABMs in economics.

Accelerating the Construction of Neural Network Potential Energy Surfaces: A Fast Hybrid Training Algorithm

Machine learning approaches have been promising in constructing high-dimensional potential energy surfaces (PESs) for molecules and materials. Neural networks (NNs) are one of the most popular such tools because of its simplicity and efficiency. The training algorithm for NNs becomes essential to achieve a fast and accurate fit with numerous data. The Levenberg-Marquardt (LM) algorithm has been recognized as one of the fastest and robust algorithms to train medium sized NNs and widely applied in recent NN based high quality PESs. However, when the number of ab initio data becomes large, the efficiency of LM is limited, making the training time consuming. Extreme learning machine (ELM) is a recently proposed algorithm which determines the weights and biases of a single hidden layer NN by a linear solution and is thus extremely fast. It, however, does not produce sufficiently small fitting error because of its random nature. Taking advantages of both algorithms, we report a generalized hybrid algorithm in training multilayer NNs. Tests on H+H2 and CH4+Ni(111) systems demonstrate the much higher efficiency of this hybrid algorithm (ELM-LM) over the original LM. We expect that ELM-LM will find its widespread applications in building up high-dimensional NN based PESs.

Selfie video based continuous Indian sign language recognition system

Abstract This paper introduces a novel method to bring sign language closer to real time application on mobile platforms. Selfie captured sign language video is processed by constraining its computing power to that of a smart phone. Pre-filtering, segmentation and feature extraction on video frames creates a sign language feature space. Minimum Distance and Artificial Neural Network classifiers on the sign feature space is trained and tested iteratively. Sobel edge operator's power is enhanced with morphology and adaptive thresholding giving a near perfect segmentation of hand and head portions compensating for the small vibrations of the selfie stick. Word matching score (WMS) gives the performance of the proposed method with an average WMS of around 85.58% for MDC and 90% for ANN with a small variation of 0.3 s in classification times. Neural network classifiers with fast training algorithms will certainly make this novel selfie sign language recognizer application into app stores.

An efficient beam-training scheme for the optimally designed subarray structure in mmWave LoS MIMO systems

This paper studies the fundamental relations between key design parameters of millimeter wave (mmWave) multiple input multiple output (MIMO) communication systems and subarray structures deployed at both the transmitter and the receiver with each radio frequency (RF) chain connected to only a specific subset of the antennas. The concept of effective degrees of freedom (EDoF) is introduced to measure the maximum spatial multiplexing gain available for the MIMO system. An analytical expression for the EDoF with respect to the parameters of antenna configuration and transmission distance is obtained for the line of sight (LoS) scenario. In addition, the upper and lower bounds of the EDoF are further obtained for some special cases. A fast beam training algorithm based on the codebook is developed to reduce the number of training for the designed mmWave system. Extensive simulation results indicate that the proposed scheme reduces the computational load of the exhaustive approach with only minimal loss in performance. Moreover, the proposed design is robust to the geometrical change and misplacement.