Input Weight Matrix Research Articles

Hybrid Extreme Learning for Reliable Short-Term Traffic Flow Forecasting

Reliable forecasting of short-term traffic flow is an essential component of modern intelligent transport systems. However, existing methods fail to deal with the non-linear nature of short-term traffic flow, often making the forecasting unreliable. Herein, we propose a reliable short-term traffic flow forecasting method, termed hybrid extreme learning, that effectively learns the non-linear representation of traffic flow, boosting forecasting reliability. This new algorithm probes the non-linear nature of short-term traffic data by exploiting the artificial bee colony that selects the best-implied layer deviation and input weight matrix to enhance the multi-structural information perception capability. It speeds up the forecasting time by calculating the output weight matrix, which guarantees the real usage of the forecasting method, boosting the time reliability. We extensively evaluate the proposed hybrid extreme learning method on well-known short-term traffic flow forecasting datasets. The experimental results show that our method outperforms existing methods by a large margin in both forecasting accuracy and time, effectively demonstrating the reliability improvement of the proposed method. This reliable method may open the avenue of deep learning techniques in short-term traffic flow forecasting in real scenarios.

Research on Pattern Classification Based on Double Pseudo-Inverse Extreme Learning Machine

This research aims to address the limitations inherent in the traditional Extreme Learning Machine (ELM) algorithm, particularly the stochastic determination of input-layer weights and hidden-layer biases, which frequently leads to an excessive number of hidden-layer neurons and inconsistent performance. To augment the neural network’s efficacy in pattern classification, Principal Component Analysis (PCA) is employed to reduce the dimensionality of the input matrix and alleviate multicollinearity issues during the computation of the input weight matrix. This paper introduces an enhanced ELM methodology, designated the PCA-DP-ELM algorithm, which integrates PCA with Double Pseudo-Inverse Weight Determination (DP). The PCA-DP-ELM algorithm proposed in this study consistently achieves superior average classification accuracy across various datasets, irrespective of whether assessed through longitudinal or cross-sectional experiments. The results from both experimental paradigms indicate that the optimized algorithm not only enhances accuracy but also improves stability. These findings substantiate that the proposed methodology exerts a positive influence on pattern classification.

Using reservoir computing to solve FPK equations for stochastic dynamical systems under Gaussian or Non-Gaussian excitation

This paper presents a new approach that uses the Reservoir Computing Algorithm to solve Fokker-Planck-Kolmogorov (FPK) equation excited by both Gaussian white noise and non-Gaussian noise. Unlike typical numerical methods, this methodology does not necessitate spatial reconstruction or numerical supplementation. The novelty of this paper lies in the modifications made to the conventional Reservoir Computing algorithm. We altered the approach for calculating values of the input weight matrix and incorporated autoregressive techniques in the reservoir layer. In addition, we applied data normalization to the training data before training the algorithm to avoid a zero solution. The efficacy of this approach was verified through multiple arithmetic examples, showcasing its practicality and efficiency in solving FPK equations. Moreover, the Reservoir Computing-FPK algorithm is capable of solving high-dimensional and fractional-order FPK equations with a smaller training set than earlier algorithms. Finally, we analyzed how values of the input weight matrix and regularization parameter affected the performance of the algorithm. The findings suggest that the careful selection of hyperparameters can greatly improve the performance of the Reservoir Computing algorithm.

Time Series Prediction of ESN Based on Chebyshev Mapping and Strongly Connected Topology

This paper introduces a novel approach called Chebyshev mapping and strongly connected topology for optimization of echo state network (ESN). To enhance the predictive performance of ESNs for time series data, Chebyshev mapping is employed to optimize the irregular input weight matrix. And the reservoir of the ESN is also replaced using an adjacency matrix derived from a digital chaotic system, resulting in a reservoir with strong connectivity properties. Numerical experiments are conducted on various time series datasets, including the Mackey–Glass time series, Lorenz time series and solar sunspot numbers, validating the effectiveness of the proposed optimization methods. Compared with the traditional ESNs, the optimization method proposed in this paper has higher predictive performance, and effectively reduce the reservoir’s size and model complexity.

Practical deadbeat MPC design via controller matching with applications in power electronics

Several applications, including power electronics and electrical machines, require a fast system response, which is typically achieved via deadbeat control. Hence, there has been an interest in developing model predictive control (MPC) algorithms with deadbeat control properties, i.e., finite-time convergence to a set-point, for power electronics applications. In this paper, we design a practical deadbeat MPC via controller matching. We make use of an existing result for tuning the weight matrices of the MPC cost function such that the corresponding unconstrained MPC solution matches a desired deadbeat controller. This approach allows for a positive definite input weight matrix and provides stability and recursive feasibility guarantees for the resulting MPC controller. We additionally propose a vertex relaxation of the matching problem, which reduces conservatism, and a method for enlarging the terminal set of the deadbeat MPC controller. Three benchmark examples from the power electronics field are used to show the effectiveness of the proposed MPC design.

Semi-supervised echo state network with partial correlation pruning for time-series variables prediction in industrial processes

For an ordinary echo state network (ESN), redundant information in the huge reservoir will lead to degradation of the prediction performance of the network, especially when the labels of the samples are limited. To solve this problem, a semi-supervised ESN with partial correlation pruning (PCP-S2ESN) is proposed in this paper to scientifically capture the essential association between two reservoir variables while controlling for the influence of other factors. In this way, redundant neurons and their connection weights in the reservoir are eliminated, so that the prediction accuracy is significantly enhanced by optimizing the network structure. Moreover, an unsupervised pre-training procedure is introduced to modify the input weight matrix and reservoir connection weight matrix of the ESN, which successfully achieves precise prediction of time-series variables with limited labels. The superiority of the PCP-S2ESN model is demonstrated through two benchmark prediction tasks and the fed-batch penicillin cultivation process.

Minimum inertia demand estimation of new power system considering diverse inertial resources based on deep neural network

AbstractWith the high‐proportion integration of renewable energy and power electronic equipment, the inertia supporting ability of new power system continues to decline, which seriously threatens the frequency stability of power grids. In order to clarify the operation boundary, and realise the rapid analysis and prediction of the minimum inertia demand of new power systems, this study proposes a minimum inertia demand estimation method based on deep neural network (DNN). Firstly, this study establishes the system frequency response model of new power systems containing diverse inertia resources including renewable energy, induction machine and so on. Considering the constraints of rate of change of frequency and maximum frequency deviation, the minimum inertia demand estimation model is established to ensure the system frequency stability. DNN is introduced to effectively map non‐linear relations in complex situations, which can quickly estimate and predict the minimum inertia of new power systems. Adam algorithm is utilised to optimise the input weight matrix and hidden layer feature vector of the network to improve accuracy. Finally, the simulations and analysis are conducted in IEEE‐39 system to verify the accuracy and generalisation ability of the proposed method in this paper.

Power-Efficient Deep Neural Network Accelerator Minimizing Global Buffer Access without Data Transfer between Neighboring Multiplier—Accumulator Units

This paper presents a novel method for minimizing the power consumption of weight data movements required by a convolutional operation performed on a two-dimensional multiplier–accumulator (MAC) array of a deep neural-network accelerator. The proposed technique employs a local register file (LRF) at each MAC unit in a manner such that once weight pixels are read from the global buffer into the LRF, they are reused from the LRF as many times as desired instead of being repeatedly fetched from the global buffer in each convolutional operation. One of the most evident merits of the proposed method is that the procedure is completely free from the burden of data transfer between neighboring MAC units. It was found from our simulations that the proposed method provides a power saving of approximately 83.33% and 97.62% compared with the power savings recorded by the conventional methods, respectively, when the dimensions of the input data matrix and weight matrix are 128 × 128 and 5 × 5, respectively. The power savings increase as the dimensions of the input data matrix or weight matrix increase.

Echo state network with logistic mapping and bias dropout for time series prediction

An echo state network (ESN) is a special structure of a recurrent neural network in which the recurrent neurons are randomly connected. ESN models that have achieved high accuracy on time series prediction tasks can be utilized as time series prediction models in many fields. Nevertheless, in most ESN models, the input weights are irregularly generated and the reservoir layer units are generally redundant, which cannot guarantee that the ESN models will always be optimal for a given task. In this paper, a novel ESN model that combines logistic mapping (LM) and bias dropout (BD) algorithms is proposed to optimize the irregular input weight matrix and generate a superior and simpler reservoir. Initially, the initial input weight matrix of ESN is replaced by an LM input weight matrix that is generated by the recurrent LM algorithm. Meanwhile, the reservoir, which can convert the input space into a high-dimensional feature space, is formed by the input signals and the LM input weight matrix. Then, the units with low activation values that are determined by a reservoir dropout probability are discarded through the BD algorithm. The dropout probability is determined by the contributions of the reservoir units to the training performance. Three multivariable benchmark tasks and four univariate real-world time series tasks indicate that the proposed LM-BD-ESN model is effective in reducing the testing time, reservoir size, and model complexity while improving the performance of the traditional ESN.

Chaotic time series prediction using echo state network based on selective opposition grey wolf optimizer

Chaos prediction of nonlinear system is of great significance for proposing control strategies early. On the other hand, echo state network (ESN) as an artificial neural recursive network is widely used in time series prediction, while it has significant disadvantages due to the random input weight matrix. This work proposes a chaos prediction model based on ESN optimized by the selective opposition grey wolf optimizer (SOGWO). Firstly, the input weight matrix of ESN is restructured to express the position of gray wolf optimizer (GWO). Secondly, the selective opposition strategy is introduced into the GWO to enhance the exploration and exploitation capability. Thirdly, the optimal network is achieved by iteratively updating the position of SOGWO where the input weight is replaced by optimal search agent. Finally, taking two typical chaotic time series of Mackey–Glass and Lorenz as the objects, the effectiveness of the proposed SOGWO-ESN is verified by numerical simulation. The experimental results of SOGWO-ESN are compared with those of the traditional echo state network, the network model optimized by the particle swarm optimization (PSO) and the traditional gray wolf optimizer. The results imply that the proposed SOGWO-ESN has better prediction performance and is able to accurately predict a much longer period of time.

Line loss prediction based on particle swarm optimization combined with extreme learning machine

As a key economic index to measure the planning and management level of power grid company, line loss is not only the main content of completing the construction and transformation task of power grid company, but also an important assessment standard for the operation level of power grid company. In order to improve the generalization ability and prediction accuracy of extreme learning machine (ELM), particle swarm optimization (PSO) algorithm is applied to extreme learning machine (ELM), and a line loss prediction method based on PSO combined with ELM is proposed. The particle swarm optimization algorithm is used to select the optimal hidden layer deviation and input weight matrix to calculate the output weight matrix, so as to improve the accuracy and stability of the extreme learning machine. Experimental results show that the proposed method can effectively improve the accuracy of line loss prediction.

A Combination Interval Prediction Model Based on Biased Convex Cost Function and Auto-Encoder in Solar Power Prediction

Due to the intermittent and stochastic nature of solar power, solar power interval prediction is of great importance for grid management and power dispatching. A combination interval prediction model based on the lower and upper bound estimation (LUBE) is proposed to efficiently quantify the solar power prediction uncertainty. In the proposed model, the upper and lower bounds are separately predicted by two prediction engines. The extreme learning machine (ELM) is selected as the basic prediction engine. The auto-encoder technique is used to initialize the input weight matrix of ELM for efficient feature learning. A novel biased convex cost function is developed for ELM to predict the interval boundary. The output weight matrix of ELM can be solved via the convex optimization technique instead of the conventional heuristic algorithm. The proposed interval prediction model can be formulated as a bi-level optimization problem. In the lower-level problem, the lower and upper ELMs are trained under different candidate hyper-parameters of the biased cost function. In the upper-level problem, the optimal combination of the lower and upper prediction engines is determined by evaluating the interval prediction performance. Comprehensive experiments based on public data set are conducted to validate the superiority of the proposed interval prediction model.

Compute-in-Memory Chips for Deep Learning: Recent Trends and Prospects

Compute-in-memory (CIM) is a new computing paradigm that addresses the memory-wall problem in hardware accelerator design for deep learning. The input vector and weight matrix multiplication, i.e., the multiply-and-accumulate (MAC) operation, could be performed in the analog domain within memory sub-array, leading to significant improvements in throughput and energy efficiency. Static random access memory (SRAM) and emerging non-volatile memories such as resistive random access memory (RRAM) are promising candidates to store the weights of deep neural network (DNN) models. In this review, firstly we survey the recent progresses in SRAM and RRAM based CIM macros that have been demonstrated in silicon. Then we discuss general design challenges of the CIM chips including analog-to-digital conversion (ADC) bottleneck, variations in analog compute, and device non-idealities. Next we introduce the DNN+NeuroSim benchmark framework that is capable of evaluating versatile device technologies for CIM inference and training performance from software/hardware co-design's perspective.

An Arbitrarily Reconfigurable Extreme Learning Machine Inference Engine for Robust ECG Anomaly Detection

Extreme learning machine (ELM) has shown to be an effective and low-power approach for real-time electrocardiography (ECG) anomaly detection. However, prior ELM inference chips are noise-prone and lacking in reconfigurability. In this article, we present an arbitrarily reconfigurable ELM inference engine fabricated in 40-nm CMOS technology for robust ECG anomaly detection. By combining Adaptive boosting (Adaboost) and Eigenspace denoising with ELM (AE-ELM), robust classification under noisy conditions is achieved and saves the number of required multiplications by 95.9%. For chip implementation, a reconfigurable VLSI architecture is designed to support arbitrary complexity of AE-ELM, accounting for dynamic change in application requirements. On the other hand, we propose to construct the input weight matrix of ELM as a Bernoulli random matrix, which further reduces the number of multiplications by 55.2%. For real-time detection, parallel computing is exploited to reduce the latency by up to 86.8%. Overall, the 0.21-mm2 AE-ELM inference engine shows its robustness against noisy signals and achieves $1.83\times$ AEE compared with the state-of-the-art ELM design.

Neural Network for Low-Memory IoT Devices and MNIST Image Recognition Using Kernels Based on Logistic Map

This study presents a neural network which uses filters based on logistic mapping (LogNNet). LogNNet has a feedforward network structure, but possesses the properties of reservoir neural networks. The input weight matrix, set by a recurrent logistic mapping, forms the kernels that transform the input space to the higher-dimensional feature space. The most effective recognition of a handwritten digit from MNIST-10 occurs under chaotic behavior of the logistic map. The correlation of classification accuracy with the value of the Lyapunov exponent was obtained. An advantage of LogNNet implementation on IoT devices is the significant savings in memory used. At the same time, LogNNet has a simple algorithm and performance indicators comparable to those of the best resource-efficient algorithms available at the moment. The presented network architecture uses an array of weights with a total memory size from 1 to 29 kB and achieves a classification accuracy of 80.3–96.3%. Memory is saved due to the processor, which sequentially calculates the required weight coefficients during the network operation using the analytical equation of the logistic mapping. The proposed neural network can be used in implementations of artificial intelligence based on constrained devices with limited memory, which are integral blocks for creating ambient intelligence in modern IoT environments. From a research perspective, LogNNet can contribute to the understanding of the fundamental issues of the influence of chaos on the behavior of reservoir-type neural networks.

Data Fusion Algorithm for Heterogeneous Wireless Sensor Networks Based on Extreme Learning Machine Optimized by Particle Swarm Optimization

Data fusion can reduce the data communication time between sensor nodes, reduce energy consumption, and prolong the lifetime of the network, making it an important research focus in the field of heterogeneous wireless sensor networks (HWSNs). Normal sensor nodes are susceptible to external environmental interferences, which affect the measurement results. In addition, raw data contain redundant information. The transmission of redundant information consumes excess energy, thereby reducing the lifetime of the network. We propose a data fusion method based on an extreme learning machine optimized by particle swarm optimization for HWSNs. The spatiotemporal correlation between the data of the HWSNs is determined, and the extreme learning machine method is used to process the data collected by the sensor nodes in the hierarchical routing structure of the HWSN. The particle swarm optimization algorithm is used to optimize the input weight matrix and the hidden layer bias of the extreme learning machine. An output weight matrix is created to reduce the number of hidden layer nodes and improve the generalization ability of the model. The data fusion model fuses the original data collected by the sensor nodes. The simulation results show that the proposed algorithm reduces network energy consumption and improves the lifetime of the network, the efficiency of data fusion, and the reliability of data transmission compared with other data fusion methods.

A weighted voting ensemble of efficient regularized extreme learning machine

The exact evaluation of Extreme Learning Machine (ELM) compactness is difficult due to the randomness in hidden layer nodes number, weight and bias values. To overcome this randomness, and other problems such as resultant overfitting and large variance, a selective weighted voting ensemble model based on regularized ELM is investigated. It can strongly enhance the overall performance including accuracy, variance and time consumption. Efficient Prediction Sum of Squares (PRESS) criteria that utilizing Singular Value Decomposition (SVD) is proposed to address the slow execution. Furthermore, an ensemble pruning approach based on the eigenvalues for the input weight matrix is developed. In this work, the ensemble base classifiers weights are calculated based on the same PRESS error metric used for the solutions of the output weight vector (β) in RELM, thus, it can reduce computational cost and space requirement. Different state-of-the-art learning approaches and various well-known facial expressions faces and object recognition benchmark datasets were examined in this work.

Predicting rockburst with database using particle swarm optimization and extreme learning machine

Rockburst is a major type of geological hazard that has a very adverse impact on underground engineering in deeply buried areas under high geo-stress. In this study, extreme learning machine (ELM) was used to predict and classify rockburst intensity, and particle swarm optimization (PSO) was used to optimize the input weight matrix and the hidden layer bias in ELM. Six quantitative rockburst parameters were used as input for the PSO-ELM network, including the maximum tangential stress of the surrounding rock σθ, the uniaxial compressive strength of rock σc, the tensile strength of rock σt, the stress ratio σθ/σc, the rock brittleness ratio σc/σt and the elastic energy index Wet. The network was used to learn from a database of 344 collected worldwide rockburst cases, on which the PSO-ELM rockburst prediction model was established and verified using 8-fold cross-validation and independent test set validation. The model was then tested on a new set of fifteen typical rockburst cases from Jiangbian hydropower station in China. The results showed that the PSO-ELM model performed well in rockburst level prediction. In addition, the model showed superior performance compared with previously proposed machine learning models for rockburst prediction and the rockburst empirical criteria, which underscores its utility in future rockburst prediction.

Ultra-Short-Term Wind Power Prediction by Salp Swarm Algorithm-Based Optimizing Extreme Learning Machine

Wind power generation accounts for an increasing proportion of the power grid, so efficient and accurate real-time wind power prediction is particularly important for wind power grid. In view of the strong randomness and fluctuation of wind and the difficulty of predicting wind power, a Salp Swarm Algorithms-Extremely Learning Machine (SSA-ELM) based ultra-short-term wind power prediction model is proposed. In this case, the multi-input sample set is composed of historical wind speed, temperature, wind direction, atmospheric pressure and other climatic factors that are highly correlated with wind power, and the network parameters are determined in the training process. In order to improve the adaptability and accuracy of the prediction model, the input weight matrix and hidden layer deviation of the Extreme Learning Machine (ELM) are optimized by exploring and developing the Salp Swarm Algorithm in the iterative process. Finally, the simulation experiment is conducted with the actual data of a wind farm in Henan Province, and the comparison with the traditional Extreme Learning Machine, Particle Swarm Optimization Extreme Learning Machine (PSO-ELM) and Back Propagation (BP) neural network model shows that the new method avoids falling into the local extreme value, and has faster convergence speed and higher prediction accuracy.

Solar Power Interval Prediction via Lower and Upper Bound Estimation with a New Model Initialization Approach

This paper proposes a new model initialization approach for solar power prediction interval based on the lower and upper bound estimation (LUBE) structure. The linear regression interval estimation (LRIE) was first used to initialize the prediction interval and the extreme learning machine auto encoder (ELM-AE) is then employed to initialize the input weight matrix of the LUBE. Based on the initialized prediction interval and input weight matrix, the output weight matrix of the LUBE could be obtained, which was close to optimal values. The heuristic algorithm was employed to train the LUBE prediction model due to the invalidation of the traditional training approach. The proposed model initialization approach was compared with the point prediction initialization and random initialization approaches. To validate its performance, four heuristic algorithms, including particle swarm optimization (PSO), simulated annealing (SA), harmony search (HS), and differential evolution (DE), were introduced. Based on the experiment results, the proposed model initialization approach with different heuristic algorithms was better than the point prediction initialization and random initialization approaches. The PSO can obtain the best efficiency and effectiveness of the optimal solution searching in four heuristic algorithms. Besides, the ELM-AE can weaken the over-fitting phenomenon of the training model, which is brought in by the heuristic algorithm, and guarantee the model stable output.