Nonlinear Approximation Capability Research Articles

Online adaptive critic designs with tensor product B-splines and incremental model techniques

As an effective optimal control scheme in the field of reinforcement learning, adaptive dynamic programming (ADP) has attracted extensive attention in recent decades. Neural networks (NNs) are commonly employed in ADP algorithms to realize nonlinear function approximation. However, the learning process with NN approximation typically requires a substantial amount of training data and places a heavy computational burden. To improve learning efficiency and alleviate computational load, this paper develops a novel model-free ADP approach based on multivariate splines and incremental control techniques, aimed at achieving online learning of nonlinear control. By utilizing input and output data, an incremental linear approximation model is identified online without any prior knowledge of system dynamics. To improve nonlinear approximation capabilities and lessen computational demands, tensor product B-splines, instead of NNs, are integrated into the critic module to approximate the value function, and the recursive least squares temporal difference (RLS-TD) algorithm is employed to update the weights of spline bases. Convergence analysis of the proposed control scheme is conducted based on the dual-timescale stochastic approximation theory. To illustrate the effectiveness and feasibility of the proposed control scheme, numerical simulations are performed in solving the online nonlinear control problem within the context of the inverted pendulum system.

Reservoir structure optimization of echo state networks: A detrended multiple cross-correlation pruning perspective

Reservoir structure optimization of echo state networks (ESN) is an important enabler for improving network performance. In this regard, pruning provides an effective means to optimize reservoir structure by removing redundant components in the network. Existing studies achieve reservoir pruning by removing insignificant neuronal connections. However, such processing causes the optimized neurons to still remain in the reservoir and thus hinder network inference by participating in computations, leading to suboptimal utilization of pruning benefits by the network. To solve this problem, this paper proposes an adaptive pruning algorithm for ESN within the detrended multiple cross-correlation (DMC2) framework, i.e., DMAP. On the whole, it contains two main functional parts: DMC2 measure of reservoir neurons and reservoir pruning. Specifically, the former is used to quantify the correlation among neurons. Based on this, the latter can remove neurons with high correlation from the reservoir completely, and finally obtain the optimal network structure by retraining the output weights. Experiment results show that DMAP-ESN outperforms its competitors in nonlinear approximation capability and reservoir stability.

Design and implementation of student management system of integrated programmable device programming system

With the continuous growth and development of society, the reform of higher education is gradually taking shape, and colleges and universities are taking more and more responsibilities in promoting education. The main element of university management is the student management system, which is very important to the development of the university. Under the objective environment of colleges and universities seeking to expand the scale of running schools and rapid economic and social development, the current student management system has been unable to meet the various needs of contemporary students. The integration of programmable device programming systems offers a student management system distinct advantages in terms of reliability, flexibility, and user-friendly operation. This study focuses on developing an effective and affordable student management system by incorporating a programmable device programming system. To evaluate the system's performance, this paper suggests the utilization of a BP neural network, renowned for its high nonlinear approximation capabilities and effectiveness in handling complex nonlinear functional relationships. The experimental findings highlight a significant contribution, demonstrating that the system achieved a throughput of 180 times per second, with a maximum CPU utilization of 99%. Notably, the system's stability exceeded 95%, contrasting sharply with the traditional student management system's stability at a mere 65%. These results underscore the substantial contribution of the proposed system, showcasing its enhanced stability compared to conventional student management systems.

Physics-constrained and flow-field-message-informed graph neural network for solving unsteady compressible flows

With the increasing use of deep neural networks as surrogate models for accelerating computational simulations in mechanics, the application of artificial intelligence in computational fluid dynamics has seen renewed interest in recent years. However, the application of deep neural networks for flow simulations has mainly concentrated on relatively simple cases of incompressible flows. The strongly discontinuous structures that appear in compressible flows dominated by convection, such as shock waves, introduce significant challenges when approximating the nonlinear solutions or governing equations. In this work, we propose a novel physics-constrained, flow-field-message-informed (FFMI) graph neural network for spatiotemporal flow simulations of compressible flows involving strong discontinuities. To enhance the nonlinear approximation capability of strong discontinuities, a shock detector method is leveraged to extract the local flow-field messages. These messages are embedded into the graph representation to resolve the discontinuous solutions accurately. A new FFMI sample-and-aggregate-based message-passing layer, which aggregates the edge-weighted attributes with node features on different hop layers, is then developed to diffuse and process the flow-field messages. Furthermore, an end-to-end paradigm is established within the encoder–decoder framework to transform the extracted information from the flow field into latent knowledge about the underlying fluid mechanics. Finally, a variety of one- and two-dimensional cases involving strong shock waves are considered to demonstrate the effectiveness and generalizability of the proposed FFMI graph neural network.

Joint Inversion of Rayleigh Wave Dispersion and Ellipticity Based on Deep Learning

The joint inversion of Rayleigh wave dispersion curves and ellipticity curves is an important method to obtain information about the velocity structure of the stratum, which has the advantages of high resolution and speed. However, there are problems in the application of this technique, such as complicated data, low inversion efficiency, and large influence factors of nonlinear inversion. Deep learning has excellent nonlinear approximation capability, which can compensate for most of the above defects, and has higher efficiency and accuracy compared with traditional methods. In this paper, in order to obtain the near-surface stratigraphic structure quickly and accurately, we propose to introduce the deep learning method into the joint inversion, use the CNN-LSTM network to perform the joint inversion of Rayleigh wave dispersion curves and ellipticity curves. And in order to expand the variety of data set, we propose to add noise to the sample data set, through which the empirical risk can be reduced, and the generalization performance of the network can be increased. Model experiments show that the deep learning network can effectively and accurately perform the joint inversion of Rayleigh waves, and adding noise to the training set is an effective way to improve the generalization and stability of the network.

Dense Skip Attention Based Deep Learning for Day-Ahead Electricity Price Forecasting

The forecasting of the day-ahead electricity price (DAEP) has become more of interest to decision makers in the liberalized market, as it can help optimize bidding strategies and maximize profits with the gradual market expansion. Deep learning (DL) is a promising method for its strong nonlinear approximation capabilities. However, it is challenging for traditional DL models to obtain a high forecasting precision for the DAEP, due to its internal temporal and feature-wise variabilities. To address the issue, this paper proposes a dense skip attention based DL model. In this model, to tackle the feature-wise variability, a mechanism of dense skip attention is proposed to efficiently assign learnable weights on the features for training. In terms of the temporal variability, a drop-connected structure based on the advanced residual unshared convolutional neural network (ARUCNN) and gate recurrent units (GRUs) is further proposed. In this structure, the ARUCNN is developed by embedding advanced activations to deal with the short-term dependencies and degradation while GRUs addressing the long-term ones, and they are integrated via a drop connection to reduce the overfitting. Through validating on real DAEP data in day-ahead markets of Sweden, Denmark, Norway and Finland, the results verify our proposed approach outperforms the existing methods in the deterministic and interval forecasting of DAEP.

Deterministic learning-based neural network control with adaptive phase compensation

Under the persistent excitation (PE) condition, the real dynamics of the nonlinear system can be obtained through the deterministic learning-based radial basis function neural network (RBFNN) control. However, in this scheme, the learning speed and accuracy are limited by the tradeoff between the PE levels and the approximation capabilities of the neural network (NN). Inspired by the frequency domain phase compensation of linear time-invariant (LTI) systems, this paper presents an adaptive phase compensator employing the pure time delay to improve the performance of the deterministic learning-based adaptive feedforward control with the reference input known a priori. When the adaptive phase compensation is applied to the hidden layer of the RBFNN, the nonlinear approximation capability of the RBFNN is effectively improved such that both the learning performance (learning speed and accuracy) and the control performance of the deterministic learning-based control scheme are improved. Theoretical analysis is conducted to prove the stability of the proposed learning control scheme for a class of systems which are affine in the control. Simulation studies demonstrate the effectiveness of the proposed phase compensation method.

A bearing prognosis framework based on deep wavelet extreme learning machine and particle filtering

Bearing prognosis plays an active role in preventing excessive or inadequate maintenance for major equipment. This paper develops a hybrid prognosis framework for bearings based on time-varying 3σ criterion, deep wavelet extreme learning machine (DWELM) and particle filtering (PF). To be specific, a time-varying 3σ criterion is proposed for bearing health monitoring to detect the fault occurrence time (FOT). Then, DWELM is established to evaluate the bearing performance degradation in degradation stage and construct a linear trend health indicator (HI) in a supervised way, termed as DWELM-HI. Compared to the original ELM, DWELM is equipped with more powerful feature representation and nonlinear approximation capabilities to map various nonlinear degradation trend to linear trend by deep structure and wavelet kernel. Additionally, a linear model is adopted to forecast the time evolution of the DWELM-HI. PF is collaboratively utilized to reduce random errors and estimate the probability of residual useful life (RUL). Finally, bearing prognosis is fully conducted on publicly available XJTU-SY bearing dataset to illustrate the effectiveness of the proposed method. The results show that the proposed method can detect an appropriate FOT and accurately estimate the RUL. Moreover, comparisons with other competing methods show that it performs better in bearing prognosis application.

Neural network disturbance observer with extended weight matrix for spacecraft disturbance attenuation

A modified neural network-based disturbance observer with high-accuracy estimation performance is fabricated for spacecraft attitude control in this paper. To reduce the approximation error of the neural network with a limited number of neurons for complex disturbances, the weight matrix of the neural network is extended into a time series expansion and updated by a modified back propagation algorithm. Thus, the high-order dynamics of the weight matrix can be reconstructed, and the disturbance estimation accuracy is improved. Then, by introducing an integral state of the lumped disturbance into the disturbance model and utilizing it as the virtual disturbance measurement, an improved neural network-based disturbance observer is designed. The proposed observer takes advantages of the nonlinear approximation capability of the neural network and the high-order dynamics estimation ability of the extended observer structure. To realize high-performance attitude control of a spacecraft, a composite controller based on the proposed observer is designed, and then tested by comparative simulations. Simulation results show that, the estimation error of the designed observer can be reduced by an order of magnitude than the traditional observers. It is suitable for high-performance disturbance attenuation of verities of applications.

An Enhanced Full-Form Model-Free Adaptive Controller for SISO Discrete-Time Nonlinear Systems.

This study focuses on the full-form model-free adaptive controller (FFMFAC) for SISO discrete-time nonlinear systems, and proposes enhanced FFMFAC. The proposed technique design incorporates long short-term memory neural networks (LSTMs) and fuzzy neural networks (FNNs). To be more precise, LSTMs are utilized to adjust vital parameters of the FFMFAC online. Additionally, due to the high nonlinear approximation capabilities of FNNs, pseudo gradient (PG) values of the controller are estimated online. EFFMFAC is characterized by utilizing the measured I/O data for the online training of all introduced neural networks and does not involve offline training and specific models of the controlled system. Finally, the rationality and superiority are verified by two simulations and a supporting ablation analysis. Five individual performance indices are given, and the experimental findings show that EFFMFAC outperforms all other methods. Especially compared with the FFMFAC, EFFMFAC reduces the by 21.69% and 11.21%, respectively, proving it to be applicable for SISO discrete-time nonlinear systems.

An Adaptive Sigmoidal Activation Function for Training Feed Forward Neural Network Equalizer

Feed for word neural networks (FFNN) have attracted a great attention, in digital communication area. Especially they are investigated as nonlinear equalizers at the receiver, to mitigate channel distortions and additive noise. The major drawback of the FFNN is their extensive training. We present a new approach to enhance their training efficiency by adapting the activation function. Adapting procedure for activation function extensively increases the flexibility and the nonlinear approximation capability of FFNN. Consequently, the learning process presents better performances, offers more flexibility and enhances nonlinear capability of NN structure thus the final state kept away from undesired saturation regions. The effectiveness of the proposed method is demonstrated through different challenging channel models, it performs quite well for nonlinear channels which are severe and hard to equalize. The performance is measured throughout, convergence properties, minimum bit error achieved. The proposed algorithm was found to converge rapidly, and accomplish the minimum steady state value. All simulation shows that the proposed method improves significantly the training efficiency of FFNN based equalizer compared to the standard training one.

Development of two-phase logic-oriented fuzzy AND/OR network

The architecture of AND/OR fuzzy neural networks exhibits outstanding learning abilities and significant interpretation capabilities. However, AND/OR networks suffer from structure-related problems namely low efficiency and slow convergence of learning due to several reasons such as high dimensionality and gradient-based learning algorithms which lead to a visible computing overhead. In this paper, we present a two-phase fuzzy logic-oriented network design that is composed of AND/OR neurons. This design takes advantages of Randomized Neural Network (RNN) to achieve higher convergence while exhibiting good nonlinear approximation capabilities. A gradient–based learning algorithm is implemented in the second phase of the design to further reduce values of performance index. The quality of the proposed design and resulting architecture is quantified through the use of numeric data along with fuzzy sets (information granules). Experimental results meet the research’s objectives and the proposed design methodology opens up new future directions for proceeding with more improvements.

Neuro‐controller for broadcast lighting light‐emitting diode to express white light with high color rendering index

AbstractTechnical characteristics analysis related to correlated color temperature (CCT), color rendering, and illuminance is required to use light‐emitting diode (LED) as broadcast lighting. In general, to realize a white light source with a high color rendering index (CRI), we selected the appropriate emission intensity of RGBW LED through trial and error. However, the characteristics of the LED light source and environmental conditions make it difficult to perform the procedure several times. The objective of this study was to design a system that could control illuminance, CCT, and ∆uv while having high CRI, as an LED control method for broadcasting lighting. The controller implements using a feed‐forward neural network with excellent nonlinear function approximation capability. We measure data directly from the red green blue white (RGBW) LED system for neural network training. We then select data with high CRI from the measured raw data and choose data for neural network learning by removing measurement noise using the quadratic polynomial interpolation method. The performance evaluation confirms that the proposed neural network controller shows excellent results as an RGBW LED controller for broadcast lighting in the Planckian locus and all regions of white light.

Small sample state of health estimation based on weighted Gaussian process regression

-Battery state of health (SOH) estimation is essential for the safety and reliability of electric vehicles. Data-driven approaches are compelling in SOH estimation as they work effectively without human intervention and have excellent nonlinear approximation capabilities. Most studies assume that the training data is sufficient. However, in practical applications, data acquisition is often expensive and time-consuming. A novel weighted Gaussian process regression SOH estimation method is proposed to reduce the model's dependence on data through knowledge transfer. The squared exponential covariance function is introduced with a penalty mechanism to control the cross-battery knowledge transfer process. Experiments are carried out with battery cyclic aging data under different working conditions. Experimental results show that the proposed weighted Gaussian process SOH estimation model can obtain reliable prediction results, although the training data only accounts for 20% of the total dataset.11Corresponding author: Hanmin Sheng email: shenghanmin@hotmail.comThis paper is sponsored by National Key R&D Program of China (No. 2018YFC1505203), National Natural Science Foundation of China (No. 61903066), China Postdoctoral Science Foundation (No. 2021M690560, No. 2018M640905), and Sichuan Province Science and Technology Program (No. 2021YFH0042)

Finite‐time adaptive neural control for nonlinear systems under state‐dependent sensor attacks

AbstractThis article proposes a finite‐time control scheme for a class of uncertain nonlinear systems in the presence of sensor attacks. Specifically, based on the backstepping technology and the nonlinear function approximation capability of radial basis function neural networks, we develop an adaptive controller, guaranteeing the finite‐time stability of the closed‐loop system despite sensor attacks. In particular, the unknown time‐varying state‐feedback gains caused by sensor attacks are handled by Nussbaum functions. To decrease the design difficulty of the finite‐time controller, a novel practical finite‐time stability criterion is given. Finally, two simulation examples are provided, demonstrating the effectiveness of the proposed adaptive control scheme.

Multi-layer gated temporal convolution network for residual useful life prediction of rotating machinery

The classical recurrent neural networks (RNNs) face the defect of long-term dependence in the prediction of time series and thus have a poor generalization ability. In the meantime, their loss functions are generally obtained by means of traversing the whole training data set for supervised learning, which increases the time complexity and space complexity. As a result, classical RNNs usually show poor prediction accuracy and undesirable computation efficiency in the prediction of residual useful life (RUL) of rotating machinery (RM). In view of this, a multi-layer gated temporal convolution network (MLGTCN) is proposed to predict RUL of RM in this paper. In our proposed MLGTCN, a multi-layer temporal convolution network structure is innovatively constructed to perform convolution operations on input data for expanding the receptive field, and the long-term historical information can be traced, which solves the problem of long-term dependence and enhances the generalization ability of MLGTCN. Moreover, the gated linear units (GLUs) are creatively constructed to filter out the important information hierarchically, which endows MLGTCN with a high nonlinear approximation capability. Additionally, in order to improve the global optimization ability and convergence speed, a reinforcement learning algorithm based on semi-gradient temporal difference (semi-gradient TD) is adopted and a novel action controller is designed for updating the convolution kernels and bias values of MLGTCN, which can use the increment information between the successive predicted values, thus rapidly approaching the optimal strategy. Owing to the above MLGTCN’s advantages, high prediction accuracy and desirable computation efficiency can be achieved in the RUL prediction of RM. The effectiveness of our proposed method is experimentally validated with the RUL prediction of double-row roller bearings.

Design of UWB Antenna Based on Improved Deep Belief Network and Extreme Learning Machine Surrogate Models

In the design of conventional microwave devices, the parameters need to be continuously optimized to meet the desired targets, and the whole process is time-consuming and laborious. As a surrogate model, machine learning is an effective optimization method. However, in the modeling process, the high-dimensional data processing and the complex nonlinear relationship between parameters is a problem to be solved. This paper proposes a deep learning model for designing UWB antennas, which determines the model structure of deep belief network (DBN) by particle swarm algorithm (PSO), and then combines DBN and extreme learning machine (ELM). The proposed model can obtain higher feature learning capability and nonlinear function approximation capability, and has been applied to the optimal design of the whole structure of the fractal antenna and the notch structure of the MIMO antenna, and its S-parameters are well fitted while meeting the requirements of the design targets. The DBN-ELM method obtains the good results when compared with common modeling methods using the same training samples (the root mean square error tested is 11.87% in the fractal antenna and 3.56% in the MIMO antenna). Overall, the proposed DBN-ELM model has higher predictive and generalization capabilities, which can also be used to model more complex antenna structures.

Deep Learning Based Densely Connected Network for Load Forecasting

Load forecasting is of crucial importance for operations of electric power systems. In recent years, deep learning based methods are emerging for load forecasting because their strong nonlinear approximation capabilities can provide more forecasting precision than conventional statistical methods. However, they usually suffer from some problems, e.g., the gradient vanishment and over-fitting. In order to address these problems, an unshared convolution based deep learning model with densely connected network is proposed. In this model, the backbone is the unshared convolutional neural network and a densely connected structure is adopted, which could alleviate the gradient vanishment. What is more, we use a regularization method named clipped <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_2$</tex-math></inline-formula> -norm to overcome over-fitting, and design a trend decomposition strategy to address the possible distribution differences between the training and validation data. Finally, we conduct five case studies to verify the outperformance of our proposed deep learning model for deterministic and interval load forecasting. Two high-voltage and an medium-voltage real load datasets from Australia, Germany and America are used for model training and validation, respectively. Results show that the proposed model can achieve higher load forecasting accuracy, compared with other existing methods including the popular conventional methods such as naive forecast and generalized additive model, and deep learning methods, e.g., long short-term memory network, convolutional neural network, fully connected network, etc.

Aerodynamic Modeling of ATTAS Aircraft using Mamdani Fuzzy Inference Network

This article presents aerodynamic modeling of the fixed-wing aircraft using the Mamdani fuzzy inference network (MFIN). A Mamdani fuzzy inference system with a Gaussian membership function has been used as a nonlinear regression functional node to create a multilayer network, called MFIN. The multilayered MFIN incorporates the nonlinear function approximation capability of the multilayered neural network in addition to robustness against uncertainties and measurement noises. The limited-memory Broyden–Fletcher–Goldfarb–Shanno optimization technique has been used to optimize network parameters to learn the nonlinear yawing moment dynamics of the Advanced Technology Testing Aircraft System (ATTAS) aircraft. Since every node in the network learns the nonlinearity of the dynamics, the proposed MFIN becomes capable of learning highly nonlinear dynamics. The adequacy of the proposed network is validated using the recorded flight data from the ATTAS aircraft of the DLR German Aerospace Centre in two cases: 1) trimmed low-angle-of-attack flight condition and 2) quasi-steady stall high-angle-of-attack (highly nonlinear complex) flight condition. The simulated time history tracking performance, mean square error, $R^2$ score, and explained variance score of the proposed network are compared with state-of-the-art methods. Also, the robustness of the proposed approach is demonstrated by evaluating its performance against test data corrupted with additive white Gaussian noise.

A White-Box Equivalent Neural Network Circuit Model for SoC Estimation of Electrochemical Cells.

Smart grids, microgrids, and pure electric powertrains are the key technologies for achieving the expected goals concerning the restraint of CO2 emissions and global warming. In this context, an effective use of electrochemical energy storage systems (ESSs) is mandatory. In particular, accurate state of charge (SoC) estimations are helpful for improving the ESS performances. To this aim, developing accurate models of electrochemical cells is necessary for implementing effective SoC estimators. Therefore, a novel neural network modeling technique is proposed in this paper. The main contribution consists in the development of a white-box neural design that provides helpful insights into the cell physics, together with a powerful nonlinear approximation capability, and a flexible system identification procedure. In order to do that, the system equations of a white-box equivalent circuit model (ECM) have been combined with computational intelligence techniques by approximating each circuit element with a dedicated neural network. The model performances have been analyzed in terms of model accuracy, SoC estimation effectiveness, and computational cost over two realistic data sets. Moreover, the proposed model has been compared with a white-box ECM and a gray-box neural network model. The results prove that the proposed modeling technique is able to provide useful improvements in the SoC estimation task with a competing computational cost.