High-order Neural Networks Research Articles

A pruning algorithm with relaxed conditions for high-order neural networks based on smoothing group [formula omitted] regularization and adaptive momentum

To enhance the sparseness of the network, improve its generalization ability and accelerate its training, we propose a novel pruning approach for sigma-pi-sigma neural network (SPSNN) under the relaxed condition by adding smoothing group L1/2 regularization and adaptive momentum. The main strength of this method is that it can prune both the redundant nodes between groups in the network, and also the redundant weights of the non-redundant nodes within the group, so as to achieve the sparseness of the network. Another strength is that the non-smooth absolute value function in the traditional L1/2 regularization method is replaced by a smooth function. This reduces the oscillations of learning and enables us to more effectively prove the convergence of the proposed algorithm. Finally, the numerical simulation results demonstrate the effectiveness of the proposed algorithm.

Semi-empirical dynamics modeling of a bicycle robot based on feature selection and RHONN

Aiming at the difference between a bicycle robot dynamics model based on prior knowledge and a real robot, the robot modeling problem is studied. A set of features is extracted from the dynamics model of a bicycle robot. The trigonometric functions in the features are replaced with a linear combination of several sigmoid functions to form a new set of features. An index describing the importance of features is designed based on the maximal information coefficient (MIC). The features are ranked using the feature importance index. The index of system identification fitness (FIT) is regarded as the likelihood function term in Bayesian information criterion (BIC). The penalty term of BIC is constructed with the number of both the features and weight parameters in the recurrent high order neural network (RHONN) model. A semi-empirical dynamics modeling method based on feature selection is proposed. An inverse optimal controller is designed to stabilize the bicycle robot. The effectiveness of the modeling method is verified through simulation and real robot experiments.

Compact Modeling of Nanocluster Functionality as a Higher-Order Neuron

Disordered nanoclusters with multielectrode input–output functionality had recently been experimentally realized with energy-efficient and emergent computational capacity, and thus an interconnected network of several such nanoclusters had been proposed to realize artificial neural networks. To aid that end, here we show that nanocluster functionality can be fit to the simplest dendritic neuron model (DNM), where the only form of nonlinearity is due to multiplicative interactions. This work brings into the spotlight higher-order neural networks (known for their efficient encoding of geometric invariances) to serve as an explainable baseline model of nano-networks against which experimentalists can compare more sophisticated models (deep neural networks or physics-based models such as the lin-min network introduced here) and provides ground for designing novel approximate hardware and a statistical mechanics analysis of the learning performance of interconnected nanoclusters versus perceptrons (where neurons output a nonlinear function of the weighted sum of their inputs). A network with just ten higher-order neurons is shown to achieve a classification accuracy of more than 96% on the MNIST benchmark for handwritten digit recognition (which required 100 times more neurons in three-layer perceptrons).

Prediction of Life Insurance Premium during Pre-and Post-Covid-19: A Higher-Order Neural Network Approach

The impact of Covid-19 drastically changed the economical, social, behavioral, and psychological aspects of human life. Just after the official announcement of this pandemic by the World Health Organization, the entire world has locked for several months. As people come across these unpredictable situations, insurance has emerged as one of the ways to financially protect themselves and their families. Consumers were confused in terms of financial gain and risk associated with this disease. In this critical situation, the study on the consumer decision-making process is an integral part to evaluate consumer behavior. An evolutionary improved FLANN is developed to analyze the growth of insurance during the pandemic and observed the insignificance of this pandemic effect on consumer behavior toward life insurance. The proposed study has undertaken an asymmetrical data analysis on 24 life insurance companies of India from January 2015 to December 2020 and predicted the premium collection for the future. The proposed method is compared with other similar machine learning methods and found to be superior.

RHONN Modelling-Enabled Nonlinear Predictive Control for Lateral Dynamics Stabilization of an In-Wheel Motor Driven Vehicle

Featuring the fast response and flexibility in control allocation, an electric vehicle with in-wheel motors is a good platform for implementing advanced vehicle dynamics control. Among many active safety functions of an in-wheel motor driven vehicle (IMDV), lateral stability control is a key technology, which can be realized through torque vectoring. To further advance the lateral stabilization performance of the IMDV, in this article a novel data-driven nonlinear model predictive control (NMPC) is proposed based the recurrent high-order neural network (RHONN) modelling method. First, the new RHONN model is developed to represent vehicle’s nonlinear dynamic behaviors. Different from the conventional physics-based modelling method, the RHONN model forms high-order polynomials by neuron states to feature nonlinear dynamics. Based on the RHONN model, the steady-state responses of vehicle’s yaw rate and sideslip angle are iteratively optimized and set as the control objectives for low-level controller, aiming to improve the system robustness. Besides, a nonlinear model predictive controller is designed based on the RHONN, which is expected to improve the prediction accuracy during the receding horizon control. Further, a constrained optimization problem is formulated to derive the required yaw moment for vehicle lateral dynamics stabilization. Finally, the performance of the developed RHONN-based nonlinear MPC is validated on an IMDV in the CarSim/Simulink simulation environment. The validation results show that the developed approach outperforms the conventional method, and further improves the stable margin of the system. It is able to enhance the lateral stabilization performance of the IMDV under various driving scenarios, demonstrating the feasibility and effectiveness of the proposed approach.

Simulation of AC drive control for supercapacitor trams based on high-order neural network pattern discrimination algorithm

Simulation of AC drive control for supercapacitor trams based on high-order neural network pattern discrimination algorithm

Adaptive neural network-based sliding mode control for a hydraulic rotary drive joint

Hydraulic system has specific nonlinear and unknown modeling characteristics, realizing precise tracking control is a very challenging task. In this study, a sliding mode controller based on an adaptive higher-order neural network is proposed for realizing precise position-tracking control of a hydraulic rotary drive joint. First, the structural design and working principle of the target joint are introduced, and the mathematical model of the corresponding valve-controlled hydraulic position servo system is developed. Then, the adaptive neural network algorithm and sliding mode control are effectively combined. Based on the measurement information obtained from the control process, the feedback error is used to adaptively approximate the control parameters and realize online adjustment of the controller output. Finally, the parameter identification and position tracking of the system are validated, and the results indicate that the proposed strategy exhibits a 20% improvement in position-tracking control compared to the conventional sliding mode control methods.

Forecasting Oil Production Flowrate Based on an Improved Backpropagation High-Order Neural Network with Empirical Mode Decomposition

Developing a forecasting model for oilfield well production plays a significant role in managing mature oilfields as it can help to identify production loss earlier. It is very common that mature fields need more frequent production measurements to detect declining production. This study proposes a machine learning system based on a hybrid empirical mode decomposition backpropagation higher-order neural network (EMD-BP-HONN) for oilfields with less frequent measurement. With the individual well characteristic of stationary and non-stationary data, it creates a unique challenge. By utilizing historical well production measurement as a time series feature and then decomposing it using empirical mode decomposition, it generates a simpler pattern to be learned by the model. In this paper, various algorithms were deployed as a benchmark, and the proposed method was eventually completed to forecast well production. With proper feature engineering, it shows that the proposed method can be a potentially effective method to improve forecasting obtained by the traditional method.

A Graph-Related High-Order Neural Network Architecture via Feature Aggregation Enhancement for Identification Application of Diseases and Pests.

Diseases and pests are essential threat factors that affect agricultural production, food security supply, and ecological plant diversity. However, the accurate recognition of various diseases and pests is still challenging for existing advanced information and intelligence technologies. Disease and pest recognition is typically a fine-grained visual classification problem, which is easy to confuse the traditional coarse-grained methods due to the external similarity between different categories and the significant differences among each subsample of the same category. Toward this end, this paper proposes an effective graph-related high-order network with feature aggregation enhancement (GHA-Net) to handle the fine-grained image recognition of plant pests and diseases. In our approach, an improved CSP-stage backbone network is first formed to offer massive channel-shuffled features in multiple granularities. Secondly, relying on the multilevel attention mechanism, the feature aggregation enhancement module is designed to exploit distinguishable fine-grained features representing different discriminating parts. Meanwhile, the graphic convolution module is constructed to analyse the graph-correlated representation of part-specific interrelationships by regularizing semantic features into the high-order tensor space. With the collaborative learning of three modules, our approach can grasp the robust contextual details of diseases and pests for better fine-grained identification. Extensive experiments on several public fine-grained disease and pest datasets demonstrate that the proposed GHA-Net achieves better performances in accuracy and efficiency surpassing several other existing models and is more suitable for fine-grained identification applications in complex scenes.

Distributed Cooperative Neural Inverse Optimal Control of Microgrids for Island and Grid-Connected Operations

In this paper, a novel distributed secondary cooperative control based on neural inverse optimal control scheme is synthesized for microgrids considering island and grid-connected modes. This approach is developed for each distributed generator to track frequency and voltage predefined values for island mode, and to track active and reactive power references for grid-connected mode, and for smooth switching between these two modes. To achieve these goals, a secondary control layer is proposed to define the required currents trajectories, which then are controlled using the primary controller. Both control layers are synthesized using recurrent high order neural network on-line trained using an extended Kalman filter, which builds up neural models and is to be used to implement the respective inverse optimal controllers. In addition, a pinning technique is also used for achieving synchronization of all distributed generators only requiring neighborhood information. Thus, a large reduction in the communication network complexity is obtained and the control system reliability is improved. Simulations are performed to evaluate the effectiveness of the proposed scheme with a microgrid benchmark. The obtained results illustrate adequate performances of the proposed scheme to operate the microgrids in island mode, grid-connected mode, and to achieve seamless switching between these two modes.

Optimal Operation for Reduced Energy Consumption of an Air Conditioning System Using Neural Inverse Optimal Control

For a comfortable thermal environment, the main parameters are indoor air humidity and temperature. These parameters are strongly coupled, causing the need to search for multivariable control alternatives that allow efficient results. Therefore, in order to control both the indoor air humidity and temperature for direct expansion (DX) air conditioning (A/C) systems, different controllers have been designed. In this paper, a discrete-time neural inverse optimal control scheme for trajectories tracking and reduced energy consumption of a DX A/C system is presented. The dynamic model of the plant is approximated by a recurrent high-order neural network (RHONN) identifier. Using this model, a discrete-time neural inverse optimal controller is designed. Unscented Kalman filter (UKF) is used online for the neural network learning. Via simulation the scheme is tested. The proposed approach effectiveness is illustrated with the obtained results and the control proposal performance against disturbances is validated.

A Hybrid Approach of Spotted Hyena Optimization Integrated with Quadratic Approximation for Training Wavelet Neural Network.

Spotted hyena optimization (SHO) is one of the newly evolved swarm-based metaheuristic optimization methods based on the social life cycle of hyenas. In recent times SHO is being applied to various engineering applications as well as to solve real-life complications. In this paper, we have hybridized SHO with quadratic approximation operator (QAO), termed as QASHO. The proposed QASHO has been scrutinized to enhance the exploitation ability, aiming to achieve global optimum, as QAO performs better within the local confinement region. Furthermore, the proposed approach shows improved strength in terms of escaping from the local minima trap, as in each iteration we discard some of the worst individuals by some suitable ones. To validate the proficiency of the proposed QASHO approach, 28 standard problems have been preferred in connection with IEEE-CEC-2017. The outcome observed from the suggested method has also equated with contemporary metaheuristic approaches. To prove the statistical significance, a nonparametric test has also been accomplished. Additionally as a real-life application, the suggested approach QASHO has utilized to train wavelet higher-order neural networks (HONN) by choosing datasets from the UCI store. The above correlations reveal that QASHO can deal with complex optimization tasks.

Neural High Order Sliding Mode Control for Doubly Fed Induction Generator based Wind Turbines

Wind energy has many advantages because it does not pollute and is an inexhaustible source of energy. In this paper Neural High Order Sliding Mode (NHOSM) control is developed for Doubly Fed Induction Generator (DFIG) based Wind Turbine (WT). The stator winding is directly coupled with the main network, whereas a Back-to-Back converter is installed to connect its rotor to the grid. The proposed control scheme is composed of Recurrent High Order Neural Network (RHONN) trained with the Extended Kalman Filter (EKF), which is used to build-up the DFIG models. Based on such identifier, the High Order Sliding Mode (HOSM) using Super-Twisting (ST) algorithm is synthesized. To show the potential of the selected scheme, a comparison study considering the NHOSM, Conventional Sliding mode (CSM), and the HOSM control is done. To ensure maximum power extractions and to protect the system, the Maximum Point Power Tracking (MPPT) algorithm and the h control are also implemented. Simulation results demonstrate the effectiveness of the proposed scheme for enhancing robustness, reducing chattering, and improving quality and quantity of the generated power.

Software Effort Estimation Development From Neural Networks to Deep Learning Approaches

Software Engineering is a branch of computers that includes the development of structured software applications. Estimation is a significant measure of software engineering projects, and the skill to yield correct effort estimates influences vital economic processes, which include budgeting and bid tenders. But it is challenging to estimate at an initial stage of project development. Numerous conventional and machine learning-based methods are utilized for estimating effort and still, it is a challenge to achieve consistency in precise predictions. In this research exploration, various ANN-based models are compared with conventional algorithmic methods. The study also presents the comparison of results on various datasets from the artificial neural network models, deep learning models, higher-order Neural Network models, leading to the conclusion that hybrid methods yield better results. This paper also includes an analysis of primary data collected from Software Project professionals using the questionnaire method involving questions related to software cost estimation.

Global exponential stability and existence of almost periodic solutions in distribution for Clifford-valued stochastic high-order Hopfield neural networks with time-varying delays

<abstract><p>In this paper, we consider a class of Clifford-valued stochastic high-order Hopfield neural networks with time-varying delays whose coefficients are Clifford numbers except the time delays. Based on the Banach fixed point theorem and inequality techniques, we obtain the existence and global exponential stability of almost periodic solutions in distribution of this class of neural networks. Even if the considered neural networks degenerate into real-valued, complex-valued and quaternion-valued ones, our results are new. Finally, we use a numerical example and its computer simulation to illustrate the validity and feasibility of our theoretical results.</p></abstract>

UKF-Based Neural Training for Nonlinear Systems Identification and Control Improvement

Numerous academic works have addressed the identification and control problem for complex dynamic systems. In recent decades, the use of control algorithms based on neural networks (NNs) has been highlighted, which have shown satisfactory results in the trajectory tracking control for a class of discrete-time nonlinear systems. The present work proposes an efficient learning law for discrete-time recurrent high order neural networks (RHONNs), using a training algorithm based on an unscented Kalman filter (UKF). This learning law is applied through a decentralized neural block control using UKF (DNBC-UKF), for trajectory tracking control of a class of discrete-time nonlinear systems. The proposed controller is experimentally evaluated by means of real-time tests in a two degrees of freedom (DOF) vertical direct-drive robotic manipulator against a decentralized neural block control using EKF (DNBC-EKF). The Lyapunov stability theory is used to show that the identification errors of RHONNs are semi-globally uniformly ultimately bounded (SGUUB), the RHONN weights remains bounded, and the closed-loop tracking errors go to zero.

Symbolic Regression Based Feature Extraction of Shallow Neural-Networks for Identification and Prediction

This paper proposes a feature extraction method to improve the performance of shallow neural-network models with less number of parameters to apply especially on embedded system design at remote applications. Feature extraction method is designed using fuzzy c-means clustering based fuzzy system design cascaded a layer of symbolic operators and functions, respectively. During the training stage of neural-networks, symbolic operators and functions are selected using random-learning theory with the unity internal weights such that based on the prediction performance, optimal sequences are recorded for feature extraction to be utilized on testing phase. Extracted features are here used to empower the single-layer neural-network (SLNN) with sigmoid hyperbolic activation functions, functional-link neural- network (FLNN) with Chebyshev polynomials and Pi-Sigma higher-order neural-network (PSNN) with sigmoid activation functions, respectively. The internal and output parameters of the appended shallow neural-networks are optimized using batch optimization methods. Proposed regression models are first tested on identification of an artificial discrete-time dynamic system and real-time inverted pendulum then also for prediction of the sunspot time-series and traffic density estimation. As a result, the prediction performance of shallow neural networks is improved to be used in future applications.

Semisupervised Classification with High-Order Graph Learning Attention Neural Network

At present, the graph neural network has achieved good results in the semisupervised classification of graph structure data. However, the classification effect is greatly limited in those data without graph structure, incomplete graph structure, or noise. It has no high prediction accuracy and cannot solve the problem of the missing graph structure. Therefore, in this paper, we propose a high-order graph learning attention neural network (HGLAT) for semisupervised classification. First, a graph learning module based on the improved variational graph autoencoder is proposed, which can learn and optimize graph structures for data sets without topological graph structure and data sets with missing topological structure and perform regular constraints on the generated graph structure to make the optimized graph structure more reasonable. Then, in view of the shortcomings of graph attention neural network (GAT) that cannot make full use of the graph high-order topology structure for node classification and graph structure learning, we propose a graph classification module that extends the attention mechanism to high-order neighbors, in which attention decays according to the increase of neighbor order. HGLAT performs joint optimization on the two modules of graph learning and graph classification and performs semisupervised node classification while optimizing the graph structure, which improves the classification performance. On 5 real data sets, by comparing 8 classification methods, the experiment shows that HGLAT has achieved good classification results on both a data set with graph structure and a data set without graph structure.

Second-order component analysis for fault detection

Process monitoring based on neural networks is getting more and more attention. Compared with classical neural networks, high-order neural networks have natural advantages in dealing with heteroscedastic data. However, high-order neural networks might bring the risk of overfitting, which learning both the key information from original data and noises or anomalies. Orthogonal constraints can greatly reduce correlations between extracted features, thereby reducing the overfitting risk. This paper proposes a novel fault detection method called second-order component analysis (SCA). SCA rules out the heteroscedasticity of process data by optimizing a second-order autoencoder with orthogonal constraints. In order to deal with this constrained optimization problem, a geometric conjugate gradient algorithm is adopted in this paper, which performs geometric optimization on the combination of Stiefel manifold and Euclidean manifold. Extensive experiments on the Tennessee-Eastman benchmark process show that SCA outperforms the compared state-of-the-art methods with missed detection rate (MDR) and false alarm rate (FAR).

Evaluation Method of Reservoir Heterogeneity Based on Neural Network Technology

Reservoir is the underground storage and accumulation place of oil and natural gas. The accuracy of reservoir heterogeneity evaluation has great economic value for correctly guiding the production and development of oil and natural gas. The high-order neural network method is used to comprehensively evaluate the heterogeneity of the reservoir. This method was applied to the evaluation of reservoir heterogeneity in the PK area. The results show that the heterogeneity of sandy clastic flow sand bodies is the weakest, the sandy landslide sand bodies are medium, and the turbidity current sand bodies are strongest. The evaluation method of reservoir heterogeneity based on high-order neural network technology effectively solves the problem of inconsistent conclusions of single-parameter evaluation of heterogeneity in conventional methods, and can quantitatively characterize the degree of reservoir heterogeneity.