New Class Of Neural Networks Research Articles

Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks.

Modeling and simulation of complex non-linear systems are essential in physics, engineering, and signal processing. Neural networks are widely regarded for such tasks due to their ability to learn complex representations from data. Training deep neural networks traditionally requires large amounts of data, which may not always be readily available for such systems. Contrarily, there is a large amount of domain knowledge in the form of mathematical models for the physics/behavior of such systems. A new class of neural networks called Physics-Informed Neural Networks (PINNs) has gained much attention recently as a paradigm for combining physics into neural networks. They have become a powerful tool for solving forward and inverse problems involving differential equations. A general framework of a PINN consists of a multi-layer perceptron that learns the solution of the partial differential equation (PDE) along with its boundary/initial conditions by minimizing a multi-objective loss function. This is formed by the sum of individual loss terms that penalize the output at different collocation points based on the differential equation and initial and boundary conditions. However, multiple loss terms arising from PDE residual and boundary conditions in PINNs pose a challenge in optimizing the overall loss function. This often leads to training failures and inaccurate results. We propose advanced gradient statistics-based weighting schemes for PINNs to address this challenge. These schemes utilize backpropagated gradient statistics of individual loss terms to appropriately scale and assign weights to each term, ensuring balanced training and meaningful solutions. In addition to the existing gradient statistics-based weighting schemes, we introduce kurtosis-standard deviation-based and combined mean and standard deviation-based schemes for approximating solutions of PDEs using PINNs. We provide a qualitative and quantitative comparison of these weighting schemes on 2D Poisson's and Klein-Gordon's equations, highlighting their effectiveness in improving PINN performance.

A wavelet neural network informed by time-domain signal preprocessing for bearing remaining useful life prediction

Rolling bearings are important components in rotating machinery in various industries. Conducting intelligent prognostics for bearing remaining useful life (RUL) prediction plays an important role in the health management and reliability assessment of these machinery systems. While the bearing fault prognosis using measured system response through machine learning techniques has attracted significant attention and demonstrated promising potential, completely data-driven approaches face some challenges in understanding complex domains with data efficient learning. In this research, we develop an integral bearing fault prognosis framework informed by the well-designed time-domain signal preprocessing to conduct the bearing RUL prediction. This framework is built upon physical feature-oriented signal preprocessing and an associated wavelet neural network (WNN). Sequential procedures of time-domain analyses are proposed to extract the physical features of bearing degradation. Empirical mode decomposition (EMD) is specifically chosen owing to its capability of handling bearing fault signals that are nonstationary with underlying nonlinearities. The WNN is a new class of neural networks that combines the classic neural network and wavelet analysis. Here in this research a WNN model built upon B-spline mother wavelet to suit the preceding EMD-based signal preprocessing is constructed to process the bearing degradation features extracted and then identify their correlation with bearing RUL. The combination of EMD-based signal preprocessing and B-spline mother wavelet in WNN enables the network to learn the input-output correlation in a physical sense. Case studies formulated upon multiple bearing datasets and multiple benchmark methods are carried out to systematically validate the proposed framework. The results consistently demonstrate prediction accuracy and performance robustness.

A new class of neural networks for NCPs using smooth perturbations of the natural residual function

We present a new class of neural networks for solving nonlinear complementarity problems (NCPs) based on some family of real-valued functions (denoted by ℱ) that can be used to construct smooth perturbations of the level curve defined by ΦNR(x,y)=0, where ΦNR is the natural residual function (also called the “min” function). We introduce two important subclasses of ℱ, which deserve particular attention because of their significantly different theoretical and numerical properties. One of these subfamilies yields a smoothing function for ΦNR, while the other subfamily only yields a smoothing curve for ΦNR(x,y)=0. We also propose a simple framework for generating functions from these subclasses. Using the smoothing approach, we build two types of neural networks and provide sufficient conditions to guarantee asymptotic and exponential stability of equilibrium solutions. Finally, we present extensive numerical experiments to validate the theoretical results and to illustrate the difference in numerical performance of functions from the two subclasses. Numerical comparisons with existing neural networks for NCPs are also demonstrated.

Построение робастных нейронных сетей с различными функциями потерь

The paper considers the problem of building robust neural networks using different robust loss functions. Applying such neural networks is reasonably when working with noisy data, and it can serve as an alternative to data preprocessing and to making neural network architecture more complex. In order to work adequately, the error back-propagation algorithm requires a loss function to be continuously or two-times differentiable. According to this requirement, two five robust loss functions were chosen (Andrews, Welsch, Huber, Ramsey and Fair). Using the above-mentioned functions in the error back-propagation algorithm instead of the quadratic one allows obtaining an entirely new class of neural networks. For investigating the properties of the built networks a number of computational experiments were carried out. Different values of outliers’ fraction and various numbers of epochs were considered. The first step included adjusting the obtained neural networks, which lead to choosing such values of internal loss function parameters that resulted in achieving the highest accuracy of a neural network. To determine the ranges of parameter values, a preliminary study was pursued. The results of the first stage allowed giving recommendations on choosing the best parameter values for each of the loss functions under study. The second stage dealt with comparing the investigated robust networks with each other and with the classical one. The analysis of the results shows that using the robust technique leads to a significant increase in neural network accuracy and in a learning rate.

Edge-aware filtering with Siamese neural networks

Recent demands to reduce turnaround times and expedite investment decisions in seismic exploration have invited new ways to process and interpret seismic data. Among these ways is a more integrated collaboration between seismic processors and geologist interpreters aiming to build preliminary geologic models for early business impact. A key aspect has been quick and streamlined delivery of clean high-fidelity 3D seismic images via postmigration filtering capabilities. We present a machine learning-based example of such a capability built on recent advances in deep learning systems. In particular, we leverage the power of Siamese neural networks, a new class of neural networks that is powerful at learning discriminative features. Our novel adaptation, edge-aware filtering, employs a deep Siamese network that ranks similarity between seismic image patches. Once the network is trained, we capitalize on the learned features and self-similarity property of seismic images to achieve within-image stacking power endowed with edge awareness. The method generalizes well to new data sets due to the few-shot learning ability of Siamese networks. Furthermore, the learning-based framework can be extended to a variety of noise types in 3D seismic data. Using a convolutional architecture, we demonstrate on three field data sets that the learned representations lead to superior filtering performance compared to structure-oriented filtering. We examine both filtering quality and ease of application in our analysis. Then, we discuss the potential of edge-aware filtering as a data conditioning tool for rapid structural interpretation.

Global exponential synchronization of quaternion-valued memristive neural networks with time delays

This paper extends the memristive neural networks (MNNs) to quaternion field, a new class of neural networks named quaternion-valued memristive neural networks (QVMNNs) is then established, and the problem of drive-response global synchronization of this type of networks is investigated in this paper. Two cases are taken into consideration: one is with the conventional differential inclusion assumption, the other without. Criteria for the global synchronization of these two cases are achieved respectively by appropriately choosing the Lyapunov functional and applying some inequality techniques. Finally, corresponding simulation examples are presented to demonstrate the correctness of the proposed results derived in this paper.

Synchronization control of quaternion-valued memristive neural networks with and without event-triggered scheme.

In this paper, the real-valued memristive neural networks (MNNs) are extended to quaternion field, a new class of neural networks named quaternion-valued memristive neural networks (QVMNNs) is then established. The problem of master-slave synchronization of this type of networks is investigated in this paper. Two types of controllers are designed: the traditional feedback controller and the event-triggered controller. Corresponding synchronization criteria are then derived based on Lyapunov method. Moreover, it is demonstrated that Zeno behavior can be avoided in case of the event-triggered strategy proposed in this work. Finally, corresponding simulation examples are proposed to demonstrate the correctness of the proposed results derived in this work.

Global Uniform Asymptotic Fixed Deviation Stability and Stability for Delayed Fractional-order Memristive Neural Networks with Generic Memductance

In this paper, we study global uniform asymptotic fixed deviation stability and stability for a wide class of memristive neural networks with time-varying delays. Firstly, a new mathematical expression of the generic memductance (memristance) is proposed according to the feature of the memristor and the general current–voltage characteristic and a new class of neural networks is designed. Next, a new concept of stability (fixed deviation stability) is proposed in order to describe veritably the stability characteristics of the discontinuous system and the sufficient conditions are given to guarantee the global uniform asymptotic fixed deviation stability and stability of the new system. Finally, two numerical examples are provided to show the applicability and effectiveness of our main results.

A new class of neural networks and its applications

We present a new class of neural network using a variable structure model of neuron (VSMN). From this structure, we generate four models of neurons. For each model, we study different behaviors such as stable or equilibrium, degraded, hole, alternated, oscillator, harmonic, fractal, and chaos behaviors. Then we design different topologies and architectures of neural networks. These architectures are different from the classical ones; each layer of network contains different models of neurons, neurons can take four models by configuration of VSMN. We also present a numerical study describing the behavior of some models of neuron. We illustrate some results to show the efficiency of this new class of neural networks. We show that these neurons put tracks on their stimulators such as: signal track and half bounded region track with two high and low directions. Two applications in chaos and robotic are also given.

Wavelet neural networks using particle swarm optimization training in modeling regional ionospheric total electron content

Wavelet neural networks (WNNs) are a new class of neural networks (NNs) that has been developed using a combined method of multi-layer artificial neural networks and wavelet analysis (WA). In this paper, WNNs is used for modeling and prediction of total electron content (TEC) of ionosphere with high spatial and temporal resolution. Generally, back-propagation (BP) algorithm is used to train the neural network. While this algorithm proves to be very effective and robust in training many types of network structures, it suffers from certain disadvantages such as easy entrapment in a local minimum and slow convergence. To improve the performance of WNN in training step, the adjustment of network weights using particle swarm optimization (PSO) was proposed. The results obtained in this paper were compared with standard NN (SNN) by BP training algorithm (SNN-BP), SNN by PSO training algorithm (SNN-PSO) and WNN by BP training algorithm (WNN-BP). For numerical experiments, observations collected at 36 GPS stations in 5 days of 2012 from Iranian permanent GPS network (IPGN) are used. The average minimum relative errors in 5 test stations for WNN-PSO, WNN-BP, SNN-BP and SNN-PSO compared with GPS TEC are 10.59%, 12.85%, 13.18%, 13.75% and average maximum relative errors are 14.70%, 17.30%, 18.53% and 20.83%, respectively. Comparison of diurnal predicted TEC values from the WNN-PSO, SNN-BP, SNN-PSO and WNN-BP models with GPS TEC revealed that the WNN-PSO provides more accurate predictions than the other methods in the test area.

Constructing general partial differential equations using polynomial and neural networks

Sum fraction terms can approximate multi-variable functions on the basis of discrete observations, replacing a partial differential equation definition with polynomial elementary data relation descriptions. Artificial neural networks commonly transform the weighted sum of inputs to describe overall similarity relationships of trained and new testing input patterns. Differential polynomial neural networks form a new class of neural networks, which construct and solve an unknown general partial differential equation of a function of interest with selected substitution relative terms using non-linear multi-variable composite polynomials. The layers of the network generate simple and composite relative substitution terms whose convergent series combinations can describe partial dependent derivative changes of the input variables. This regression is based on trained generalized partial derivative data relations, decomposed into a multi-layer polynomial network structure. The sigmoidal function, commonly used as a nonlinear activation of artificial neurons, may transform some polynomial items together with the parameters with the aim to improve the polynomial derivative term series ability to approximate complicated periodic functions, as simple low order polynomials are not able to fully make up for the complete cycles. The similarity analysis facilitates substitutions for differential equations or can form dimensional units from data samples to describe real-world problems.

Improved Complex-valued Radial Basis Function (ICRBF) neural networks on multiple crack identification

This paper introduces a new class of neural networks in complex space called Complex-valued Radial Basis Function (CRBF) neural networks and also an improved version of CRBF called Improved Complex-valued Radial Basis Function (ICRBF) neural networks. They are used for multiple crack identification in a cantilever beam in the frequency domain. The novelty of the paper is that, these complex-valued neural networks are first applied on inverse problems (damage identification) which come under the category of function approximation. The conventional CRBF network was used in the first stage of ICRBF network and in the second stage a reduced search space moving technique was employed for accurate crack identification. The effectiveness of proposed ICRBF neural network was studied first on a single crack identification problem and then applied to a more challenging problem of multiple crack identification in a cantilever beam with zero noise as well as 5% noise polluted signals. The results proved that, the proposed ICRBF and real-valued Improved RBF (IRBF) neural networks have identified the single and multiple cracks with less than 1% absolute mean percentage error as compared to conventional CRBF and RBF neural networks, mainly because of their second stage reduced search space moving technique. It appears that IRBF neural network is a good compromise considering all factors like accuracy, simplicity and computational effort.

Radial-Basis-Function-Network-Based Prediction of Performance and Emission Characteristics in a Bio Diesel Engine Run on WCO Ester

Radial basis function neural networks (RBFNNs), which is a relatively new class of neural networks, have been investigated for their applicability for prediction of performance and emission characteristics of a diesel engine fuelled with waste cooking oil (WCO). The RBF networks were trained using the experimental data, where in load percentage, compression ratio, blend percentage, injection timing, and injection pressure were taken as the input parameters, and brake thermal efficiency (BTE), brake specific energy consumption (BSEC), exhaust gas temperature (), and engine emissions were used as the output parameters. The number of RBF centers was selected randomly. The network was initially trained using variable width values for the RBF units using a heuristic and then was trained by using fixed width values. Studies showed that RBFNN predicted results matched well with the experimental results over a wide range of operating conditions. Prediction accuracy for all the output parameters was above 90% in case of performance parameters and above 70% in case of emission parameters.

Forecasting of Short-Term Electric Load Using Application of Wavelets with Feed-Forward Neural Networks

An accurate and efficient Short-Term Load Forecasting (STLF) plays a vital role for economic operational planning of both regulated and restructured power systems. This paper presents the STLF models by introducing wavelet transforms, in different ways, with feed-forward Neural Networks (NNs). First, a wavelet-based NN is modeled, where the forecasting has been accomplished in three-stages and the wavelet technique is employed to decompose/reconstruct the original signals and non-linearity of the decomposed signals. Second, an Adaptive Wavelet Neural Network (AWNN) is modeled, which is a new class of NN with continuous wavelet function as the hidden layer node's activation function. Unlike the first model, AWNN does not externally decompose/ reconstruct the original signals and, therefore, this model deals with the problem related to loss of high frequency information that might occur in the wavelet-based NN model. AWNN continuously updates the wavelet parameters (translation and dilation) and layer weights through a back-propagation training algorithm as in classical NNs. The performances of these two models are compared with Multi-Layer Perceptron NN (MLPNN) with the application of day-ahead and hour-ahead load forecasting in the California electricity market. The results are also compared with California Independent System Operator (CAISO)'s forecasted system loads. It is found that due to faster and accurate training capability, AWNN outperforms the MLPNN, wavelet-based NN and CAISO load forecasts.

Estimation of the parameters of an infectious disease model using neural networks

In this paper, we propose a realistic mathematical model taking into account the mutual interference among the interacting populations. This model attempts to describe the control (vaccination) function as a function of the number of infective individuals, which is an improvement over the existing susceptible–infective epidemic models. Regarding the growth of the epidemic as a nonlinear phenomenon we have developed a neural network architecture to estimate the vital parameters associated with this model. This architecture is based on a recently developed new class of neural networks known as co-operative and supportive neural networks. The application of this architecture to the present study involves preprocessing of the input data, and this renders an efficient estimation of the rate of spread of the epidemic. It is observed that the proposed new neural network outperforms a simple feed-forward neural network and polynomial regression.

Some thoughts on neural network modelling of microabrasion–corrosion processes

There is increasing interest in the interactions of microabrasion, involving small particles of less than 10 μm in size, with corrosion. This is because such interactions occur in many environments ranging from the offshore to health care sectors. In particular, microabrasion–corrosion can occur in oral processing, where the abrasive components of food interacting with the acidic environment, can lead to degradation of the surface dentine of teeth. Artificial neural networks (ANNs) are computing mechanisms based on the biological brain. They are very effective in various areas such as modelling, classification and pattern recognition. They have been successfully applied in almost all areas of engineering and many practical industrial applications. Hence, in this paper an attempt has been made to model the data obtained in microabrasion–corrosion experiments on polymer/steel couple and a ceramic/lasercarb coating couple using ANN. A multilayer perceptron (MLP) neural network is applied and the results obtained from modelling the tribocorrosion processes will be compared with those obtained from a relatively new class of neural networks namely resource allocation network.

An efficient gradient-based learning algorithm applied to neural networks with selective actuation neurons

A new class of Neural Networks (NN), designated the Multiple Feed-Forward (MFF) networks, and a new gradient-based learning algorithm, Multiple Back-Propagation (MBP), are proposed and analyzed. MF...

Integral-differential calculus computations by short-term potentiation and depression in NTS-pontine pathways

Neural integrators are a class of neural networks that afford extended temporal summation of neural activity or, equivalently, low-pass frequency filtering of neurotransmission [1]. In the respiratory system, a biphasic neural integrator in the carotid chemoreflex response is demonstrated by the classic short-term potentiation (STP) and afterdischarge of phrenic activity [2]. Recently, a new class of neural networks called neural differentiators, which afford high-pass frequency filtering of afferent inputs through short-term depression (STD) of neurotransmission, has been identified in the vagal-carotid control of the expiratory off-switch [3]. The differentiator response is characterized by: 1) a habituation-like adaptation to the vagal Hering-Breuer (HB) reflex lengthening or carotid chemore-flex shortening of expiratory duration; 2) secondary adaptive effects in the form of pontine-mediated and NMDA receptor-dependent post-stimulus rebound and recovery of these vagal or carotid [4](or hypoxia [5]) mediated responses. To elucidate the mechanism of the habituation of the HB reflex, we obtained extracellular field potentials (FPs) in the medial and commissural regions of the nucleus tractus solitarius (NTS) evoked by electrical vagal stimulation (1 min) in urethane-anesthetized, paralyzed and mechanically ventilated rats (n =17) in vivo. FPs from vagal A-fibers (stimulus current, 20–70 μA, response latency, 4–5 ms) revealed a stimulus frequency-dependent (20–80 Hz) STD similar to the synaptic accommodation observed in vitro [6] and the habituation of HB reflex in vivo [4]. FPs from C-fibers (150–350 μA, 20–45 ms) also demonstrated a frequency-dependent (5–20 Hz) STD during afferent activation but their amplitude remained depressed for >50 s post-stimulation. Microinjection of the NMDA receptor antagonist D-APV abolished the short-term memory following C-fiber activation but not the frequency-dependent STD with A- or C-fibers. The characteristics of the vagal A-and C-fiber mediated responses are reminiscent of the type I and type II NTS neurons found in vitro [7]. To elucidate the mechanism of the pontine-mediated adaptation of the HB reflex, we obtained extracellular recordings of pontine neuronal activity simultaneously with phrenic nerve recording during electrical vagal stimulation (1.5×T, 80 Hz, 1 min). Neurons with tonic and respiratory modulated activity were found within or near the parabrachialis (PB) and Kolliker-Fuse (KF) complex. Most tonic neurons in KF (n = 11) were depressed during and/or shortly after vagal stimulation that elicited the characteristic biphasic adaptation of the HB reflex in the phrenic activity, whereas most tonic neurons in medial PB (n = 22) were not affected. Activity of respiratory modulated neurons (n = 4) recorded in KF remained in phase with phrenic activity during vagal stimulation. Results showed that vagal input induced STD of tonic neurotransmission in KF. These findings support a dual-process nonassociative learning model of integral-differential calculus computations in the brain [8] through activity-dependent STP and STD of neurotransmission in primary and secondary afferent pathways.

Habituation based neural networks for spatio-temporal classification

A new class of neural networks is proposed for the dynamic classification of spatio-temporal signals. These networks are designed to classify signals of different durations, taking into account correlations among different signal segments. Such networks are applicable to SONAR and speech signal classification problems, among others. Network parameters are adapted based on the biologically observed habituation mechanism. This allows the storage of contextual information, without a substantial increase in network complexity. We introduce the concept of a complete memory. We then prove mathematically that a network with a complete memory temporal encoding stage followed by a sufficiently powerful feedforward network is capable of approximating arbitrarily well any continuous, causal, time-invariant discrete-time system with a uniformly bounded input domain. The memory mechanisms of the habituation based networks are complete memories, and involve nonlinear transformations of the input signal. In networks such as the time delay neural network (TDNN) [35] and focused gamma networks [8], nonlinearities are present in the feedforward stage only. This distinction is made important by recent theoretical results concerning the limitations of structures with linear temporal encoding stages. Results are reported on classification of high dimensional feature vectors obtained from Banzhaf sonograms.