Quadratic Neurons Research Articles

Quadratic Unet for seismic random noise attenuation

In the field of seismic data processing, seismic denoising is essential to improve the quality of seismic data. Various deep learning methods have been proposed, showing promising performance for seismic denoising. However, most deep learning methods are based on lin ear neurons, resulting in limited expressive ability for complex seismic signals. To enhance denoising capability using non-linear neurons, we propose a supervised quadratic U-shaped network (QUnet) for seismic random noise attenuation. The quadratic neurons in QUnet are represented by a second-order polynomial of input data and weighted parameters. Nu merical synthetic seismic data experiments show that the proposed QUnet method achieves higher signal-to-noise ratio results and preserves the continuity of reflectors more effectively. We have also tested the effectiveness of QUnet on both prestack field data and post-stack seismic data. The detailed structures and weak signals of seismic data are better preserved by our proposed QUnet method.

A Mean Field to Capture Asynchronous Irregular Dynamics of Conductance-Based Networks of Adaptive Quadratic Integrate-and-Fire Neuron Models.

Mean-field models are a class of models used in computational neuroscience to study the behavior of large populations of neurons. These models are based on the idea of representing the activity of a large number of neurons as the average behavior of mean-field variables. This abstraction allows the study of large-scale neural dynamics in a computationally efficient and mathematically tractable manner. One of these methods, based on a semianalytical approach, has previously been applied to different types of singleneuron models, but never to models based on a quadratic form. In this work, we adapted this method to quadratic integrate-and-fire neuron models with adaptation and conductance-based synaptic interactions. We validated the mean-field model by comparing it to the spiking network model. This mean-field model should be useful to model large-scale activity based on quadratic neurons interacting with conductance-based synapses.

Piecewise quadratic neuron model: A tool for close-to-biology spiking neuronal network simulation on dedicated hardware.

Spiking neuron models simulate neuronal activities and allow us to analyze and reproduce the information processing of the nervous system. However, ionic-conductance models, which can faithfully reproduce neuronal activities, require a huge computational cost, while integral-firing models, which are computationally inexpensive, have some difficulties in reproducing neuronal activities. Here we propose a Piecewise Quadratic Neuron (PQN) model based on a qualitative modeling approach that aims to reproduce only the key dynamics behind neuronal activities. We demonstrate that PQN models can accurately reproduce the responses of ionic-conductance models of major neuronal classes to stimulus inputs of various magnitudes. In addition, the PQN model is designed to support the efficient implementation on digital arithmetic circuits for use as silicon neurons, and we confirm that the PQN model consumes much fewer circuit resources than the ionic-conductance models. This model intends to serve as a tool for building a large-scale closer-to-biology spiking neural network.

Attention-Embedded Quadratic Network (Qttention) for Effective and Interpretable Bearing Fault Diagnosis

Bearing fault diagnosis is of great importance to decrease the damage risk of rotating machines and further improve economic profits. Recently, machine learning, represented by deep learning, has made great progress in bearing fault diagnosis. However, applying deep learning to such a task still faces major challenges such as effectiveness and interpretability: i) When bearing signals are highly corrupted by noise, the performance of deep learning models drops dramatically; ii) A deep network is notoriously a black box. It is difficult to know how a model classifies faulty signals from the normal and the physics principle behind the classification. To solve these issues, first, we prototype a convolutional network with recently-invented quadratic neurons. This quadratic neuron-empowered network can qualify the noisy bearing data due to the strong feature representation ability of quadratic neurons. Moreover, we independently derive the attention mechanism from a quadratic neuron, referred to as qttention, by factorizing the learned quadratic function in analogue to the attention, making the model made of quadratic neurons inherently interpretable. Experiments on the public and our datasets demonstrate that the proposed network can facilitate effective and interpretable bearing fault diagnosis. Our code is available at https://github.com/asdvfghg/ QCNN_for_bearing_diagnosis.

Superiority of quadratic over conventional neural networks for classification of gaussian mixture data

To enrich the diversity of artificial neurons, a type of quadratic neurons was proposed previously, where the inner product of inputs and weights is replaced by a quadratic operation. In this paper, we demonstrate the superiority of such quadratic neurons over conventional counterparts. For this purpose, we train such quadratic neural networks using an adapted backpropagation algorithm and perform a systematic comparison between quadratic and conventional neural networks for classificaiton of Gaussian mixture data, which is one of the most important machine learning tasks. Our results show that quadratic neural networks enjoy remarkably better efficacy and efficiency than conventional neural networks in this context, and potentially extendable to other relevant applications.

Artificial intelligence in a rugged design based on multi-bit rules

In this paper, the technologies for training large artificial neural networks are considered: the first technology is based on the use of multilayer “deep” neural networks; the second technology involves the use of a “wide” single-layer network of neurons giving 256 private binary solutions. A list of attacks aimed at the simplest one-bit neural network decision rule is given: knowledge extraction attacks and software data modification attacks; their content is considered. All single-bit decision rules are unsafe for applying. It is necessary to use other decision rules. The security of applying neural network decision rules in relation to deliberate hacker attacks is significantly reduced if you use a decision rule of a large number of output bits. The most important property of neural network transducers is that when it is trained using 20 examples of the “Friend” image, the “Friend” output code of 256 bits long is correctly reproduced with a confidence level of 0.95. This means that the entropy of the “Friend” output codes is close to zero. A well-trained neural network virtually eliminates the ambiguity of the “Friend” image data. On the contrary, for the “Foe” images, their initial natural entropy is enhanced by the neural network. The considered works made it possible to create a draft of the second national standard for automatic training of networks of quadratic neurons with multilevel quantizers.

Universal approximation with quadratic deep networks

Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant quadratic neuron enjoys an enhanced expressive capability over the conventional neuron. However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been studied up to now, preferably in relation to that of a conventional neural network. Specifically, we ask four basic questions in this paper: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight into universal approximation? (4) To approximate the same class of functions with the same error bound, could a quantized quadratic network have a lower number of weights than a quantized conventional network? Our main contributions are the four interconnected theorems shedding light upon these four questions and demonstrating the merits of a quadratic network in terms of expressive efficiency, unique capability, compact architecture and computational capacity respectively.

Quadratic Autoencoder (Q-AE) for Low-Dose CT Denoising.

Inspired by complexity and diversity of biological neurons, our group proposed quadratic neurons by replacing the inner product in current artificial neurons with a quadratic operation on input data, thereby enhancing the capability of an individual neuron. Along this direction, we are motivated to evaluate the power of quadratic neurons in popular network architectures, simulating human-like learning in the form of "quadratic-neuron-based deep learning". Our prior theoretical studies have shown important merits of quadratic neurons and networks in representation, efficiency, and interpretability. In this paper, we use quadratic neurons to construct an encoder-decoder structure, referred as the quadratic autoencoder, and apply it to low-dose CT denoising. The experimental results on the Mayo low-dose CT dataset demonstrate the utility and robustness of quadratic autoencoder in terms of image denoising and model efficiency. To our best knowledge, this is the first time that the deep learning approach is implemented with a new type of neurons and demonstrates a significant potential in the medical imaging field.

Fuzzy logic interpretation of quadratic networks

Over past several years, deep learning has achieved huge successes in various applications. However, such a data-driven approach is often criticized for lack of interpretability. Recently, we proposed artificial quadratic neural networks consisting of quadratic neurons in potentially many layers. In cellular level, a quadratic function is used to replace the inner product in a traditional neuron, and then undergoes a nonlinear activation. With a single quadratic neuron, any fuzzy logic operation, such as XOR, can be implemented. In this sense, any deep network constructed with quadratic neurons can be interpreted as a deep fuzzy logic system. Since traditional neural networks and quadratic counterparts can represent each other and fuzzy logic operations are naturally implemented in quadratic neural networks, it is plausible to explain how a deep neural network works with a quadratic network as the system model. In this paper, we generalize and categorize fuzzy logic operations implementable with individual quadratic neurons, and then perform statistical/information-theoretic analyses of exemplary quadratic neural networks.

GMDH-GEP to predict free span expansion rates below pipelines under waves

ABSTRACTIn this research, group method of data handling (GMDH) as a one of the self-organized approaches is utilized to predict three-dimensional free span expansion rates around pipeline due to waves. The GMDH network is developed using gene-expression programming (GEP) algorithm. In this way, GEP was performed in each neuron of GMDH instead of polynomial quadratic neuron. Effective parameters on the three-dimensional scour rates include sediment size, pipeline geometry, and wave characteristics upstream of pipeline. Four-dimensionless parameters are considered as input variables by means of dimensional analysis technique. Furthermore, scour rates along the pipeline, vertical scour rate, and additionally scour rates in the left and right of pipeline are determined as output parameters. Results of the proposed GMDH-GEP models for the training stages and testing ones are evaluated using various statistical indices. Performances of the GMDH-GEP models are compared with artificial neural network (ANN), GEP, GMDH, and traditional equations-based regression models. Moreover, sensitivity analysis and parametric study are conducted to perceive influences of different input parameters on the three-dimensional scour rates.

New neural network-based response surface method for reliability analysis of structures

In this paper, a new algorithm is introduced for reliability analysis of structures using response surface method based on a group method of data handling-type neural networks with general structure (GS-GMDH-type NN). A multilayer network of quadratic neurons, GMDH, offers an effective solution to modeling nonlinear systems without an explicit limit state function. In the proposed method, the response surface function is determined using GMDH-type neural networks. This is then connected to a reliability method, such as first-order or second-order reliability methods (FORM or SORM) or Monte Carlo simulation method to predict the failure probability (Pf). In the proposed method, the use of the GMDH-type neural network with general structure, where all neurons from previous layers are used to produce neurons in the new layer, can improve the limit state function. In addition, the structure of the neural network and its weight are simultaneously optimized by genetic algorithm and singular value decomposition. As a result, the obtained model has no significant error, despite its simplicity. Moreover, the obtained limit state function is explicit and allows direct use of FORM and SORM methods. To determine the accuracy and efficiency of the proposed method, four numerical examples are solved and their results are compared to other conventional methods. The results show that the proposed method is simply applicable to analyzing the reliability of large complex and sophisticated structures without an explicit limit state function. The proposed approach is a high accurate method that can significantly reduce computing time compared with direct Monte Carlo method.

BLDC Motor Control System Based on Quadratic Single Neuron Adaptive PID Algorithm

Brushless DC (BLDC) motors are widely used for many industrial applications because of their high efficiency, high torque and low volume. In view of the problem that the current control method of speed regulation system of BLDC motor has poor control effect caused by fixed parameters of PID controller, an adaptive PID algorithm with quadratic single neuron (QSN) was designed. Quadratic performance index was introduced in adjustment of weight coefficients; expected optimization effect was gotten by calculating control law. QSN adaptive PID controller can change its parameters online when operating conditions are changed, it can also change its control characteristic automatically. Matlab simulations and experiment results showed that the proposed approach has less overshoot, faster response, stronger ability of anti-disturbance, the results also showed more effectiveness and efficiency than the conventional PID model in motor speed control.

Group Method of Data Handling for Modeling Magnetorheological Dampers

This paper proposes the use of Group Method of Data Handling (GMDH) technique for modeling Magneto-Rheological (MR) dampers in the context of system identification. GMDH is a multilayer network of quadratic neurons that offers an effective solution to modeling non-linear systems. As such, we propose the use of GMDH to approximate the forward and inverse dynamic behaviors of MR dampers. We also introduce two enhanced GMDH-based solutions. Firstly, a two-tier architecture is proposed whereby an enhanced GMD model is generated by the aid of a feedback scheme. Secondly, stepwise regression is used as a feature selection method prior to GMDH modeling. The proposed enhancements to GMDH are found to offer improved prediction results in terms of reducing the root-mean-squared error by around 40%.

A neural tree and its application to spam e-mail detection

This paper presents a new approach to constructing a neural tree to integrate the advantages of decision trees and neural networks. The proposed neural tree, called a quadratic-neuron-based neural tree (QUANT), is a tree-structured neural network composed of neurons with quadratic neural-type junctions for pattern classification. A quadratic neuron is capable of forming a hyper-ellipsoid that can be varied in sizes and in locations on the space spanned by the input variables. Via a batch-mode training algorithm, the QUANT grows a neural tree containing quadratic neurons in its nodes. These quadratic neurons recursively partition the feature space into hyper-ellipsoidal-shaped sub-regions. The QUANT has the partial incremental capability so that it does not need to re-construct a new neural tree to accommodate new training data whenever new data are introduced to a trained QUANT. To demonstrate the performance of the proposed QUANT, one pattern recognition problem and the spam e-mail detection problem were tested.

Response of integrate-and-fire neurons to noisy inputs filtered by synapses with arbitrary timescales: firing rate and correlations

Delivery of neurotransmitter produces on a synapse a current that flows through the membrane and gets transmitted into the soma of the neuron, where it is integrated. The decay time of the current depends on the synaptic receptor's type and ranges from a few (e.g., AMPA receptors) to a few hundred milliseconds (e.g., NMDA receptors). The role of the variety of synaptic timescales, several of them coexisting in the same neuron, is at present not understood. A prime question to answer is which is the effect of temporal filtering at different timescales of the incoming spike trains on the neuron's response. Here, based on our previous work on linear synaptic filtering, we build a general theory for the stationary firing response of integrate-and-fire (IF) neurons receiving stochastic inputs filtered by one, two, or multiple synaptic channels, each characterized by an arbitrary timescale. The formalism applies to arbitrary IF model neurons and arbitrary forms of input noise (i.e., not required to be gaussian or to have small amplitude), as well as to any form of synaptic filtering (linear or nonlinear). The theory determines with exact analytical expressions the firing rate of an IF neuron for long synaptic time constants using the adiabatic approach. The correlated spiking (cross-correlations function) of two neurons receiving common as well as independent sources of noise is also described. The theory is illustrated using leaky, quadratic, and noise-thresholded IF neurons. Although the adiabatic approach is exact when at least one of the synaptic timescales is long, it provides a good prediction of the firing rate even when the timescales of the synapses are comparable to that of the leak of the neuron; it is not required that the synaptic time constants are longer than the mean interspike intervals or that the noise has small variance. The distribution of the potential for general IF neurons is also characterized. Our results provide powerful analytical tools that can allow a quantitative description of the dynamics of neuronal networks with realistic synaptic dynamics.

Application of quadratic neural networks to seismic signal classification

This paper solves the seismic signal classification problem using the quadratic neural networks with closed-boundary discriminating surfaces. In this study, we have demonstrated the quadratic neural network (QNN) potential capabilities in application to the seismic signal classification problems and show that the efficiency achieved here, is much better to what obtained with conventional multilayer neural networks. Firstly, we have performed some pre-processing on the long period recordings to cancel out the instrumental and attenuation side effects. Secondly, we have extracted the ARMA filter coefficients of the windowed P-wave phase through some matrix manipulations using the conventional Prony ARMA modeling scheme. The derived coefficients are then applied to QNN for training and classification. The results have shown that a quadratic neuron is likely to have a performance similar to that of a multilayer perceptron when the target is to discriminate distribution of points in clusters within the input space.