Small-world Neural Network Research Articles

A Fast Feedforward Small-World Neural Network for Nonlinear System Modeling.

It is well-documented that cross-layer connections in feedforward small-world neural networks (FSWNNs) enhance the efficient transmission for gradients, thus improving its generalization ability with a fast learning. However, the merits of long-distance cross-layer connections are not fully utilized due to the random rewiring. In this study, aiming to further improve the learning efficiency, a fast FSWNN (FFSWNN) is proposed by taking into account the positive effects of long-distance cross-layer connections, and applied to nonlinear system modeling. First, a novel rewiring rule by giving priority to long-distance cross-layer connections is proposed to increase the gradient transmission efficiency when constructing FFSWNN. Second, an improved ridge regression method is put forward to determine the initial weights with high activation for the sigmoidal neurons in FFSWNN. Finally, to further improve the learning efficiency, an asynchronous learning algorithm is designed to train FFSWNN, with the weights connected to the output layer updated by the ridge regression method and other weights by the gradient descent method. Several experiments are conducted on four benchmark datasets from the University of California Irvine (UCI) machine learning repository and two datasets from real-life problems to evaluate the performance of FFSWNN on nonlinear system modeling. The results show that FFSWNN has significantly faster convergence speed and higher modeling accuracy than the comparative models, and the positive effects of the novel rewiring rule, the improved weight initialization, and the asynchronous learning algorithm on learning efficiency are demonstrated.

Detecting stochastic multiresonance in neural networks via statistical complexity measure

This paper employs statistical complexity measure (SCM) to investigate the occurrence of stochastic multiresonance (SMR) induced by noise and time delay in small-world neural networks coupled with FitzHugh-Nagumo (FHN) neurons. Our findings reveal that SCM exhibits four local maxima at four optimal noise levels, providing evidence for the occurrence of quadruple stochastic resonances. When time delay τ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ au$$\\end{document} is taken into account in the information transmission, under moderate noise levels, SCM shows several local maxima when τ=nTe\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ au = nT_{e}$$\\end{document} with n\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$n$$\\end{document} being a positive integer and Te\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$T_{e}$$\\end{document} being the period of subthreshold signal. This indicates the appearance of delay-induced SMR at the multiples of the period of subthreshold signal. Intriguingly, at low noise levels, a strong coherence between time delay and neuronal firing dynamics emerges at τ=nTe-2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ au = nT_{e} - 2$$\\end{document}, as confirmed by a series of SCM maxima at these time delays. Furthermore, the study demonstrates that by adjusting the degrees and sizes of small-world networks, as well as the coupling strength, it is possible to optimize the strength of delay-induced SMR, thus maximizing the detection capability of subthreshold signal. The research results may provide us with an effective approach for understanding the role of time delay in signal detection and information transmission.

Mosaic: in-memory computing and routing for small-world spike-based neuromorphic systems

The brain’s connectivity is locally dense and globally sparse, forming a small-world graph—a principle prevalent in the evolution of various species, suggesting a universal solution for efficient information routing. However, current artificial neural network circuit architectures do not fully embrace small-world neural network models. Here, we present the neuromorphic Mosaic: a non-von Neumann systolic architecture employing distributed memristors for in-memory computing and in-memory routing, efficiently implementing small-world graph topologies for Spiking Neural Networks (SNNs). We’ve designed, fabricated, and experimentally demonstrated the Mosaic’s building blocks, using integrated memristors with 130 nm CMOS technology. We show that thanks to enforcing locality in the connectivity, routing efficiency of Mosaic is at least one order of magnitude higher than other SNN hardware platforms. This is while Mosaic achieves a competitive accuracy in a variety of edge benchmarks. Mosaic offers a scalable approach for edge systems based on distributed spike-based computing and in-memory routing.

A Hub-Based Self-Organizing Algorithm for Feedforward Small-World Neural Network

A Hub-Based Self-Organizing Algorithm for Feedforward Small-World Neural Network

The inverse stochastic resonance in a small-world neuronal network under electromagnetic stimulation

Electromagnetic stimulation is an important neuromodulation technique that modulates the electrical activity of neurons and affects cortical excitability for the purpose of modulating the nervous system. The phenomenon of inverse stochastic resonance is a response mechanism of the biological nervous system to external signals and plays an important role in the signal processing of the nervous system. In this paper, a small-world neural network with electrical synaptic connections was constructed, and the inverse stochastic resonance of the small-world neural network under electromagnetic stimulation was investigated by analyzing the dynamics of the neural network. The results showed that: the Levy channel noise under electromagnetic stimulation could cause the occurrence of inverse stochastic resonance in small-world neural networks; the characteristic index and location parameter of the noise had significant effects on the intensity and duration of the inverse stochastic resonance in neural networks; the larger the probability of randomly adding edges and the number of nearest neighbor nodes in small-world networks, the more favorable the anti-stochastic resonance was; by adjusting the electromagnetic stimulation parameters, a dual regulation of the inverse stochastic resonance of the neural network can be achieved. The results of this study provide some theoretical support for exploring the regulation mechanism of electromagnetic nerve stimulation technology and the signal processing mechanism of nervous system.

Desynchronization of neuronal firing in multiparameter ultrasound stimulation

Low-intensity transcranial ultrasound stimulation, a novel neuromodulation technique, that possesses the advantages of non-invasiveness, high penetration depth, and high spatial resolution, has achieved positive neuromodulation effects in animal studies. But the regulatory mechanism remains controversial. The intramembrane cavitation effect is considered one of the mechanisms for ultrasound neuromodulation. In this study, the modified equations of ultrasonic cavitation bubble dynamics were coupled with the dual-coupled neuron Hindmarsh-Rose model, small-world neural network model, and the Jansen-Rit neural mass model, which simulate simple coupled neurons, complex neuronal networks, and discharge signals in epileptic disorders respectively. The results demonstrated that ultrasound stimulation has an appreciable modulatory effect on neuronal firing desynchronization in Hindmarsh-Rose model and small-world neural network model. The desynchronization effect is related to the stimulation frequency and intensity. Furthermore, ultrasound stimulation has an inhibitory effect on epileptic seizures, and the effect is enhanced by increasing ultrasound frequency from 0.1–1.0 MHz. This is the first combination of ultrasonic intramembrane cavitation effect theory with neurons and neural network firing desynchronization, which can provide guidance of parametric and theories support for the studies of neurological diseases such as epilepsy and Parkinson’s disease.

Detecting the subthreshold signal in a neural network via statistical complexity measure

This paper proposes an information theory approach for detecting the subthreshold signal in a small-world network composed of Fitz Hugh-Nagumo (FHN) neurons. Statistical complexity measure (SCM) and normalized Shannon-entropy (NSE) have been defined based on the specific and nonconsecutive firing time intervals series, and employed to quantify the stochastic multiresonance (SMR) phenomena in this small-world neural network. The results show that there are several maxima of SCM and several minima of NSE at various optimal noise levels, which is regarded as the signature of the occurrence of SMR. This also implies that the subthreshold signal can be accurately detected across multiple levels of noise. More intriguingly, we have discovered that the degree of this neural network can induce the generation of multiple resonance-like behaviors. In addition, our findings demonstrate that the SCM outperforms the traditional signal-to-noise ratio (SNR) in accurately identifying firing dynamical regularity under certain noisy conditions. Furthermore, the SCM can capture subtle resonance-like behavior induced by the degree, which the SNR is unable to detect. Thus, the SCM could have potential power on detecting weak signals in neural networks.

A pruning feedforward small-world neural network by dynamic sparse regularization with smoothing l1/2 norm for nonlinear system modeling

Aiming to solve the problem of the relatively large architecture for the small-world neural network and improve its generalization ability, we propose a pruning feedforward small-world neural network based on a dynamic regularization method with the smoothing l1/2 norm (PFSWNN-DSRL1/2) and apply it to nonlinear system modeling. A feedforward small-world neural network is first constructed by the rewiring rule of Watts–Strogatz. By minimizing the modified error function added with a smoothing l1/2 norm, redundant weights are pruned to generate a sparse architecture. A dynamic adjusting strategy is further designed for the regularization strength to balance the tradeoff between the training accuracy and the sparsity. Several experiments are carried out to evaluate the performance of the proposed PFSWNN-DSRL1/2 on nonlinear system modeling. The results show that the PFSWNN-DSRL1/2 can achieve the satisfactory modeling accuracy with an average of 17% pruned weights. The comparative results demonstrate that the generalization performance of the proposed model is improved by 8.1% relative to the baseline method (FSWNN) but with a sparse structure, and the pruning does not degenerate its small-world property.

Energy aware Clustered blockchain data for IoT: An end-to-end lightweight secure & Enroute filtering approach

Internet of things (IoT) include smart homes, smart buildings and smart factories, in which all the devices are connected over the internet through wireless network for data transmission or sharing. IoT faces many security challenges due to sharing the data in an open source. Security and energy are the most focusing challenges in IoT due to its resource constrained nature. IoT devices have low energy that is wasted by the attackers, and the data also gets hacked by the attacker, which leads to poor security. This paper proposed CLIENT (Clustered blockchain IoT with En-route) to provide energy efficient and secure communication. For that, this research proposed five processes. Such as Chaotic map based Elgamal authentication, credit score based clustering, capuchin search optimization based packets routing, signature based Enroute filtering and deep leaning based anomaly packet detection. First process is Chaotic Map based Elgamal Authentication, in which chaotic map is used in Elgamal key generation. Second process is Credit score based clustering which minimizes the complexity and overhead during packet transmission. Third process is Capuchin search optimization based packets routing which is used to select optimal route from cluster member to cluster head (CH). Fourth process is signature based Enroute filtering which used to detect and drop false report from the network. Edwards signature based digital signature algorithm (EdDSA) is used for verification purpose. And final process is deep learning based anomaly packets detection, this process used Katz centrality based feed forward small world neural network (FFSWNN) for packet classification (normal or malicious). All the transactions are stored in blockchain to enhance security. The proposed system implemented by using NS3. The result shows that proposed system achieves superior performance compared to existing works in terms of energy consumption, packet loss rate, end to end delay, routing overhead and network life time, precision, recall, accuracy, false alarm rate, encryption time, decryption time and security strength.

Coherence resonance and stochastic synchronization in a small-world neural network: an interplay in the presence of spike-timing-dependent plasticity

Coherence resonance (CR), stochastic synchronization (SS), and spike-timing-dependent plasticity (STDP) are ubiquitous dynamical processes in biological neural networks. Whether there exists an optimal network and STDP configuration at which CR and SS are both pronounced is a fundamental question of interest that is still elusive. We expect such a configuration to enable the brain to make synergistic and optimal use of these phenomena to process information efficiently. This paper considers a small-world network of excitable Hodgkin–Huxley neurons driven by channel noise and STDP with an asymmetric Hebbian time window. Numerical results indicate specific network topology and STDP parameter intervals in which CR and SS can be simultaneously enhanced. Our results imply that an optimally tuned inherent background noise, STDP rule, and network topology can play a constructive role in enhancing both the time precision of firing and the synchronization in neural systems.

Internet of Things resource monitoring through proactive fault prediction

Continuous monitoring of internet of things (IoT) resources generates a stream of monitoring data and a proactive model works based on the information extracted from this data. Such proactive models often suffer from lack of the availability of relevant data as well as the non-stationary distribution of data. In this work, an incremental deep neural network (DNN) is employed as a predictive model to predict the maintenance notification as instances of data are incrementally added to the existing pool of data. The presented model further addresses the existing challenges through a distribution preserving technique for effective sampling and early detection of the change in data distribution. The proposed model is tested over different benchmark datasets and it shows a fair amount of accuracy in predicting faults. It improves the accuracy at least by 5%–10% in comparison to the existing state-of-the-art methods and the recent fault prediction approaches such as deep small-world neural network (DSWNN), attention octave convolution residual networks (AOC-ResNet50), and distributed log mining using ensemble learning for fault prediction (DLME). The results further demonstrate that the approach is adaptive to imbalance and non-stationary distribution of the monitoring data.

The influence of autapse on synchronous firing in small-world neural networks

The synchronization of the nervous system is strongly related to diseases such as Parkinson’s, epilepsy, and schizophrenia. Given that the existence of autapse has been proved experimentally, the influence of autapse on the synchronization in a neural network is studied numerically. The results show that increasing coupling intensity could destroy the synchronization of the neural firing pattern, and reduce the firing rate in the network. Especially, an inhibition zone, in which the neural firing is inhibited completely, exists for changes of both coupling intensity and time delay in all types of autapses. As a key factor for different types of autapses, the transmission time delay influences the synchronization complicatedly, i.e., increasing time delay could modulate synchronization for different types of autapse and parameter regions. The theoretical results in this paper shed some light on the study about the mechanism of neural synchronization.

Analysis and Optimization of Network Properties for Bionic Topology Hopfield Neural Network Using Gaussian-Distributed Small-World Rewiring Method

The fully connected topology, which coordinates the connection of each neuron with all other neurons, remains the most commonly used structure in Hopfield-type neural networks. However, fully connected neurons may form a highly complex network, resulting in a high training cost and making the network biologically unrealistic. Biologists have observed a small-world topology with sparse connections in the actual brain cortex. The bionic small-world neural network structure has inspired various application scenarios. However, in previous studies, the long-range wirings in the small-world network have been found to cause network instability. In this study, we investigate the influence of neural network training on the small-world topology. The role of the path length and clustering coefficient of neurons is expounded in the neural network training process. We employ Watt and Strogatz’s small-world model as the topology for the Hopfield neural network and conduct computer simulations. We observe that the random existence of neuron connections may cause unstable network energies and generate oscillations during the training process. A new method is proposed to mitigate the instability of small-world networks. The proposed method starts with a neuron as the pattern centroid along the radial, which arranges its wirings in compliance with the Gaussian distribution. The new method is tested on the MNIST handwritten digit dataset. The simulation confirms that the new small-world series has higher stability in terms of the learning accuracy and a higher convergence speed compared with Watt and Strogatz’s small-world model.

Anomaly Detection of Wind Turbines Based on Deep Small-World Neural Network

Anomaly Detection of Wind Turbines Based on Deep Small-World Neural Network

A pruning feedforward small-world neural network based on Katz centrality for nonlinear system modeling

Approaching to the biological neural network, small-world neural networks have been demonstrated to improve the generalization performance of artificial neural networks. However, the architecture of small-world neural networks is typically large and predefined. This may cause the problems of overfitting and time consuming, and cannot obtain an optimal network structure automatically for a given problem. To solve the above problems, this paper proposes a pruning feedforward small-world neural network (PFSWNN), and applies it to nonlinear system modeling. Firstly, a feedforward small-world neural network (FSWNN) is constructed according to the rewiring rule of Watts–Strogatz. Secondly, the importance of each hidden neuron is evaluated based on its Katz centrality. If the Katz centrality of a hidden neuron is below the predefined threshold, this neuron is considered to be an unimportant node and then merged with its most correlated neuron in the same hidden layer. The connection weights are trained using the gradient-based algorithm, and the convergence of the proposed PFSWNN is theoretically analyzed in this paper. Finally, the PFSWNN model is tested on some problems for nonlinear system modeling, including the approximation for a rapidly changing function, CATS missing time-series prediction, four benchmark problems of UCI public datasets and a practical problem for wastewater treatment process. Experimental results demonstrate that PFSWNN exhibits superior generalization performance by small-world property as well as the pruning algorithm, and the training time of PFSWNN is shortened owning to a compact structure.

Anomaly Detection of Wind Turbines Based on Deep Small-World Neural Network

Accurate and efficient condition monitoring is the key to enhance the reliability and security of wind turbines. In recent years, an intelligent anomaly detection method based on deep learning networks has been receiving increasing attention. Since accurately labeled data are usually difficult to obtain in real industries, this paper proposes a novel Deep Small-World Neural Network (DSWNN) on the basis of unsupervised learning to detect the early failures of wind turbines. During network construction, a regular auto-encoder network with multiple restricted Boltzmann machines is first constructed and pre-trained by using unlabeled data of wind turbines. After that, the trained network is transformed into a DSWNN model by randomly add-edges method, where the network parameters are fine-tuned by using minimal amounts of labeled data. In order to guard against the changes and disturbances of wind speed and reduce false alarms, an adaptive threshold based on extreme value theory is presented as the criterion of anomaly judgment. The DSWNN model is excellent in depth mining data characteristics and accurate measurement error. Last, two failure cases of wind turbine anomaly detection are given to demonstrate its validity and accuracy of the proposed methodology contrasted with the deep belief network and deep neural network.

Wind power compound model-free adaptive predictive control based on full wind speed

Aiming at the problem that the existing model-based control strategy cannot fully reflect stochastic fluctuation of wind power, this paper presents a model-free adaptive predictive controller (MFAPC) for variable pitch system with speed disturbance suppression. First, an improved small-world neural network with topology optimization is used for 15-second-ahead forecasting of wind speed, whose rolling time is 1s, and the predicted value is served as a feedforward to obtain the early compensation variation of pitch angle. Second, a function of multi-objective optimization at full wind speed with optimal power point tracking and minimum control variation is constructed, and an advanced one-step adaptive predictive control algorithm for wind power is proposed based on the online estimation and prediction of the time-varying pseudo partial derivative (PPD). Further, the compound MFAPC framework is synthetically obtained, whose closed-loop effectiveness is verified by a BP-built pitch system based on the SCADA data with all working conditions. Robustness of the schemes has been analyzed in terms of parametric uncertainties and different operating conditions, and a detailed comparison is finally presented. The results show that the proposed MFAPC can not only effectively suppress the random disturbance of wind speed, but also meet the stability of wind power and the security of grid-connected at all operating conditions.

Scaling behaviors and self-organized criticality of two-dimensional small-world neural networks

It is widely believed that the brains of human beings work at or near the state of self-organized criticality (SOC). In the present work, we investigate two-dimensional small-world neural networks (2D SWNN) with Bak–Sneppen (BS)-type neurons as their nodes. By taking threshold firing and refractory period as the key features of neurons in the simulations, a few power laws are obtained for suitable range of parameters. The SOC characterized by the power-law distribution of avalanche sizes as well as 1∕f noise emerges in the present model. Moreover, a set of scaling relations are found to exhibit criticality. The exponent for the power spectrum of all return time is α=0.71, which is comparable with what were found in medical experiments.

Dynamic Fault Monitoring of Pitch System in Wind Turbines using Selective Ensemble Small-World Neural Networks

Pitch system failures occur primarily because wind turbines typically work in dynamic and variable environments. Conventional monitoring strategies show limitations of continuously identifying faults in most cases, especially when rapidly changing winds occur. A novel selective-ensemble monitoring strategy is presented to diagnose the most pitch failures using Supervisory Control and Data Acquisition (SCADA) data. The proposed strategy consists of five steps. During the first step, the SCADA data are partitioned according to the turbine’s four working states. Correlation Information Entropy (CIE) and 10 indicators are used to select correlation signals and extract features of the partition data, respectively. During the second step, multiple Small-World Neural Networks (SWNNs) are established as the ensemble members. Regarding the third step, all the features are randomly sampled to train the SWNN members. The fourth step involves using an improved global correlation method to select appropriate ensemble members while in the fifth step, the selected members are fused to obtain the final classification result based on the weighted integration approach. Compared with the conventional methods, the proposed ensemble strategy shows an effective accuracy rate of over 93.8% within a short delay time.

WSBP/NWBP small-world neural network and its performance for wind power forecasting

A small-world neural network has stronger generalization ability with high transfer efficiency than that of the regular neural networks. This paper presents two novel small-world neural networks, the Watts-Strogatz small-world based on a BP neural network (WSBP) and a Newman-Watts small-world neural network based on a BP neural network (NWBP), related to previous research of complex networks. The algorithms are developed separately by adopting WS and NW small-world networks as their topological structures, and their derivation and convergence criterion are progressively discussed. After that, the proposed models are subsequently tested by two typical nonlinear functions which confirm their significant improvement over the regular BP networks and other algorithms. Finally, a wind power prediction system is advanced to verify their generalization abilities, and show that the models are practically feasible and effective with improved accuracy and acceptable forecasting errors caused by wind fluctuation and randomness with a time scale up to 24 h.