Stochastic Neural Networks Research Articles

Multiple Sclerosis Detection via 6-layer Stochastic Pooling Convolutional Neural Network and Multiple-way Data Augmentation

Multiple sclerosis is one of most widespread autoimmune neuroinflammatory diseases which mainly damages body function such as movement, sensation, and vision. Despite of conventional clinical presentation, brain magnetic resonance imaging of white matter lesions is often applied to diagnose multiple sclerosis at the early stage. In this article, we proposed a 6-layer stochastic pooling convolutional neural network with multiple-way data augmentation for multiple sclerosis detection in brain MRI images. Our approach does not demand hand-crafted features unlike those traditional machine learning methods. Via application of stochastic pooling and multiple-way data augmentation, our 6-layer CNN achieved equivalent performance against those deep learning methods which consist of so many layers and parameters that ordinarily bring difficulty to training. The results showed that this 6-layer CNN obtained a sensitivity of 95.98±0.46%, a specificity of 95.67±0.92%, and an accuracy of 95.82±0.58%. According to comparison experiments, our results are better than state-of-the-art approaches. Further, we also conducted ablation experiments to examine the contribution of stochastic pooling and multiple-way data augmentation to the original CNN model. The contrast experiments revealed that our scheme of stochastic pooling and multiple-way data augmentation enhanced the original 6-layer CNN model compared to those using maximum pooling or average pooling and inadequate data augmentation.

Stability of stochastic delay switched neural networks with all unstable subsystems: A multiple discretized Lyapunov-Krasovskii functionals method

This paper considers switching stochastic delay neural networks (SSDNNs) with all unstable subsystems. By using discretized Lyapunov-Krasovskii functions (DLKFs) combined with the dwell time method, exponential stability of SSDNNs with all unstable subsystems are analyzed, and several novel stability criteria in mean square are obtained. Comparing with the existing works, our results focus on all unstable subsystems rather than other combinations such as all stable or partially stable subsystems, which is of more research significance. Finally, the correctness of the conclusion is checked by the feasible solutions of two numerical examples.

SwingNet

Sports analytics in the wild (i.e., ubiquitously) is a thriving industry. Swing tracking is a key feature in sports analytics. Therefore, a centimeter-level tracking resolution solution is required. Recent research has explored deep neural networks for sensor fusion to produce consistent swing-tracking performance. This is achieved by combining the advantages of two sensor modalities (IMUs and depth sensors) for golf swing tracking. Here, the IMUs are not affected by occlusion and can support high sampling rates. Meanwhile, depth sensors produce significantly more accurate motion measurements than those produced by IMUs. Nevertheless, this method can be further improved in terms of accuracy and lacking information for different domains (e.g., subjects, sports, and devices). Unfortunately, designing a deep neural network with good performance is time consuming and labor intensive, which is challenging when a network model is deployed to be used in new settings. To this end, we propose a network based on Neural Architecture Search (NAS), called SwingNet, which is a regression-based automatic generated deep neural network via stochastic neural network search. The proposed network aims to learn the swing tracking feature for better prediction automatically. Furthermore, SwingNet features a domain discriminator by using unsupervised learning and adversarial learning to ensure that it can be adaptive to unobserved domains. We implemented SwingNet prototypes with a smart wristband (IMU) and smartphone (depth sensor), which are ubiquitously available. They enable accurate sports analytics (e.g., coaching, tracking, analysis and assessment) in the wild. Our comprehensive experiment shows that SwingNet achieves less than 10 cm errors of swing tracking with a subject-independent model covering multiple sports (e.g., golf and tennis) and depth sensor hardware, which outperforms state-of-the-art approaches.

Mean square exponential stability of discrete-time Markov switched stochastic neural networks with partially unstable subsystems and mixed delays

In this paper, we study the mean square exponential stability of discrete-time stochastic neural networks with partially unstable subsystems and mixed delays. The mixed delays under consideration involve discrete delay and distributed delay. Moreover, the discrete delay term satisfies the Bernoulli distribution. Different from the deterministic switching, we consider Markov switching and our system has partially unstable subsystems. By constructing a novel Lyapunov–Krasovskii functional and using the stationary distribution of Markov chain, we give sufficient conditions for the mean square exponential stability of the suggested system. Finally, two numerical examples are given to check the theory results.

Input-to-State Stability for Stochastic Delay Neural Networks with Markovian Switching

Input-to-State Stability for Stochastic Delay Neural Networks with Markovian Switching

Mean-square exponential input-to-state stability of stochastic quaternion-valued neural networks with time-varying delays

In this paper, we first consider the stability problem for a class of stochastic quaternion-valued neural networks with time-varying delays. Next, we cannot explicitly decompose the quaternion-valued systems into equivalent real-valued systems; by using Lyapunov functional and stochastic analysis techniques, we can obtain sufficient conditions for mean-square exponential input-to-state stability of the quaternion-valued stochastic neural networks. Our results are completely new. Finally, a numerical example is given to illustrate the feasibility of our results.

Passivity Analysis of Coupled Stochastic Neural Networks with Multiweights

In this paper, we devote to the investigation of passivity in two types of coupled stochastic neural networks (CSNNs) with multiweights and incompatible input and output dimensions. First, some new definitions of passivity are proposed for stochastic systems that may have incompatible input and output dimensions. By utilizing stochastic analysis techniques and Lyapunov functional method, several sufficient conditions are respectively developed for ensuring that CSNNs without and with multiple delay couplings can realize passivity. Besides, the synchronization criteria for CSNNs with multiweights are established by employing the results of output-strictly passivity. Finally, two simulation examples are given to illustrate the validity of the theoretical results.

STOCHASTIC PSEUDOSPIN NEURAL NETWORK WITH TRIDIAGONAL SYNAPTIC CONNECTIONS

Context. To reduce the computational resource time in the problems of diagnosing and recognizing distorted images based on a fully connected stochastic pseudospin neural network, it becomes necessary to thin out synaptic connections between neurons, which is solved using the method of diagonalizing the matrix of synaptic connections without losing interaction between all neurons in the network. Objective. To create an architecture of a stochastic pseudo-spin neural network with diagonal synaptic connections without loosing the interaction between all the neurons in the layer to reduce its learning time. Method. The paper uses the Hausholder method, the method of compressing input images based on the diagonalization of the matrix of synaptic connections and the computer mathematics system MATLAB for converting a fully connected neural network into a tridiagonal form with hidden synaptic connections between all neurons. Results. We developed a model of a stochastic neural network architecture with sparse renormalized synaptic connections that take into account deleted synaptic connections. Based on the transformation of the synaptic connection matrix of a fully connected neural network into a Hessenberg matrix with tridiagonal synaptic connections, we proposed a renormalized local Hebb rule. Using the computer mathematics system “WolframMathematica 11.3”, we calculated, as a function of the number of neurons N, the relative tuning time of synaptic connections (per iteration) in a stochastic pseudospin neural network with a tridiagonal connection Matrix, relative to the tuning time of synaptic connections (per iteration) in a fully connected synaptic neural network. Conclusions. We found that with an increase in the number of neurons, the tuning time of synaptic connections (per iteration) in a stochastic pseudospin neural network with a tridiagonal connection Matrix, relative to the tuning time of synaptic connections (per iteration) in a fully connected synaptic neural network, decreases according to a hyperbolic law. Depending on the direction of pseudospin neurons, we proposed a classification of a renormalized neural network with a ferromagnetic structure, an antiferromagnetic structure, and a dipole glass.

Optimizing COVID-19 vaccine distribution across the United States using deterministic and stochastic recurrent neural networks.

Optimizing COVID-19 vaccine distribution can help plan around the limited production and distribution of vaccination, particularly in early stages. One of the main criteria for equitable vaccine distribution is predicting the geographic distribution of active virus at the time of vaccination. This research developed sequence-learning models to predict the behavior of the COVID-19 pandemic across the US, based on previously reported information. For this objective, we used two time-series datasets of confirmed COVID-19 cases and COVID-19 effective reproduction numbers from January 22, 2020 to November 26, 2020 for all states in the US. The datasets have 310 time-steps (days) and 50 features (US states). To avoid training the models for all states, we categorized US states on the basis of their similarity to previously reported COVID-19 behavior. For this purpose, we used an unsupervised self-organizing map to categorize all states of the US into four groups on the basis of the similarity of their effective reproduction numbers. After selecting a leading state (the state with earliest outbreaks) in each group, we developed deterministic and stochastic Long Short Term Memory (LSTM) and Mixture Density Network (MDN) models. We trained the models with data from each leading state to make predictions, then compared the models with a baseline linear regression model. We also remove seasonality and trends from a dataset of non-stationary COVID-19 cases to determine the effects on prediction. We showed that the deterministic LSTM model trained on the COVID-19 effective reproduction numbers outperforms other prediction methods.

Intermittent delay stabilization of complex-valued stochastic complex network

In this paper, the stabilization issue for complex-valued stochastic Markovian switching complex network with time delay and time-varying coupling structure (CSMNDC) is investigated via intermittent delay feedback control. Different from intermittent control based on current state in previous work, a class of intermittent control on the basis of past state is designed for the first time, by combining with the advantages of aperiodically intermittent control and delay feedback control. Then, some sufficient conditions are derived to guarantee the exponential stability in mean square of CSMNDC based on Lyapunov method, graph theory as well as some techniques of inequalities. In particular, the stabilization of networks is studied on complex space directly without splitting their real and imaginary parts by using complex generalized Itô’s formula. Additionally, both delay feedback control and aperiodically intermittent control are employed to solve the stabilization issue for CSMNDC here. Whereafter, the stabilization issue of complex-valued stochastic Markovian switching Cohen-Grossberg neural network with time delay and time-varying coupling structure is researched as a practical application of our theoretical results. Ultimately, a numerical example is presented to verify the validity and effectiveness of the theoretical results.

Almost sure exponential stability of hybrid stochastic delayed Cohen–Grossberg neural networks

AbstractBenefit from the significant work of Song and Mao (2018), this paper focuses on the almost sure exponential stability of hybrid stochastic delayed Cohen–Grossberg neural network (SDCGNN) with the nonlinear disturbance. By virtue of stationary distribution, Lyapunov stability theory and LMI tool, we will propose that if the corresponding delay‐free stochastic neural network is almost surely exponentially stable, then there exists a positive number τ∗ such that the SDCGNN is also almost surely exponentially stable so long as τ < τ∗. Finally, an example is given to verify the effectiveness of our results, and an implicit lower bound for τ∗ is provided.

Periodic measures of impulsive stochastic differential equations

This paper is concerned with the periodic stochastic differential equations with nonlinear impulses. By using the properties of periodic Markov processes, the existence of periodic measures for the impulsive stochastic equations is established. As applications, we study the existence of periodic measures of impulsive periodic stochastic logistic equations and impulsive periodic stochastic neural networks, respectively.

Implementation of Stochastic Optimization Method-Assisted Radial Basis Neural Network for Transport Phenomenon in Non-Newtonian Third-Grade Fluids: Assessment of Five Optimization Tools

Implementation of radial basis neural network is demonstrated by considering a test case of non-Newtonian third-grade fluid flow and heat transfer through two parallel plates. Five commonly used stochastic optimization methods: genetic algorithm, global search algorithm, multiple starting point algorithm, simulated annealing algorithm, and pattern search algorithm, are employed to optimize the RBNN. Flow of a non-Newtonian third-grade fluid through two parallel plates, subjected to uniform heat flux, is considered. At first, governing equations, describing the flow and heat transfer problem, are solved by the least-square method, a semi-analytical tool. The velocity and the temperature profiles are obtained for different values of third-grade fluid parameter ‘A’, which are then used for training different stochastic optimization method-assisted RBNNs (SOMARBNNs). For proper functioning of RBNN, a suitable value for an important attribute called ‘spread’ is required. Deciding the value for ‘spread’ requires experience and knowledge of neural networks. The present approach makes the selection of proper value of ‘spread’ very easy, and beginners can use the RBNN for problem-solving. With the help of different stochastic optimization methods, the value of spread for the RBNN is determined. Once all SOMARBNNs are trained, the temperature profile and the corresponding third-grade fluid parameter ‘A’ are obtained as output, corresponding to any new velocity profile fed as input. Further, the data for training are perturbed by different levels of noise, and different SOMARBNNs are successfully employed to get the output. The performance evaluation of different SOMARBNNs is carried out in terms of CPU time and error in result. The results indicate that PSAARBNN is better than other SOMARBNNs, as it is able to generate results with high accuracy for both low noise data and high noise data. Moreover, the CPU time requirement by PSAARBNN is lowest.

Simple and Effective Stochastic Neural Networks

Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, robustness against adversarial attack and label noise, and model calibration. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense, learning with label noise, and model calibration.

Application of the edge of chaos in combinatorial optimization* *Project supported by the National Natural Science Foundation of China (Grant No. 12074335) and the National Science and Technology Major Project of the Ministry of Science and Technology of China (Grant No. 2016YFA0300402).

Many problems in science, engineering and real life are related to the combinatorial optimization. However, many combinatorial optimization problems belong to a class of the NP-hard problems, and their globally optimal solutions are usually difficult to solve. Therefore, great attention has been attracted to the algorithms of searching the globally optimal solution or near-optimal solution for the combinatorial optimization problems. As a typical combinatorial optimization problem, the traveling salesman problem (TSP) often serves as a touchstone for novel approaches. It has been found that natural systems, particularly brain nervous systems, work at the critical region between order and disorder, namely, on the edge of chaos. In this work, an algorithm for the combinatorial optimization problems is proposed based on the neural networks on the edge of chaos (ECNN). The algorithm is then applied to TSPs of 10 cities, 21 cities, 48 cities and 70 cities. The results show that ECNN algorithm has strong ability to drive the networks away from local minimums. Compared with the transiently chaotic neural network (TCNN), the stochastic chaotic neural network (SCNN) algorithms and other optimization algorithms, much higher rates of globally optimal solutions and near-optimal solutions are obtained with ECNN algorithm. To conclude, our algorithm provides an effective way for solving the combinatorial optimization problems.

Input-to-state stability of hybrid stochastic systems with unbounded delays and impulsive effects

In this paper, we investigate input-to-state stability (ISS) problem for a general model of hybrid stochastic systems with unbounded delays and impulsive effects, in which stochastic disturbances involve white noise and Markov chain. Firstly, we establish a novel differential inequality with unbounded delays and variable inputs, which extends and improves some existing results. Then, by using the obtained inequality, the sufficient conditions are derived to determine ISS properties on impulsive robustness and impulsive stabilization for the impulsive stochastic systems with unbounded delays. The results not only show that the ISS properties still remain under certain impulsive perturbations for some continuous stable systems, but also indicate that an unstable system can be successfully stabilized to be input-to-state stable by impulses even if the corresponding continuous system is unstable. The obtained criteria are applied to study the ISS problems for impulsive stochastic neural networks and the systems of energy-storing electrical circuit. Finally, two numerical examples and their simulations are given to illustrate the validity of theoretical results.

(2012-6329) Adaptive control design for fixed-time synchronization of fuzzy stochastic cellular neural networks with discrete and distributed delay

This paper studies the fixed-time synchronization problem of fuzzy stochastic cellular neural networks (FSCNNs) with discrete and distributed delay. Compared with the finite-time synchronization in the existing literature, the fixed-time synchronization of FSCNNs is studied for the first time, and the convergence time obtained does not depend on the upper bound of the initial value of the system. In addition, two kinds of control are designed, one is feedback control and the other is adaptive control. Besides, it is the first time to achieve fixed-time synchronization of FSCNNs via adaptive control. Finally, two numerical examples are also proposed to illustrate the practicability and validity of the results we proposed.

Proposed mechanisms of opioid‐induced respiratory suppression mediated by actions at excitatory synapses within the inspiratory network: Results from a neuromechanical computational model of the respiratory control system

Administration of opioids effect the frequency and magnitude of inspiratory motor drive by activating mu-opioid receptors that are located throughout the respiratory control network in the brainstem, specifically in the medulla and pons. The precise neural mechanisms that suppress breathing during opioid use are not fully understood. It is well known that inspiratory neurons within this network, including the pre-BÖtzinger complex, exhibit synchrony with one another, consistent with shared excitation from other neuron populations and/or recurrent mechanisms. One possible target for opioid suppression of inspiratory drive are synapses that drive inspiratory neurons. Reduced excitability of these presynaptic elements would be expected to result in disfacilitation and reduced synchrony among inspiratory neurons. We utilized a computational model of the respiratory network to test the plausibility that reduced synchrony, mediated by decreased conductance of excitatory synapses that drive selected inspiratory neuron populations, and suppresses inspiratory motor drive. Using a stochastic neural network simulator that consisted of discrete “integrate and fire” populations, we simulated an opioid-mediated decrease in the presynaptic excitability by separately reducing conductance at synapses that excite inspiratory driver (I-Driver), inspiratory decrementing (I-Dec), or inspiratory augmenting (I-Aug) neurons. Simulations indicated that decreasing the excitability of recurrent excitatory synapses within the I-Driver neuron population by up to 75% only had a limited depressive effect on the breathing pattern. Decreasing the conductance of the excitatory synaptic population from I-Driver neurons to I-Dec neurons by 50% induced apnea. Reducing the conductance of the recurrent excitatory synapse within the I-Aug neuron population by up to 75% resulted in longer inspiratory phase durations (TI) consistent with that observed during administration of opioids. When the presynaptic conductance of the excitatory synaptic population from I-Driver neurons to I-Aug neurons was reduced, TI decreased and I amplitude increased. Overall, our simulation data support the plausibility of presynaptic effects of opioids on the inspiratory network. Our results also suggest that highly selective, rather than generalized, actions of these drugs on excitatory synapses within the inspiratory network are plausible.

Formula omitted] estimation for stochastic semi-Markovian switching CVNNs with missing measurements and mode-dependent delays

This article is devoted to the H∞ estimation problem for stochastic semi-Markovian switching complex-valued neural networks subject to incomplete measurement outputs, where the time-varying delay also depends on another semi-Markov process. A sequence of random variables with known statistical property is introduced to depict the missing measurement phenomenon. Based on the generalized Itoˆ’s formula in complex form concerning with the semi-Markovian systems, complex-valued reciprocal convex inequality as well as intensive stochastic analysis method, some mode-dependent sufficient conditions are presented guaranteeing the estimation error system to be exponentially mean-square stable with a prespecified H∞ disturbance attenuation level. In addition, the mode-dependent estimator gain matrices are appropriately designed according to the feasible solutions of certain complex matrix inequalities. In the end, one numerical example is provided to illustrate effectiveness of the theoretical results.

Exponential Synchronization of Stochastic Neural Networks with Time-Varying Delays and Lévy Noises via Event-Triggered Control

This study is related to the exponential synchronization problem of stochastic neural networks. A dynamic model of master-slave neural networks is established, which contains time-varying delays and Levy noises. The main purpose is to enable the slave system to follow the master system under the condition of limited communication capacity. Both the master system and the slave system are affected by random noises. Some sufficient conditions are given by means of linear matrix inequality methods which are established by applying Lyapunov functional together with the generalized Dynkin’s formula. Furthermore, a discrete event-triggered control is adopted in master-slave systems, which not only reduces the transmission resources but also avoids the Zeno phenomenon. At last, a numerical example is provided to verify the usefulness of judgment conditions in this study.