Stochastic Neural Network Model Research Articles

Mean random dynamical systems and random attractors of stochastic Hopfield neural models with locally Lipschitz noise and time delay

In this paper, we investigate the long term dynamical behaviour of stochastic Hopfield neural network models driven by white noise with time delay, where the nonlinear drift and diffusion terms are both locally Lipschitz continuous. We establish the global-in-time solvability of these systems with locally Lipschitz continuous nonlinear drift and diffusion terms by using mean square uniform estimates of solutions. Furthermore, we demonstrate that the mean random dynamical system generated by the solution operators possesses a unique weak pullback mean random attractor in the Hilbert space L 2 ( Ω ~ , F ι ; l 2 ) × L 2 ( Ω ~ , F ι ; L 2 ( ( − ϱ , 0 ) , l 2 ) ) .

Forecasting Stock Indices: Stochastic and Artificial Neural Network Models

Forecasting Stock Indices: Stochastic and Artificial Neural Network Models

Multi-parameter MRI radiomic features may contribute to predict progression-free survival in patients with WHO grade II meningiomas.

This study aims to investigate the potential value of radiomic features from multi-parameter MRI in predicting progression-free survival (PFS) of patients with WHO grade II meningiomas. Kaplan-Meier survival curves were used for survival analysis of clinical features. A total of 851 radiomic features were extracted based on tumor region segmentation from each sequence, and Max-Relevance and Min-Redundancy (mRMR) algorithm was applied to filter and select radiomic features. Bagged AdaBoost, Stochastic Gradient Boosting, Random Forest, and Neural Network models were built based on selected features. Discriminative abilities of models were evaluated using receiver operating characteristics (ROC) and area under the curve (AUC). Our study enrolled 164 patients with WHO grade II meningiomas. Female gender (p=0.023), gross total resection (GTR) (p<0.001), age <68 years old (p=0.023), and edema index <2.3 (p=0.006) are protective factors for PFS in these patients. Both the Bagged AdaBoost model and the Neural Network model achieved the best performance on test set with an AUC of 0.927 (95% CI, Bagged AdaBoost: 0.834-1.000; Neural Network: 0.836-1.000). The Bagged AdaBoost model and the Neural Network model based on radiomic features demonstrated decent predictive ability for PFS in patients with WHO grade II meningiomas who underwent operation using preoperative multi-parameter MR images, thus bringing benefit for patient prognosis prediction in clinical practice. Our study emphasizes the importance of utilizing advanced imaging techniques such as radiomics to improve personalized treatment strategies for meningiomas by providing more accurate prognostic information that can guide clinicians toward better decision-making processes when treating their patients' conditions effectively while minimizing risks associated with unnecessary interventions or treatments that may not be beneficial.

Predicting COVID-19 Outcomes: Machine Learning Predictions Across Diverse Datasets.

Background The COVID-19 infection has spread rapidly since its emergence and has affected a large part of the global population. With the increasing number of cases, researchers are trying to predict the prognosis of patients by using different data with artificial intelligence methods such as machine learning (ML). In this study, we aimed to predict mortality risk in COVID-19 patients using ML algorithms with different datasets. Methodology In this retrospective study, we evaluated the fever, oxygen saturation, laboratory results, thorax computed tomography (CT) findings, and comorbid diseases at admission to the hospital of 404 patients whose diagnosis was confirmed by the reverse transcription polymerase chain reaction test. Different datasets were created by combining the data. The Synthetic Minority Oversampling Technique was used to reduce the imbalance in the dataset. K-nearest neighbors, support vector machine, stochastic gradient descent, random forest, neural network, naive Bayes, logistic regression, gradient boosting, XGBoost, and AdaBoost models were used to create the ML algorithm, and the accuracy rates of mortality prediction were compared. Results When the dataset was created with CT parenchyma score, pulmonary artery and inferior vena cava diameters, and laboratory results, mortality was predicted with an accuracy of 98.4% with the gradient boosting model. Conclusions The study demonstrates that patient prognosis can be accurately predicted using simple measurements from thorax CT scans and laboratory findings.

Evaluation of Stochastic and Artificial Neural Network Models for Multi-step Lead Forecasting of NDVI

Vegetation degradation is associated with human activities and climate change leading to ecosystem changes and biodiversity losses. To reduce the impacts of vegetation degradation, forecasting of vegetation condition is vital in formulating measures to prevent and reduce the losses. Vegetation indices (VI) obtained from remote sensing data, such as the normalized difference vegetation index (NDVI) are widely used to monitor and forecast vegetation condition. In the present study, a stochastic and artificial neural network (ANN) models were compared in modeling and multi-step lead forecasting of NDVI in the Middle Tana River Basin (MTRB), Kenya. Pixel-wise NDVI data for the period 2000 - 2019 was extracted from the MOD13Q1 product of the Moderate Resolution Imaging Spectroradiometer (MODIS). Time lags of NDVI was used as inputs for the models. The results showed that the ANN model outperforms the stochastic model, with a predicting accuracy of RMSE of 0.07207, MSE of 0.00589 and MAE of 0.06417. The multi-step lead forecasting produced satisfactory results indicating the suitability of the models as tools in forecasting NDVI.

A Kernel-Expanded Stochastic Neural Network

Abstract The deep neural network suffers from many fundamental issues in machine learning. For example, it often gets trapped into a local minimum in training, and its prediction uncertainty is hard to be assessed. To address these issues, we propose the so-called kernel-expanded stochastic neural network (K-StoNet) model, which incorporates support vector regression as the first hidden layer and reformulates the neural network as a latent variable model. The former maps the input vector into an infinite dimensional feature space via a radial basis function kernel, ensuring the absence of local minima on its training loss surface. The latter breaks the high-dimensional non-convex neural network training problem into a series of low-dimensional convex optimization problems, and enables its prediction uncertainty easily assessed. The K-StoNet can be easily trained using the imputation-regularized optimization algorithm. Compared to traditional deep neural networks, K-StoNet possesses a theoretical guarantee to asymptotically converge to the global optimum and enables the prediction uncertainty easily assessed. The performances of the new model in training, prediction and uncertainty quantification are illustrated by simulated and real data examples.

A Hybrid Physics-Based and Stochastic Neural Network Model Structure for Diesel Engine Combustion Events

Estimation of combustion phasing and power production is essential to ensuring proper combustion and load control. However, archetypal control-oriented physics-based combustion models can become computationally expensive if highly accurate predictive capabilities are achieved. Artificial neural network (ANN) models, on the other hand, may provide superior predictive and computational capabilities. However, using classical ANNs for model-based prediction and control can be challenging, since their heuristic and deterministic black-box nature may make them intractable or create instabilities. In this paper, a hybridized modeling framework that leverages the advantages of both physics-based and stochastic neural network modeling approaches is utilized to capture CA50 (the timing when 50% of the fuel energy has been released) along with indicated mean effective pressure (IMEP). The performance of the hybridized framework is compared to a classical ANN and a physics-based-only framework in a stochastic environment. To ensure high robustness and low computational burden in the hybrid framework, the CA50 input parameters along with IMEP are captured with a Bayesian regularized ANN (BRANN) and then integrated into an overall physics-based 0D Wiebe model. The outputs of the hybridized CA50 and IMEP models are then successively fine-tuned with BRANN transfer learning models (TLMs). The study shows that in the presence of a Gaussian-distributed model uncertainty, the proposed hybridized model framework can achieve an RMSE of 1.3 × 10−5 CAD and 4.37 kPa with a 45.4 and 3.6 s total model runtime for CA50 and IMEP, respectively, for over 200 steady-state engine operating conditions. As such, this model framework may be a useful tool for real-time combustion control where in-cylinder feedback is limited.

VISPNN: VGG-inspired Stochastic Pooling Neural Network.

Alcoholism is a disease that a patient becomes dependent or addicted to alcohol. This paper aims to design a novel artificial intelligence model that can recognize alcoholism more accurately. We propose the VGG-Inspired stochastic pooling neural network (VISPNN) model based on three components: (i) a VGG-inspired mainstay network, (ii) the stochastic pooling technique, which aims to outperform traditional max pooling and average pooling, and (iii) an improved 20-way data augmentation (Gaussian noise, salt-and-pepper noise, speckle noise, Poisson noise, horizontal shear, vertical shear, rotation, Gamma correction, random translation, and scaling on both raw image and its horizontally mirrored image). In addition, two networks (Net-I and Net-II) are proposed in ablation studies. Net-I is based on VISPNN by replacing stochastic pooling with ordinary max pooling. Net-II removes the 20-way data augmentation. The results by ten runs of 10-fold cross-validation show that our VISPNN model gains a sensitivity of 97.98±1.32, a specificity of 97.80±1.35, a precision of 97.78±1.35, an accuracy of 97.89±1.11, an F1 score of 97.87±1.12, an MCC of 95.79±2.22, an FMI of 97.88±1.12, and an AUC of 0.9849, respectively. The performance of our VISPNN model is better than two internal networks (Net-I and Net-II) and ten state-of-the-art alcoholism recognition methods.

Stochastic Graph Neural Networks

Graph neural networks (GNNs) model nonlinear representations in graph data with applications in distributed agent coordination, control, and planning among others. Current GNN architectures assume ideal scenarios and ignore link fluctuations that occur due to environment, human factors, or external attacks. In these situations, the GNN fails to address its distributed task if the topological randomness is not considered accordingly. To overcome this issue, we put forth the stochastic graph neural network (SGNN) model: a GNN where the distributed graph convolution module accounts for the random network changes. Since stochasticity brings in a new learning paradigm, we conduct a statistical analysis on the SGNN output variance to identify conditions the learned filters should satisfy for achieving robust transference to perturbed scenarios, ultimately revealing the explicit impact of random link losses. We further develop a stochastic gradient descent (SGD) based learning process for the SGNN and derive conditions on the learning rate under which this learning process converges to a stationary point. Numerical results corroborate our theoretical findings and compare the benefits of SGNN robust transference with a conventional GNN that ignores graph perturbations during learning.

An Empirical Analysis of Corporate Financial Management Risk Prediction Based on Associative Memory Neural Network.

As a human brain-like computational model that can reflect the cognitive function of the brain, the problem of dynamic analysis of associative memory neural networks has attracted the attention of scholars. This paper combines associative memory neural networks with enterprise financial management risks, studies the synchronization control and stability analysis problems of unidirectional associative memory-like human brain amnestic neural networks with perturbation and mixed time-varying time lags, proposes a bidirectional associative memory-like brain stochastic amnestic neural network model with mixed time-varying time lags, designs a discrete-time sampling control strategy based on the model, and studies various types of recent financial risks. Based on the early warning research, based on the associative memory neural network method, we propose to reconstruct the risk categories, including improving the enterprise risk management system, enhancing the awareness of financial risk management from top to bottom, and strengthening the core competitiveness of the enterprise itself and control measures for financing, investment, operation, and cash flow risks.

Bayesian Stochastic Neural Network Model for Turbomachinery Damage Prediction

Turbomachinery often suffers various defects such as impeller cracking, resulting in forced outage, increased maintenance costs, and reduced productivity. Condition monitoring and damage prognostics has been widely used as an increasingly important and powerful tool to improve the system availability, reliability, performance, and maintainability, but still very challenging due to multiple sources of data uncertainties and the complexity of analytics modeling. This paper presents an intelligent probabilistic methodology for anomaly prediction of high-fidelity turbomachine, considering multiple data imperfections and multivariate correlation. The proposed method adeptly integrates several advanced state-of-the-art signal processing and artificial intelligence techniques: wavelet multi-resolution decomposition, Bayesian hypothesis testing, probabilistic principal component analysis, and fuzzy stochastic neural network modeling. The advanced signal processing is employed to reduce dimensionality and to address multivariate correlation and data uncertainty for damage prediction. Instead of conventionally using raw time series data, principal components are utilized in the establishment of stochastic neural network model and anomaly prediction. Bayesian interval hypothesis testing metric is then presented to quantitatively compare the predicted and measured data for model validation and anomaly evaluation, thus providing a confidence indicator to judge the model quality and evaluate the equipment status. Bayesian method is developed in this study for denoising the raw data with multiresolution wavelet decomposition, quantifying the model accuracy, and assessing the equipment status. The dynamic stochastic neural network model is established via the nonlinear autoregressive moving average with exogenous inputs approach. It seamlessly integrates the fuzzy clustering and independent Bernoulli random function into radial basis function neural network. A natural gradient method based on Kullback-Leibler distance criterion is employed to maximize the log-likelihood loss function. The effectiveness of the proposed methodology and procedure is demonstrated with the 11-variable time series data and the forced outage event of a real-world centrifugal compressor.

Stochastic Neural Networks for Cryptocurrency Price Prediction

Over the past few years, with the advent of blockchain technology, there has been a massive increase in the usage of Cryptocurrencies. However, Cryptocurrencies are not seen as an investment opportunity due to the market's erratic behavior and high price volatility. Most of the solutions reported in the literature for price forecasting of Cryptocurrencies may not be applicable for real-time price prediction due to their deterministic nature. Motivated by the aforementioned issues, we propose a stochastic neural network model for Cryptocurrency price prediction. The proposed approach is based on the random walk theory, which is widely used in financial markets for modeling stock prices. The proposed model induces layer-wise randomness into the observed feature activations of neural networks to simulate market volatility. Moreover, a technique to learn the pattern of the reaction of the market is also included in the prediction model. We trained the Multi-Layer Perceptron (MLP) and Long Short-Term Memory (LSTM) models for Bitcoin, Ethereum, and Litecoin. The results show that the proposed model is superior in comparison to the deterministic models.

Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode

Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of information geometry on stochastic neuromanifold, and is known to have ideal convergence properties. Despite its theoretical advantages, the pure natural gradient has some limitations that prevent its practical usage. In order to get the explicit value of the natural gradient, it is required to know true probability distribution of input variables, and to calculate inverse of a matrix with the square size of the number of parameters. Though an adaptive estimation of the natural gradient has been proposed as a solution, it was originally developed for online learning mode, which is computationally inefficient for the learning of large data set. In this paper, we propose a novel adaptive natural gradient estimation for mini-batch learning mode, which is commonly adopted for big data analysis. For two representative stochastic neural network models, we present explicit rules of parameter updates and learning algorithm. Through experiments on three benchmark problems, we confirm that the proposed method has superior convergence properties to the conventional methods.

High Resolution Range Profile Sequence Recognition Based on ARTRBM

In this paper, a novel stochastic neural network model named Attention based Recurrent Temporal Restricted Boltzmann Machine (ARTRBM) is proposed for the poor performance of the traditional HRRP recognition methods on high dimensional sequential data and noisy data. RTRBM is efficient to model high dimensional HRRP sequence because it can extract the information of temporal and spatial correlation between adjacent HRRPs. Attention mechanism is used in sequential data recognition tasks, making the model pay more attention to the major features of recognition. Therefore, the combination of RTRBM and attention mechanism makes our mode extract more internal related features effectivly and choose the important parts of the extracted features. Experiment results show that our proposed model outperforms other traditional methods, indicating that ARTRBM extracts, selects and utilizes the correlation information between adjacent HRRPs effectively, and is suitable for high dimensional data or noise corrupted data.

Attention-Based Recurrent Temporal Restricted Boltzmann Machine for Radar High Resolution Range Profile Sequence Recognition.

The High Resolution Range Profile (HRRP) recognition has attracted great concern in the field of Radar Automatic Target Recognition (RATR). However, traditional HRRP recognition methods failed to model high dimensional sequential data efficiently and have a poor anti-noise ability. To deal with these problems, a novel stochastic neural network model named Attention-based Recurrent Temporal Restricted Boltzmann Machine (ARTRBM) is proposed in this paper. RTRBM is utilized to extract discriminative features and the attention mechanism is adopted to select major features. RTRBM is efficient to model high dimensional HRRP sequences because it can extract the information of temporal and spatial correlation between adjacent HRRPs. The attention mechanism is used in sequential data recognition tasks including machine translation and relation classification, which makes the model pay more attention to the major features of recognition. Therefore, the combination of RTRBM and the attention mechanism makes our model effective for extracting more internal related features and choose the important parts of the extracted features. Additionally, the model performs well with the noise corrupted HRRP data. Experimental results on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset show that our proposed model outperforms other traditional methods, which indicates that ARTRBM extracts, selects, and utilizes the correlation information between adjacent HRRPs effectively and is suitable for high dimensional data or noise corrupted data.

Exploring the limits of learning: Segregation of information integration and response selection is required for learning a serial reversal task.

Animals are proposed to learn the latent rules governing their environment in order to maximize their chances of survival. However, rules may change without notice, forcing animals to keep a memory of which one is currently at work. Rule switching can lead to situations in which the same stimulus/response pairing is positively and negatively rewarded in the long run, depending on variables that are not accessible to the animal. This fact raises questions on how neural systems are capable of reinforcement learning in environments where the reinforcement is inconsistent. Here we address this issue by asking about which aspects of connectivity, neural excitability and synaptic plasticity are key for a very general, stochastic spiking neural network model to solve a task in which rules change without being cued, taking the serial reversal task (SRT) as paradigm. Contrary to what could be expected, we found strong limitations for biologically plausible networks to solve the SRT. Especially, we proved that no network of neurons can learn a SRT if it is a single neural population that integrates stimuli information and at the same time is responsible of choosing the behavioural response. This limitation is independent of the number of neurons, neuronal dynamics or plasticity rules, and arises from the fact that plasticity is locally computed at each synapse, and that synaptic changes and neuronal activity are mutually dependent processes. We propose and characterize a spiking neural network model that solves the SRT, which relies on separating the functions of stimuli integration and response selection. The model suggests that experimental efforts to understand neural function should focus on the characterization of neural circuits according to their connectivity, neural dynamics, and the degree of modulation of synaptic plasticity with reward.

Bifurcation Analysis on Phase-Amplitude Cross-Frequency Coupling in Neural Networks with Dynamic Synapses.

We investigate a discrete-time network model composed of excitatory and inhibitory neurons and dynamic synapses with the aim at revealing dynamical properties behind oscillatory phenomena possibly related to brain functions. We use a stochastic neural network model to derive the corresponding macroscopic mean field dynamics, and subsequently analyze the dynamical properties of the network. In addition to slow and fast oscillations arising from excitatory and inhibitory networks, respectively, we show that the interaction between these two networks generates phase-amplitude cross-frequency coupling (CFC), in which multiple different frequency components coexist and the amplitude of the fast oscillation is modulated by the phase of the slow oscillation. Furthermore, we clarify the detailed properties of the oscillatory phenomena by applying the bifurcation analysis to the mean field model, and accordingly show that the intermittent and the continuous CFCs can be characterized by an aperiodic orbit on a closed curve and one on a torus, respectively. These two CFC modes switch depending on the coupling strength from the excitatory to inhibitory networks, via the saddle-node cycle bifurcation of a one-dimensional torus in map (MT1SNC), and may be associated with the function of multi-item representation. We believe that the present model might have potential for studying possible functional roles of phase-amplitude CFC in the cerebral cortex.

Return scaling cross-correlation forecasting by stochastic time strength neural network in financial market dynamics

A return scaling cross-correlation function of exponential parameter is introduced in the present work, and a stochastic time strength neural network model is developed to predict the return scaling cross-correlations between two real stock market indexes, Shanghai Composite Index and Shenzhen Component Index. In the proposed model, the stochastic time strength function gives a weight for each historical data and makes the model have the effect of random movement. The empirical research is performed in testing the model forecasting effect of long-term cross-correlation relationships by training short-term cross-correlations, and a corresponding comparison analysis is made to the backpropagation neural network model. The empirical results show that the proposed neural network is advantageous in increasing the forecasting precision.

A stochastic artificial neural network model for investigating street vendor behavior in a night market

This article offers a hybrid computational approach that combines an artificial neural network with Bayesian probability to improve on the conventional artificial neural network model. The artificial neural network model, which is renowned for its pattern classification abilities, is a type of deterministic algorithm. However, combining artificial neural network with Bayesian probability can convert the deterministic artificial neural network model into a stochastic artificial neural network model that is useful for conducting dynamic simulations. In this study, an experiment is performed to demonstrate this hybrid computational approach. The objective of this experiment is to analyze the behavior of illegal street vendors in a night market. By applying the hybrid computational approach, we can perform a series of dynamic simulations to investigate the development process of the illegal street vendors. The results of the dynamic simulation have high similarity with the real observations. Furthermore, we can use the simulation results to evaluate the commercial values of different parts of streets and to determine which streets will be unstable due to the impacts of economic fluctuations.

Fuzzy stochastic neural network model for structural system identification

This paper presents a dynamic fuzzy stochastic neural network model for nonparametric system identification using ambient vibration data. The model is developed to handle two types of imprecision in the sensed data: fuzzy information and measurement uncertainties. The dimension of the input vector is determined by using the false nearest neighbor approach. A Bayesian information criterion is applied to obtain the optimum number of stochastic neurons in the model. A fuzzy C-means clustering algorithm is employed as a data mining tool to divide the sensed data into clusters with common features. The fuzzy stochastic model is created by combining the fuzzy clusters of input vectors with the radial basis activation functions in the stochastic neural network. A natural gradient method is developed based on the Kullback–Leibler distance criterion for quick convergence of the model training. The model is validated using a power density pseudospectrum approach and a Bayesian hypothesis testing-based metric. The proposed methodology is investigated with numerically simulated data from a Markov Chain model and a two-story planar frame, and experimentally sensed data from ambient vibration data of a benchmark structure.