Class Of Neural Network Models Research Articles

Fundamentals and recent developments of free-space optical neural networks

Machine learning with artificial neural networks has recently transformed many scientific fields by introducing new data analysis and information processing techniques. Despite these advancements, efficient implementation of machine learning on conventional computers remains challenging due to speed and power constraints. Optical computing schemes have quickly emerged as the leading candidate for replacing their electronic counterparts as the backbone for artificial neural networks. Some early integrated photonic neural network (IPNN) techniques have already been fast-tracked to industrial technologies. This review article focuses on the next generation of optical neural networks (ONNs), which can perform machine learning algorithms directly in free space. We have aptly named this class of neural network model the free space optical neural network (FSONN). We systematically compare FSONNs, IPNNs, and the traditional machine learning models with regard to their fundamental principles, forward propagation model, and training process. We survey several broad classes of FSONNs and categorize them based on the technology used in their hidden layers. These technologies include 3D printed layers, dielectric and plasmonic metasurface layers, and spatial light modulators. Finally, we summarize the current state of FSONN research and provide a roadmap for its future development.

Data processing using deep learning of the generative-adversarial neural network (GAN)

The article is devoted to the consideration of one of the areas of application of modern and promising computer technology – machine learning. This direction is based on the creation of models consisting of neural networks and their deep learning. At present, there is a need to generate new, not yet existing, images of objects of different types. Most often, text files or images act as such objects. To achieve a high quality of results, a generation method based on the adversarial work of two neural networks (generator and discriminator) was once worked out. This class of neural network models is distinguished by the complexity of topography, since it is necessary to correctly organize the structure of neural layers in order to achieve maximum accuracy and minimal error. The described program is created using the Python language and special libraries that extend the set of commands for performing additional functions: working with neural networks Keras (main library), integrating with the operating system Os, outputting graphs Matplotlib, working with data arrays Numpy and others. A description is given of the type and features of each neural layer, as well as the use of library connection functions, input of initial data, compilation and training of the obtained model. Next, the implementation of the procedure for outputting the results of evaluating the errors of the generator and discriminator and the accuracy achieved by the model depending on the number of cycles (eras) of its training is considered. Based on the results of the work, conclusions were drawn and recommendations were made for the use and development of the considered methodology for creating and training generative and adversarial neural networks. Studies have demonstrated the procedure for operating with comparatively simple and accessible, but effective means of a universal Python language with the Keras library to create and teach a complex neural network model. In fact, it has been proved that the use of this method allows to achieve high-quality results of machine learning, previously achievable only when using special software systems for working with neural networks.

Discrete-Time Stochastic Quaternion-Valued Neural Networks with Time Delays: An Asymptotic Stability Analysis

Stochastic disturbances often cause undesirable characteristics in real-world system modeling. As a result, investigations on stochastic disturbances in neural network (NN) modeling are important. In this study, stochastic disturbances are considered for the formulation of a new class of NN models; i.e., the discrete-time stochastic quaternion-valued neural networks (DSQVNNs). In addition, the mean-square asymptotic stability issue in DSQVNNs is studied. Firstly, we decompose the original DSQVNN model into four real-valued models using the real-imaginary separation method, in order to avoid difficulties caused by non-commutative quaternion multiplication. Secondly, some new sufficient conditions for the mean-square asymptotic stability criterion with respect to the considered DSQVNN model are obtained via the linear matrix inequality (LMI) approach, based on the Lyapunov functional and stochastic analysis. Finally, examples are presented to ascertain the usefulness of the obtained theoretical results.

New criteria for global stability of neutral-type Cohen–Grossberg neural networks with multiple delays

The significant contribution of this paper is the addressing the stability issue of neutral-type Cohen–Grossberg neural networks possessing multiple time delays in the states of the neurons and multiple neutral delays in time derivative of states of the neurons. By making the use of a novel and enhanced Lyapunov functional, some new sufficient stability criteria are presented for this model of neutral-type neural systems. The obtained stability conditions are completely dependent of the parameters of the neural system and independent of time delays and neutral delays. A constructive numerical example is presented for the sake of proving the key advantages of the proposed stability results over the previously reported corresponding stability criteria for Cohen–Grossberg neural networks of neutral type. Since, stability analysis of Cohen–Grossberg neural networks involving multiple time delays and multiple neutral delays is a difficult problem to overcome, the investigations of the stability conditions of the neutral-type the stability analysis of this class of neural network models have not been given much attention. Therefore, the stability criteria derived in this work can be evaluated as a valuable contribution to the stability analysis of neutral-type Cohen–Grossberg neural systems involving multiple delays.

Optimized brute-force algorithms for the bifurcation analysis of a binary neural network model.

Bifurcation theory is a powerful tool for studying how the dynamics of a neural network model depends on its underlying neurophysiological parameters. However, bifurcation theory of neural networks has been developed mostly for mean-field limits of infinite-size spin-glass models, for finite-size dynamical systems whose units have a graded, continuous output, and for models with discrete-output neurons that evolve in continuous time. To allow progress on understanding the dynamics of some widely used classes of neural network models with discrete units and finite size, which could not be studied thoroughly with the previous methodology, here we introduced algorithms that perform a semianalytical bifurcation analysis of a finite-size firing-rate neural network model with binary firing rates and discrete-time evolution. In particular, we focus on the case of small networks composed of tens of neurons, to which existing statistical methods are not applicable. Our approach is based on a numerical brute-force search of the stationary and oscillatory solutions of the model, from which we derive analytical expressions of its bifurcation structure by means of the state-to-state transition probability matrix. Our algorithms determine how the network parameters affect the degree of multistability, the emergence and the period of the neural oscillations, and the formation of spontaneous symmetry breaking in the neural populations. While this technique can be applied to networks with arbitrary (generally asymmetric) connectivity matrices, in particular we introduce a highly efficient algorithm for the bifurcation analysis of sparse networks. We also provide some examples of the obtained bifurcation diagrams and a python implementation of the algorithms.

The Vapnik–Chervonenkis dimension of graph and recursive neural networks

The Vapnik–Chervonenkis dimension (VC-dim) characterizes the sample learning complexity of a classification model and it is often used as an indicator for the generalization capability of a learning method. The VC-dim has been studied on common feed-forward neural networks, but it has yet to be studied on Graph Neural Networks (GNNs) and Recursive Neural Networks (RecNNs). This paper provides upper bounds on the order of growth of the VC-dim of GNNs and RecNNs. GNNs and RecNNs are from a new class of neural network models which are capable of processing inputs that are given as graphs. A graph is a data structure that generalizes the representational power of vectors and sequences, via the ability to represent dependencies or relationships between feature vectors. It was shown previously that the ability of recurrent neural networks to process sequences increases the VC-dim when compared to the VC-dim of Neural Networks, which are limited to processing vectors. Since graphs are a more general form than sequences, the question arises how this will affect the VC-dimension of GNNs and RecNNs. A main finding in this paper is that the upper bounds on the VC-dim for GNNs and RecNNs are comparable to the upper bounds for recurrent neural networks. The result also suggests that the generalization capability of such models increases with the number of connected nodes.

Quantized Memory-Augmented Neural Networks

Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intriguing AI tasks that would otherwise be hard to address. This paper concerns the problem of quantizing MANNs. Quantization is known to be effective when we deploy deep models on embedded systems with limited resources. Furthermore, quantization can substantially reduce the energy consumption of the inference procedure. These benefits justify recent developments of quantized multi layer perceptrons, convolutional networks, and RNNs. However, no prior work has reported the successful quantization of MANNs. The in-depth analysis presented here reveals various challenges that do not appear in the quantization of the other networks. Without addressing them properly, quantized MANNs would normally suffer from excessive quantization error which leads to degraded performance. In this paper, we identify memory addressing (specifically, content-based addressing) as the main reason for the performance degradation and propose a robust quantization method for MANNs to address the challenge. In our experiments, we achieved a computation-energy gain of 22× with 8-bit fixed-point and binary quantization compared to the floating-point implementation. Measured on the bAbI dataset, the resulting model, named the quantized MANN (Q-MANN), improved the error rate by 46% and 30% with 8-bit fixed-point and binary quantization, respectively, compared to the MANN quantized using conventional techniques.

Processing of noisy digital images with use of evolving autoencoders

A promising class of neural network models used recently to solve the problems of recognition of noisy images are denoising autoencoders. In particular, the evolutionary approach can be effectively used in DAE to determine the network architecture, weights and learning algorithm. The proposed neural network evolving autoencoder allows efficient processing of noisy images due to the iterative learning procedure even in the presence of local distortions. When using the EDAE for determining the network architecture, weights and learning algorithm, standard evolutionary procedures (population initialization, population assessment, selection, crossover, mutation), as well as the evolutionary algorithm for the ANN adjustment and special chromosome formats are used. The proposed approach to filtering and recognition of noisy images based on the EDAE application is promising for environmental monitoring of landscape and industrial areas

Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines.

Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines (S2Ms), a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. S2Ms perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate and fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based S2Ms outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware.

Adaptive Generalized Eigen-Pairs Extraction Algorithms and Their Convergence Analysis

The generalized Hermitian eigenvalue problem (GHEP) is of great use in modern signal processing. Compared with other methods, neural network model based algorithms provide an efficient way to solve such problems online. Up to now, a class of neural network model based generalized feature extraction algorithms have been reported in the literature. However, most existing noncoupled algorithms suffer from the so-called speed-stability problem. In contrast, coupled learning algorithms estimate the generalized eigenvector and generalized eigenvalue simultaneously in coupled manners. Therefore they can solve the speed-stability problem and perform better than noncoupled algorithms. In this paper, we first propose a coupled generalized system, which is obtained by using the Newton’s method and a novel generalized information criterion. Then, based on this coupled generalized system, we successfully obtain two coupled algorithms with normalization steps for minor/principal generalized eigen-pairs extraction. And the technique of multiple generalized eigen-pairs extraction is also introduced in this paper. The convergence of the proposed algorithms is justified by deterministic discrete-time (DDT) approach, with the efficiency and feasibility validated by experimental results as well. Compared with similar work, the proposed algorithms have lower computational complexity, better numerical stability and higher estimation accuracy.

Bifurcation analysis of a tri-neuron neural network model in the frequency domain

In this paper, a class of neural network models with three neurons is considered. By applying the frequency domain approach and analyzing the associated characteristic equation, the existence of the bifurcation parameter point is determined. If the coefficient μ is chosen as a bifurcation parameter, it is found that Hopf bifurcation occurs when the parameter μ passes through a critical value. The direction and the stability of Hopf bifurcation periodic solutions are determined by the Nyquist criterion and the graphical Hopf bifurcation theorem. Some numerical simulations for justifying the theoretical analysis are also provided.

Modelling commodity value at risk with Psi Sigma neural networks using open–high–low–close data

The motivation for this paper is to investigate the use of a promising class of neural network models, Psi Sigma (PSI), when applied to the task of forecasting the one-day ahead value at risk (VaR) of the oil Brent and gold bullion series using open–high–low–close data. In order to benchmark our results, we also consider VaR forecasts from two different neural network designs, the multilayer perceptron and the recurrent neural network, a genetic programming algorithm, an extreme value theory model along with some traditional techniques such as an ARMA-Glosten, Jagannathan, and Runkle (1,1) model and the RiskMetrics volatility. The forecasting performance of all models for computing the VaR of the Brent oil and the gold bullion is examined over the period September 2001–August 2010 using the last year and half of data for out-of-sample testing. The evaluation of our models is done by using a series of backtesting algorithms such as the Christoffersen tests, the violation ratio and our proposed loss function that considers not only the number of violations but also their magnitude. Our results show that the PSI outperforms all other models in forecasting the VaR of gold and oil at both the 5% and 1% confidence levels, providing an accurate number of independent violations with small magnitude.

Analysis of Nonlinear Dynamics of Neural Networks

Dynamical neural networks proved to be an important tool to solve some practical engineering problems in the areas such as optimization, image and signal processing, control systems, associative memories, and pattern recognition. When employing neural networks to solve practical engineering problems, it is crucial to be able to completely characterize the dynamical properties of neural networks. There are many various classes of neural network models that can be described in the form of nonlinear systems. Therefore, neural networks may exhibit extremely different complex dynamical behaviors depending on the model and network parameters. Hence, the analysis of nonlinear dynamics of neural networks still possesses new challenges to researchers. The aim of this special issue is to solicit theoretical and application research in the fields of neural networks exploiting their nonlinear dynamics. We believe it provides a good opportunity for reflection on current developments in the nonlinear analysis of dynamical behaviors of neural networks.The papers submitted to this special issue represent a mixture of cross-cutting investigations and provide deep insight into the current developments in the field. The accepted papers in this special issue addressed the following topics:

Activity-dependent neural network model on scale-free networks

Networks of living neurons exhibit an avalanche mode of activity, experimentally found in organotypic cultures. Moreover, experimental studies of morphology indicate that neurons develop a network of small-world-like connections, with the possibility of a very high connectivity degree. Here we study a recent model based on self-organized criticality, which consists of an electrical network with threshold firing and activity-dependent synapse strengths. We study the model on a scale-free network, the Apollonian network. The system exhibits an avalanche activity with a power law distribution of sizes and durations. The analysis of the power spectra of the electrical signal reproduces very robustly the power law behavior with the exponent 0.8, experimentally measured in electroencephalogram spectra. The exponents are found to be quite stable with respect to initial configurations and strength of plastic remodeling, indicating that universality holds for a wide class of neural network models.

Model Risk for European-Style Stock Index Options

In empirical modeling, there have been two strands for pricing in the options literature, namely the parametric and nonparametric models. Often, the support for the nonparametric methods is based on a benchmark such as the Black-Scholes (BS) model with constant volatility. In this paper, we study the stochastic volatility (SV) and stochastic volatility random jump (SVJ) models as parametric benchmarks against feedforward neural network (FNN) models, a class of neural network models. Our choice for FNN models is due to their well-studied universal approximation properties of an unknown function and its partial derivatives. Since the partial derivatives of an option pricing formula are risk pricing tools, an accurate estimation of the unknown option pricing function is essential for pricing and hedging. Our findings indicate that FNN models offer themselves as robust option pricing tools, over their sophisticated parametric counterparts in predictive settings. There are two routes to explain the superiority of FNN models over the parametric models in forecast settings. These are nonnormality of return distributions and adaptive learning.

Activation-Based Recursive Self-Organising Maps: A General Formulation and Empirical Results

We generalize a class of neural network models that extend the Kohonen Self-Organising Map (SOM) algorithm into the sequential and temporal domain using recurrent connections. Behaviour of the class of Activation-based Recursive Self-Organising Maps (ARSOM) is discussed with respect to the choice of transfer function and parameter settings. By comparing performances to existing benchmarks we demonstrate the robustness and systematicity of the ARSOM models, thus opening the door to practical applications.

Delay-dependent asymptotic stability for neural networks with time-varying delays

We study the dynamical behavior of a class of neural network models with time-varying delays. By constructing suitable Lyapunov functionals, we obtain sufficient delay-dependent criteria to ensure local and global asymptotic stability of the equilibrium of the neural network. Our results are applied to a two-neuron system with delayed connections between neurons, and some novel asymptotic stability criteria are also derived. The obtained conditions are shown to be less conservative and restrictive than those reported in the known literature. Some numerical examples are included to demonstrate our results.

A HIGH-ORDER GRAPH GENERATING SELF–ORGANIZING STRUCTURE

A large class of neural network models have their units organized in a lattice with fixed topology or generate their topology during the learning process. These network models can be used as neighborhood preserving map of the input manifold, but such a structure is difficult to manage since these maps are graphs with a number of nodes that is just one or two orders of magnitude less than the number of input points (i.e., the complexity of the map is comparable with the complexity of the manifold) and some hierarchical algorithms were proposed in order to obtain a high-level abstraction of these structures. In this paper a general structure capable to extract high order information from the graph generated by a large class of self-organizing networks is presented. This algorithm will allow to build a two layers hierarchical structure starting from the results obtained by using the suitable neural network for the distribution of the input data. Moreover the proposed algorithm is also capable to build a topology preserving map if it is trained using a graph that is also a topology preserving map.

Twentieth Century Shocks, Trends and Cycles in Industrialized Nations

Using annual data on real Gross Domestic Product per capita of seventeen industrialized nations in the twentieth century the empirical relevance of shocks, trends and cycles is investigated. A class of neural network models is specified as an extension of the class of vector autoregressive models in order to capture complex data patterns for different countries and subperiods. Empirical evidence indicates nonlinear positive trends in the levels of real GDP per capita, time varying growth rates, switching behavior of individual countries with respect to their position in the distribution of real GDP per capita levels over time. Such evidence presents challenges for economic modelling, forecasting and policy analysis for the long run.

Context-dependent selection of visuomotor maps

BackgroundBehavior results from the integration of ongoing sensory signals and contextual information in various forms, such as past experience, expectations, current goals, etc. Thus, the response to a specific stimulus, say the ringing of a doorbell, varies depending on whether you are at home or in someone else's house. What is the neural basis of this flexibility? What mechanism is capable of selecting, in a context-dependent way an adequate response to a given stimulus? One possibility is based on a nonlinear neural representation in which context information regulates the gain of stimulus-evoked responses. Here I explore the properties of this mechanism.ResultsBy means of three hypothetical visuomotor tasks, I study a class of neural network models in which any one of several possible stimulus-response maps or rules can be selected according to context. The underlying mechanism based on gain modulation has three key features: (1) modulating the sensory responses is equivalent to switching on or off different subpopulations of neurons, (2) context does not need to be represented continuously, although this is advantageous for generalization, and (3) context-dependent selection is independent of the discriminability of the stimuli. In all cases, the contextual cues can quickly turn on or off a sensory-motor map, effectively changing the functional connectivity between inputs and outputs in the networks.ConclusionsThe modulation of sensory-triggered activity by proprioceptive signals such as eye or head position is regarded as a general mechanism for performing coordinate transformations in vision. The present results generalize this mechanism to situations where the modulatory quantity and the input-output relationships that it selects are arbitrary. The model predicts that sensory responses that are nonlinearly modulated by arbitrary context signals should be found in behavioral situations that involve choosing or switching between multiple sensory-motor maps. Because any relevant circumstancial information can be part of the context, this mechanism may partly explain the complex and rich behavioral repertoire of higher organisms.