Realistic Neural Network Research Articles

Antiferromagnetic artificial neuron modeling of the withdrawal reflex.

Replicating neural responses observed in biological systems using artificial neural networks holds significant promise in the fields of medicine and engineering. In this study, we employ ultra-fast artificial neurons based on antiferromagnetic (AFM) spin Hall oscillators to emulate the biological withdrawal reflex responsible for self-preservation against noxious stimuli, such as pain or temperature. As a result of utilizing the dynamics of AFM neurons, we are able to construct an artificial neural network that can mimic the functionality and organization of the biological neural network responsible for this reflex. The unique features of AFM neurons, such as inhibition that stems from an effective AFM inertia, allow for the creation of biologically realistic neural network components, like the interneurons in the spinal cord and antagonist motor neurons. To showcase the effectiveness of AFM neuron modeling, we conduct simulations of various scenarios that define the withdrawal reflex, including responses to both weak and strong sensory stimuli, as well as voluntary suppression of the reflex.

Neurobiologically realistic neural network enables cross-scale modeling of neural dynamics

Fundamental principles underlying computation in multi-scale brain networks illustrate how multiple brain areas and their coordinated activity give rise to complex cognitive functions. Whereas brain activity has been studied at the micro- to meso-scale to reveal the connections between the dynamical patterns and the behaviors, investigations of neural population dynamics are mainly limited to single-scale analysis. Our goal is to develop a cross-scale dynamical model for the collective activity of neuronal populations. Here we introduce a bio-inspired deep learning approach, termed NeuroBondGraph Network (NBGNet), to capture cross-scale dynamics that can infer and map the neural data from multiple scales. Our model not only exhibits more than an 11-fold improvement in reconstruction accuracy, but also predicts synchronous neural activity and preserves correlated low-dimensional latent dynamics. We also show that the NBGNet robustly predicts held-out data across a long time scale (2 weeks) without retraining. We further validate the effective connectivity defined from our model by demonstrating that neural connectivity during motor behaviour agrees with the established neuroanatomical hierarchy of motor control in the literature. The NBGNet approach opens the door to revealing a comprehensive understanding of brain computation, where network mechanisms of multi-scale activity are critical.

Representation of single neuron dynamics using 1-D and 2-D Discrete dynamical systems

Biological neurons are typically modeled using the Hodgkin-Huxley formalism, which requires significant computational power to simulate. However, since realistic neural network models require thousands of synaptically coupled neurons, a faster approach is needed. Discrete dynamical systems are promising alternatives to continuous models, as they can simulate neuron activity in far fewer steps. Many existing discrete models are based on Poincaré-map-like approaches, which trace periodic activity at a cross section of the cycle. However, this approach is limited to periodic solutions. Biological neurons have many key properties beyond periodicity, such as the minimum applied current required for a resting cell to generate an action potential. To address these properties, we propose a discrete dynamical system model of a biological neuron that incorporates the threshold dynamics of the Hodgkin-Huxley model, the logarithmic relationship between applied current and frequency, modifications to relaxation oscillators, and spike-frequency adaptation in response to modulatory hyperpolarizing currents. It is important to note that several critical parameters are transferred from the continuous model to our proposed discrete dynamical system. These parameters include the membrane capacitance, leak conductance, and maximum conductance values for sodium and potassium ion channels, which are essential for accurately simulating the behavior of biological neurons. By incorporating these parameters into our model, we can ensure that it closely approximates the continuous model’s behavior, while also offering a more computationally efficient alternative for simulating neural networks.

Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regularization Favoring Flat Solutions.

Generalization is one of the most important problems in deep learning, where there exist many low-loss solutions due to overparametrization. Previous empirical studies showed a strong correlation between flatness of the loss landscape at a solution and its generalizability, and stochastic gradient descent (SGD) is crucial in finding the flat solutions. To understand the effects of SGD, we construct a simple model whose overall loss landscape has a continuous set of degenerate (or near-degenerate) minima and the loss landscape for a minibatch is approximated by a random shift of the overall loss function. By direct simulations of the stochastic learning dynamics and solving the underlying Fokker-Planck equation, we show that due to its strong anisotropy the SGD noise introduces an additional effective loss term that decreases with flatness and has an overall strength that increases with the learning rate and batch-to-batch variation. We find that the additional landscape-dependent SGD loss breaks the degeneracy and serves as an effective regularization for finding flat solutions. As a result, the flatness of the overall loss landscape increases during learning and reaches a higher value (flatter minimum) for a larger SGD noise strength before the noise strength reaches a critical value when the system fails to converge. These results, which are verified in realistic neural network models, elucidate the role of SGD for generalization, and they may also have important implications for hyperparameter selection for learning efficiently without divergence.

A Novel Biophysical Model for the Generation of Sharp Wave Ripples in CA1 Hippocampus.

Synchronous activities among neurons in the brain generate emergent network oscillations such as the hippocampal Sharp-wave ripples (SPWRs) that facilitate information processing during memory formation. However, how neurons and circuits are functionally organized to generate oscillations remains unresolved. Biophysical models of neuronal networks can shed light on how thousands of neurons interact in intricate circuits to generate such emergent SPWR network events. Here we developed a large-scale biophysically realistic neural network model of CA1 hippocampus with functionally organized circuit modules containing distinct types of neurons. Model simulations reproduced synaptic, cellular and network aspects of physiological SPWRs. The model provided insights into the role of neuronal types and their microcircuit motifs in generating SPWRs in the CA1 region. The model also suggests experimentally testable predictions including the role of specific neuron types in the genesis of hippocampal SPWRs.

Exact mean-field models for spiking neural networks with adaptation.

Networks of spiking neurons with adaption have been shown to be able to reproduce a wide range of neural activities, including the emergent population bursting and spike synchrony that underpin brain disorders and normal function. Exact mean-field models derived from spiking neural networks are extremely valuable, as such models can be used to determine how individual neurons and the network they reside within interact to produce macroscopic network behaviours. In the paper, we derive and analyze a set of exact mean-field equations for the neural network with spike frequency adaptation. Specifically, our model is a network of Izhikevich neurons, where each neuron is modeled by a two dimensional system consisting of a quadratic integrate and fire equation plus an equation which implements spike frequency adaptation. Previous work deriving a mean-field model for this type of network, relied on the assumption of sufficiently slow dynamics of the adaptation variable. However, this approximation did not succeed in establishing an exact correspondence between the macroscopic description and the realistic neural network, especially when the adaptation time constant was not large. The challenge lies in how to achieve a closed set of mean-field equations with the inclusion of the mean-field dynamics of the adaptation variable. We address this problem by using a Lorentzian ansatz combined with the moment closure approach to arrive at a mean-field system in the thermodynamic limit. The resulting macroscopic description is capable of qualitatively and quantitatively describing the collective dynamics of the neural network, including transition between states where the individual neurons exhibit asynchronous tonic firing and synchronous bursting. We extend the approach to a network of two populations of neurons and discuss the accuracy and efficacy of our mean-field approximations by examining all assumptions that are imposed during the derivation. Numerical bifurcation analysis of our mean-field models reveals bifurcations not previously observed in the models, including a novel mechanism for emergence of bursting in the network. We anticipate our results will provide a tractable and reliable tool to investigate the underlying mechanism of brain function and dysfunction from the perspective of computational neuroscience.

The Spatiotemporal Neural Dynamics of Intersensory Attention Capture of Salient Stimuli: A Large-Scale Auditory-Visual Modeling Study.

The spatiotemporal dynamics of the neural mechanisms underlying endogenous (top-down) and exogenous (bottom-up) attention, and how attention is controlled or allocated in intersensory perception are not fully understood. We investigated these issues using a biologically realistic large-scale neural network model of visual-auditory object processing of short-term memory. We modeled and incorporated into our visual-auditory object-processing model the temporally changing neuronal mechanisms for the control of endogenous and exogenous attention. The model successfully performed various bimodal working memory tasks, and produced simulated behavioral and neural results that are consistent with experimental findings. Simulated fMRI data were generated that constitute predictions that human experiments could test. Furthermore, in our visual-auditory bimodality simulations, we found that increased working memory load in one modality would reduce the distraction from the other modality, and a possible network mediating this effect is proposed based on our model.

PRIMA: general and precise neural network certification via scalable convex hull approximations

Formal verification of neural networks is critical for their safe adoption in real-world applications. However, designing a precise and scalable verifier which can handle different activation functions, realistic network architectures and relevant specifications remains an open and difficult challenge. In this paper, we take a major step forward in addressing this challenge and present a new verification framework, called PRIMA. PRIMA is both (i) general: it handles any non-linear activation function, and (ii) precise: it computes precise convex abstractions involving multiple neurons via novel convex hull approximation algorithms that leverage concepts from computational geometry. The algorithms have polynomial complexity, yield fewer constraints, and minimize precision loss. We evaluate the effectiveness of PRIMA on a variety of challenging tasks from prior work. Our results show that PRIMA is significantly more precise than the state-of-the-art, verifying robustness to input perturbations for up to 20%, 30%, and 34% more images than existing work on ReLU-, Sigmoid-, and Tanh-based networks, respectively. Further, PRIMA enables, for the first time, the precise verification of a realistic neural network for autonomous driving within a few minutes.

A neuromorphic model of olfactory processing and sparse coding in the Drosophila larva brain

Abstract Animal nervous systems are highly efficient in processing sensory input. The neuromorphic computing paradigm aims at the hardware implementation of neural network computations to support novel solutions for building brain-inspired computing systems. Here, we take inspiration from sensory processing in the nervous system of the fruit fly larva. With its strongly limited computational resources of <200 neurons and <1.000 synapses the larval olfactory pathway employs fundamental computations to transform broadly tuned receptor input at the periphery into an energy efficient sparse code in the central brain. We show how this approach allows us to achieve sparse coding and increased separability of stimulus patterns in a spiking neural network, validated with both software simulation and hardware emulation on mixed-signal real-time neuromorphic hardware. We verify that feedback inhibition is the central motif to support sparseness in the spatial domain, across the neuron population, while the combination of spike frequency adaptation and feedback inhibition determines sparseness in the temporal domain. Our experiments demonstrate that such small, biologically realistic neural networks, efficiently implemented on neuromorphic hardware, can achieve parallel processing and efficient encoding of sensory input at full temporal resolution.

Energy-Efficient Neural Network Inference with Microcavity Exciton Polaritons

We propose all-optical neural networks characterized by very high energy efficiency and performance density of inference. We argue that the use of microcavity exciton polaritons allows one to take advantage of the properties of both photons and electrons in a seamless manner. This results in strong optical nonlinearity without the use of optoelectronic conversion. We propose a design of a realistic neural network and estimate energy cost to be at the level of attojoules per bit, also when including the optoelectronic conversion at the input and output of the network, several orders of magnitude below state-of-the-art hardware implementations. We propose two kinds of nonlinear binarized nodes based either on optical phase shifts and interferometry or on polariton spin rotations.

Comprehensive Estimates of Potential Synaptic Connections in Local Circuits of the Rodent Hippocampal Formation by Axonal-Dendritic Overlap.

A quantitative description of the hippocampal formation synaptic architecture is essential for understanding the neural mechanisms of episodic memory. Yet the existing knowledge of connectivity statistics between different neuron types in the rodent hippocampus only captures a mere 5% of this circuitry. We present a systematic pipeline to produce first-approximation estimates for most of the missing information. Leveraging the www.Hippocampome.org knowledge base, we derive local connection parameters between distinct pairs of morphologically identified neuron types based on their axonal-dendritic overlap within every layer and subregion of the hippocampal formation. Specifically, we adapt modern image analysis technology to determine the parcel-specific neurite lengths of every neuron type from representative morphologic reconstructions obtained from either sex. We then compute the average number of synapses per neuron pair using relevant anatomic volumes from the mouse brain atlas and ultrastructurally established interaction distances. Hence, we estimate connection probabilities and number of contacts for >1900 neuron type pairs, increasing the available quantitative assessments more than 11-fold. Connectivity statistics thus remain unknown for only a minority of potential synapses in the hippocampal formation, including those involving long-range (23%) or perisomatic (6%) connections and neuron types without morphologic tracings (7%). The described approach also yields approximate measurements of synaptic distances from the soma along the dendritic and axonal paths, which may affect signal attenuation and delay. Overall, this dataset fills a substantial gap in quantitatively describing hippocampal circuits and provides useful model specifications for biologically realistic neural network simulations, until further direct experimental data become available.SIGNIFICANCE STATEMENT The hippocampal formation is a crucial functional substrate for episodic memory and spatial representation. Characterizing the complex neuron type circuit of this brain region is thus important to understand the cellular mechanisms of learning and navigation. Here we present the first numerical estimates of connection probabilities, numbers of contacts per connected pair, and synaptic distances from the soma along the axonal and dendritic paths, for more than 1900 distinct neuron type pairs throughout the dentate gyrus, CA3, CA2, CA1, subiculum, and entorhinal cortex. This comprehensive dataset, publicly released online at www.Hippocampome.org, constitutes an unprecedented quantification of the majority of the local synaptic circuit for a prominent mammalian neural system and provides an essential foundation for data-driven, anatomically realistic neural network models.

Towards machine learning analytics for jet substructure

The past few years have seen a rapid development of machine-learning algorithms. While surely augmenting performance, these complex tools are often treated as black-boxes and may impair our understanding of the physical processes under study. The aim of this paper is to move a first step into the direction of applying expert-knowledge in particle physics to calculate the optimal decision function and test whether it is achieved by standard training, thus making the aforementioned black-box more transparent. In particular, we consider the binary classification problem of discriminating quark-initiated jets from gluon-initiated ones. We construct a new version of the widely used N-subjettiness, which features a simpler theoretical behaviour than the original one, while maintaining, if not exceeding, the discrimination power. We input these new observables to the simplest possible neural network, i.e. the one made by a single neuron, or perceptron, and we analytically study the network behaviour at leading logarithmic accuracy. We are able to determine under which circumstances the perceptron achieves optimal performance. We also compare our analytic findings to an actual implementation of a perceptron and to a more realistic neural network and find very good agreement.

Pycabnn: Efficient and Extensible Software to Construct an Anatomical Basis for a Physiologically Realistic Neural Network Model.

Physiologically detailed models of neural networks are an important tool for studying how biophysical mechanisms impact neural information processing. An important, fundamental step in constructing such a model is determining where neurons are placed and how they connect to each other, based on known anatomical properties and constraints given by experimental data. Here we present an open-source software tool, pycabnn, that is dedicated to generating an anatomical model, which serves as the basis of a full network model. In pycabnn, we implemented efficient algorithms for generating physiologically realistic cell positions and for determining connectivity based on extended geometrical structures such as axonal and dendritic morphology. We demonstrate the capabilities and performance of pycabnn by using an example, a network model of the cerebellar granular layer, which requires generating more than half a million cells and computing their mutual connectivity. We show that pycabnn is efficient enough to carry out all the required tasks on a laptop computer within reasonable runtime, although it can also run in a parallel computing environment. Written purely in Python with limited external dependencies, pycabnn is easy to use and extend, and it can be a useful tool for computational neural network studies in the future.

Spatiotemporally random and diverse grid cell spike patterns contribute to the transformation of grid cell to place cell in a neural network model.

The medial entorhinal cortex and the hippocampus are brain regions specialized in spatial information processing. While an animal navigates around an environment, grid cells in the medial entorhinal cortex spike at multiple discrete locations, forming hexagonal grid patterns, and each grid cell is spatiotemporally dynamic with a different grid size, spacing, and orientation. In contrast, place cells in the hippocampus spike when an animal is at one or more specific locations, called a “place field”. While an animal traverses through a place field, the place cell’s spike phases relative to the hippocampal theta-frequency oscillation advance in phase, known as the “spike phase precession” phenomenon and each spike encodes the specific location within the place field. Interestingly, the medial entorhinal cortical grid cells and the hippocampal place cells are only one excitatory synapse apart. However, how the spatiotemporally dynamic multi-peaked grid cell activities are transformed into hippocampal place cell activities with spike phase precession phenomenon is yet unknown. To address this question, we construct an anatomically and physiologically realistic neural network model comprised of 10,000 grid cell models, each with a spatiotemporally dynamic grid patterns and a place cell model connected by excitatory synapses. Using this neural network model, we show that grid cells’ spike activities with spatiotemporally random and diverse grid orientation, spacing, and phases as inputs to place cell are able to generate a place field with spike phase precession. These results indicate that spatiotemporally random and diverse grid cell spike activities are essential for the formation of place cell activity observed in vivo.

How single neuron properties shape chaotic dynamics and signal transmission in random neural networks.

While most models of randomly connected neural networks assume single-neuron models with simple dynamics, neurons in the brain exhibit complex intrinsic dynamics over multiple timescales. We analyze how the dynamical properties of single neurons and recurrent connections interact to shape the effective dynamics in large randomly connected networks. A novel dynamical mean-field theory for strongly connected networks of multi-dimensional rate neurons shows that the power spectrum of the network activity in the chaotic phase emerges from a nonlinear sharpening of the frequency response function of single neurons. For the case of two-dimensional rate neurons with strong adaptation, we find that the network exhibits a state of “resonant chaos”, characterized by robust, narrow-band stochastic oscillations. The coherence of stochastic oscillations is maximal at the onset of chaos and their correlation time scales with the adaptation timescale of single units. Surprisingly, the resonance frequency can be predicted from the properties of isolated neurons, even in the presence of heterogeneity in the adaptation parameters. In the presence of these internally-generated chaotic fluctuations, the transmission of weak, low-frequency signals is strongly enhanced by adaptation, whereas signal transmission is not influenced by adaptation in the non-chaotic regime. Our theoretical framework can be applied to other mechanisms at the level of single neurons, such as synaptic filtering, refractoriness or spike synchronization. These results advance our understanding of the interaction between the dynamics of single units and recurrent connectivity, which is a fundamental step toward the description of biologically realistic neural networks.

Numerical Approximation for Nonlinear Noisy Leaky Integrate-and-Fire Neuronal Model

We consider a noisy leaky integrate-and-fire (NLIF) neuron model. The resulting nonlinear time-dependent partial differential equation (PDE) is a Fokker-Planck Equation (FPE) which describes the evolution of the probability density. The finite element method (FEM) has been proposed to solve the governing PDE. In the realistic neural network, the irregular space is always determined. Thus, FEM can be used to tackle those situations whereas other numerical schemes are restricted to the problems with only a finite regular space. The stability of the proposed scheme is also discussed. A comparison with the existing Weighted Essentially Non-Oscillatory (WENO) finite difference approximation is also provided. The numerical results reveal that FEM may be a better scheme for the solution of such types of model problems. The numerical scheme also reduces computational time in comparison with time required by other schemes.

Neuromorphic Dynamical Synapses with Reconfigurable Voltage-Gated Kinetics

Although biological synapses express a large variety of receptors in neuronal membranes, the current hardware implementation of neuromorphic synapses often rely on simple models ignoring the heterogeneity of synaptic transmission. Our objective is to emulate different types of synapses with distinct properties. Conductance-based chemical and electrical synapses were implemented between silicon neurons on a fully programmable and reconfigurable, biophysically realistic neuromorphic VLSI chip. Different synaptic properties were achieved by configuring on-chip digital parameters for the conductances, reversal potentials, and voltage dependence of the channel kinetics. The measured I-V characteristics of the artificial synapses were compared with biological data. We reproduced the response properties of five different types of chemical synapses, including both excitatory ( AMPA, NMDA) and inhibitory ( GABAA, GABAC, glycine) ionotropic receptors. In addition, electrical synapses were implemented in a small network of four silicon neurons. Our work extends the repertoire of synapse types between silicon neurons, providing greater flexibility for the design and implementation of biologically realistic neural networks on neuromorphic chips. A higher synaptic heterogeneity in neuromorphic chips is relevant for the hardware implementation of energy-efficient population codes as well as for dynamic clamp applications where neural models are implemented in neuromorphic VLSI hardware.

Optimally Removing Synchronization Overhead for CNNs in Three-Dimensional Neuromorphic Architecture

Recent three-dimensional (3-D) neuromorphic processing-in-memory (PIM) architecture provides a promising hardware-based solution to speed up the processing of convolutional neural networks. However, the limited capacity of the global buffer in this architecture is unable to efficiently handle synchronization overhead. In this paper, we jointly optimize the allocation of computation and memory resources on the 3-D-stacked PIM architecture. The objective is to minimize schedule length by removing synchronization overhead. To guarantee the generation of a feasible task schedule, we theoretically obtain the upper bound to reschedule each computation task. The target problem is further formulated as a dynamic programming model to get an optimal solution. We evaluate our technique with a variety of realistic neural network applications running on deep learning frameworks Caffe and TensorFlow. The results show that the proposed technique can achieve a significant reduction in processing time and improve the utilization of processing cores compared to previous studies.

ON INTERDISCIPLINARY COMPARATIVE STUDY OF ANALOGICAL FEEDBACK/ASSESSMENT MODELS APPLIED IN BLENDED LEARNING VERSUS COMPUTER AIDED LEARNING USING ARTIFICIAL NEURAL NETWORKS

This paper provides educationalists as well as researchers in computer science and engineering with a study of an interdisciplinary challenging pedagogical issue. More specifically, the presented study resulted in a set of interesting findings that originated from adopted realistic Artificial Neural Network (ANNs) modeling, which is associated with two educational analogical feedback/assessment processes. This piece of research considers comparative analysis and evaluation study of an educational phenomenon issue for two diverse teaching/learning methodologies namely :{ Blended Learning (BL) and Computer Aided Learning (CAL)}. More precisely, the introduced issue of this work addresses the two summative and formative assessment processes applied in educational field practice. Accordingly, this issue is concerned mainly with the modeling of two practical field case studies considering the two items of educational feedback/assessment. In other words, assessment is used in many ways in education, a great deal of attention is now given to its use in helping teaching and learning, described as the two performance assessment items (summative and formative). These are defined as: assessment for learning (A f L), or formative assessment, and assessment of learning (AO L), or summative assessment. It is noticed that (AO L), is focused on which is used to summarize what students know or can do at certain times in order to report achievement and progress. Herein, two parametric factor values of ANN (gain, and learning rate factors), have been considered for the two suggested instructional methodologies. That is considered in order to compare, analyze, and evaluate dynamically two items of academic performance namely: (Academic achievement outcome & Learning convergence time) for both methodologies. Interestingly, the running realistic ANN computer modeling for different numbers of neurons -that are contributing to the learning process- results in an investigative, comprehensive, and innovative systematic analysis of individual students’ differences. Finally, after performing a perceptive evaluation comparing two case studies of obtained experimental field results, two interesting findings have been concluded. Firstly, while comparing computed sets of statistical parameters, associated with the presented instructional methodologies (CAL & BL). That resulted in the observed analogy between both sets. Secondly, in the context of Feedback/Assessment performance, regarding both (BL& CAL); either Formative or Summative feedback /assessments, have been observed to be well analogous to each other.

A New Computational Model for Astrocytes and Their Role in Biologically Realistic Neural Networks.

Recent studies in neuroscience show that astrocytes alongside neurons participate in modulating synapses. It led to the new concept of “tripartite synapse”, which means that a synapse consists of three parts: presynaptic neuron, postsynaptic neuron, and neighboring astrocytes. However, it is still unclear what role is played by the astrocytes in the tripartite synapse. Detailed biocomputational modeling may help generate testable hypotheses. In this article, we aim to study the role of astrocytes in synaptic plasticity by exploring whether tripartite synapses are capable of improving the performance of a neural network. To achieve this goal, we developed a computational model of astrocytes based on the Izhikevich simple model of neurons. Next, two neural networks were implemented. The first network was only composed of neurons and had standard bipartite synapses. The second network included both neurons and astrocytes and had tripartite synapses. We used reinforcement learning and tested the networks on categorizing random stimuli. The results show that tripartite synapses are able to improve the performance of a neural network and lead to higher accuracy in a classification task. However, the bipartite network was more robust to noise. This research provides computational evidence to begin elucidating the possible beneficial role of astrocytes in synaptic plasticity and performance of a neural network.