Dynamics Of Neural Systems Research Articles

Memristive leaky integrate-and-fire neuron and learnable straight-through estimator in spiking neural networks.

Compared to artificial neural networks (ANNs), spiking neural networks (SNNs) present a more biologically plausible model of neural system dynamics. They rely on sparse binary spikes to communicate information and operate in an asynchronous, event-driven manner. Despite the high heterogeneity of the neural system at the neuronal level, most current SNNs employ the widely used leaky integrate-and-fire (LIF) neuron model, which assumes uniform membrane-related parameters throughout the entire network. This approach hampers the expressiveness of spiking neurons and restricts the diversity of neural dynamics. In this paper, we propose replacing the resistor in the LIF model with a discrete memristor to obtain the heterogeneous memristive LIF (MLIF) model. The memristance of the discrete memristor is determined by the voltage and flux at its terminals, leading to dynamic changes in the membrane time parameter of the MLIF model. SNNs composed of MLIF neurons can not only learn synaptic weights but also adaptively change membrane time parameters according to the membrane potential of the neuron, enhancing the learning ability and expression of SNNs. Furthermore, since the proper threshold of spiking neurons can improve the information capacity of SNNs, a learnable straight-through estimator (LSTE) is proposed. The LSTE, based on the straight-through estimator (STE) surrogate function, features a learnable threshold that facilitates the backward propagation of gradients through neurons firing spikes. Extensive experiments on several popular static and neuromorphic benchmark datasets demonstrate the effectiveness of the proposed MLIF and LSTE, especially on the DVS-CIFAR10 dataset, where we achieved the top-1 accuracy of 84.40 .

Dynamical stability and chaos in artificial neural network trajectories along training

The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network’s prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space–a time series of networks–and thus the training algorithm (e.g., gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.

Dynamics and stability of neural systems with indirect interactions involved energy levels

Dynamical modelling for neural systems with direct and indirect connections is to understand how these connections contribute to neural dynamics. Despite recent findings suggesting the existence of indirect connections in neural systems, their dynamical characteristics remain poorly understood. In this paper, we propose a simplified circuit model with indirect interactions inspired by the indirect connection between two neurons, from an energy perspective. Through bifurcation and dynamics analysis, we find that the presented model has a striking resemblance with the classical Hodgkin-Huxley neuronal model. Moreover, stability in a neural network coupled with energy is demonstrated by combining stability analysis and numerical simulation. Our analysis sheds light on the excitability dynamics and multi-stability that can emerge in biophysical systems with nonlinear interactions inspired by the neural systems and highlights the role of energy in propagating electrical activities.

An investigation into the abnormal dynamic connection mechanism of generalized anxiety disorders based on non-homogeneous Markov models

BackgroundThe dynamic and hierarchical nature of the functional brain network. The neural dynamical systems tend to converge to multiple attractors (stable fixed points or dynamical states) in long run. Little is known about how the changes in this brain dynamic “long-term” behavior of the connectivity flow of brain network in generalized anxiety disorder (GAD). MethodsThis study recruited 92 patients with GAD and 77 healthy controls (HC). We applied a reachable probability approach combining a Non-homogeneous Markov model with transition probability to quantify all possible connectivity flows and the hierarchical structure of brain functional systems at the dynamic level and the stationary probability vector (10-step transition probabilities) to describe the steady state of the system in the long run. A random forest algorithm was conducted to predict the severity of anxiety. ResultsThe dynamic functional patterns in distributed brain networks had larger possibility to converge in bilateral thalamus, posterior cingulate cortex (PCC), right superior occipital gyrus (SOG) and smaller possibility to converge in bilateral superior temporal gyrus (STG) and right parahippocampal gyrus (PHG) in patients with GAD compared to HC. The abnormal transition probability pattern could predict anxiety severity in patients with GAD. LimitationsSmall samples and subjects taking medications may have influenced our results. Future studies are expected to rule out the potential confounding effects. ConclusionOur results have revealed abnormal dynamic neural communication and integration in emotion regulation in patients with GAD, which give new insights to understand the dynamics of brain function of patients with GAD.

Dynamics of neural system under the influence of a magnetic flux

Dynamics of neural system under the influence of a magnetic flux

Improved zeroing neural models based on two novel activation functions with exponential behavior

A family of zeroing neural networks based on new nonlinear activation functions is proposed for solving various time-varying linear matrix equations (TVLME). The proposed neural network dynamical systems, symbolized as Li-VPZNN1 and Li-VPZNN2, include an exponential parameter in nonlinear activation function (AF) that leads to faster convergence to the theoretical result compared to previous categories of nonlinearly activated neural networks. Theoretical analysis as well as numerical tests in MATLAB's environment confirm the efficiency and accelerated convergence property of the novel dynamics.

A Novel Dynamic Neural System for Nonconvex Portfolio Optimization With Cardinality Restrictions

A Novel Dynamic Neural System for Nonconvex Portfolio Optimization With Cardinality Restrictions

An efficient vibration suppression technology of piezoelectric cantilever beam based on the NARX neural network

Low-frequency vibration is the core problem that hinders high-precision equipment’s positioning accuracy and control accuracy. However, the response of a nonlinear electrical-mechanical coupling system is nonlinear and quite complicated under the ambient random vibrating environment. This paper presents a vibration suppression method through NARX (Nonlinear autoregressive with external input) neural network and its controller. The experiment platform is designed, and its dynamic model is obtained through system identification. The Neural network direct dynamic inverse control system is established, and the controller is designed. Both the simulation and experimental results show that the NARX identification and the controller can effectively achieve vibration suppression, and the experiment’s highest vibration rejection ratio is 90.8%. The vibration suppression technology can be extensively applied to low-frequency vibration systems, such as aerospace equipment and high-precision equipment.

Natralization without associationist reduction: a brief rebuttal to Yoshimi

AbstractYoshimi has attempted to defuse my argument concerning the identification of network abstraction with empiricist abstraction - thus entailing psychologism - by claiming that the argument does not generalize from the example of simple feed-forward networks. I show that such details of networks are logically irrelevant to the nature of the abstractive process they employ. This is ultimately because deep artificial neural networks (ANNs) and dynamical systems theory applied to the mind (DST) are both associationisms - that is, empiricist theories that derive the principles of thought from the causal history of the organism/system. On this basis, I put forward a new aspect of the old argument by noting that ANNs & DST are the causal bases of the phenomena of passive synthesis, whereas the language of thought hypothesis (LOT) and the symbolic computational theory of mind (CTM) are the causal bases of the phenomena of active synthesis. If the phenomena of active synthesis are not distinct in kind from and are thus reducible to those of passive synthesis, psychologism results. Yoshimi’s program, insofar as it denies this fundamental phenomenological distinction, is revealed to be the true anti-pluralist program, by essentially denying the causal efficacy of the mechanistic foundations of active synthesis by referring phenomenology exclusively to associationism for its causal foundation.

On robustness of neural ODEs image classifiers

Neural Ordinary Differential Equations (Neural ODEs), as a family of novel deep models, delicately link conventional neural networks and dynamical systems, which bridges the gap between theory and practice. However, they have not made substantial progress on activation functions, and ReLU is always utilized by default. Moreover, the dynamical behavior existing in them becomes more unclear and complicated as training progresses. Fortunately, existing studies have shown that activation functions are essential for Neural ODEs in governing intrinsic dynamics. Motivated by a family of weight functions used to enhance the stability of dynamical systems, we introduce a new activation function named half-Swish to match Neural ODEs. Besides, we explore the effect of evolution time and batch size on Neural ODEs, respectively. Experiments show that our model consistently outperforms Neural ODEs with basic activation functions on robustness both against stochastic noise images and adversarial examples across Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets, which strongly validates the applicability of half-Swish and suggests that half-Swish function plays a positive role in regularizing the dynamic behavior to enhance stability. Meanwhile, our work theoretically provides a prospective framework to choose appropriate activation functions to match neural differential equations.

Pluralist neurophenomenology: a reply to Lopes

Abstract Lopes (2021) has argued against my use of neural networks and dynamical systems theory in neurophenomenology. Responding to his argument provides an opportunity to articulate a pluralist approach to neurophenomenology, according to which multiple theoretical frameworks—symbolic, dynamical systems, connectionist, etc.—can be used to study consciousness and its relationship to neural activity. Each type of analysis is best suited to specific phenomena, but they are mutually compatible and can inform and constrain one another in non-trivial ways. I use historical and conceptual arguments to elaborate on this type of pluralism as it applies to cognitive science, phenomenology, and neurophenomenology.

Scale-free behavioral dynamics directly linked with scale-free cortical dynamics.

Naturally occurring body movements and collective neural activity both exhibit complex dynamics, often with scale-free, fractal spatiotemporal structure. Scale-free dynamics of both brain and behavior are important because each is associated with functional benefits to the organism. Despite their similarities, scale-free brain activity and scale-free behavior have been studied separately, without a unified explanation. Here, we show that scale-free dynamics of mouse behavior and neurons in the visual cortex are strongly related. Surprisingly, the scale-free neural activity is limited to specific subsets of neurons, and these scale-free subsets exhibit stochastic winner-take-all competition with other neural subsets. This observation is inconsistent with prevailing theories of scale-free dynamics in neural systems, which stem from the criticality hypothesis. We develop a computational model which incorporates known cell-type-specific circuit structure, explaining our findings with a new type of critical dynamics. Our results establish neural underpinnings of scale-free behavior and clear behavioral relevance of scale-free neural activity.

Domain-aware Control-oriented Neural Models for Autonomous Underwater Vehicles

Conventional physics-based modeling is a time-consuming bottleneck in control design for complex nonlinear systems like autonomous underwater vehicles (AUVs). In contrast, purely data-driven models require a large number of observations and lack operational guarantees for safety-critical systems. Data-driven models leveraging available partially characterized dynamics have potential to provide reliable systems models in a typical data-limited scenario for high value complex systems, thereby avoiding months of expensive expert modeling time. In this work we explore this middle-ground between expert-modeled and pure data-driven modeling. We present control-oriented parametric models with varying levels of domain-awareness that exploit known system structure and prior physics knowledge to create constrained deep neural dynamical system models. We employ universal differential equations to construct data-driven blackbox and graybox representations of the AUV dynamics. In addition, we explore a hybrid formulation that explicitly models the residual error related to imperfect graybox models. We compare the prediction performance of the learned models for different distributions of initial conditions and control inputs to assess their suitability for control.

A Note on the Connection Between Trek Rules and Separable Nonlinear Least Squares in Linear Structural Equation Models

We show that separable nonlinear least squares (SNLLS) estimation is applicable to all linear structural equation models (SEMs) that can be specified in RAM notation. SNLLS is an estimation technique that has successfully been applied to a wide range of models, for example neural networks and dynamic systems, often leading to improvements in convergence and computation time. It is applicable to models of a special form, where a subset of parameters enters the objective linearly. Recently, Kreiberg et al. (Struct Equ Model Multidiscip J 28(5):725–739, 2021. https://doi.org/10.1080/10705511.2020.1835484) have shown that this is also the case for factor analysis models. We generalize this result to all linear SEMs. To that end, we show that undirected effects (variances and covariances) and mean parameters enter the objective linearly, and therefore, in the least squares estimation of structural equation models, only the directed effects have to be obtained iteratively. For model classes without unknown directed effects, SNLLS can be used to analytically compute least squares estimates. To provide deeper insight into the nature of this result, we employ trek rules that link graphical representations of structural equation models to their covariance parametrization. We further give an efficient expression for the gradient, which is crucial to make a fast implementation possible. Results from our simulation indicate that SNLLS leads to improved convergence rates and a reduced number of iterations.

Multilevel and multifaceted brain response features in spiking, ERP and ERD: experimental observation and simultaneous generation in a neuronal network model with excitation–inhibition balance

Brain as a dynamic system responds to stimulations with specific patterns affected by its inherent ongoing dynamics. The patterns are manifested across different levels of organization—from spiking activity of neurons to collective oscillations in local field potential (LFP) and electroencephalogram (EEG). The multilevel and multifaceted response activities show patterns seemingly distinct and non-comparable from each other, but they should be coherently related because they are generated from the same underlying neural dynamic system. A coherent understanding of the interrelationships between different levels/aspects of activity features is important for understanding the complex brain functions. Here, based on analysis of data from human EEG, monkey LFP and neuronal spiking, we demonstrated that the brain response activities from different levels of neural system are highly coherent: the external stimulus simultaneously generated event-related potentials, event-related desynchronization, and variation in neuronal spiking activities that precisely match with each other in the temporal unfolding. Based on a biologically plausible but generic network of conductance-based integrate-and-fire excitatory and inhibitory neurons with dense connections, we showed that the multiple key features can be simultaneously produced at critical dynamical regimes supported by excitation–inhibition (E–I) balance. The elucidation of the inherent coherency of various neural response activities and demonstration of a simple dynamical neural circuit system having the ability to simultaneously produce multiple features suggest the plausibility of understanding high-level brain function and cognition from elementary and generic neuronal dynamics.

Synchronization clusters located on epileptic onset zones in neocortical epilepsy

BackgroundBrain function is thought to rely on complex interactions of dynamic neural systems, which depend on the integrity of structural and functional networks. Focal epilepsy is considered to result from excessive focal synchronization in the network. Synchronization analysis of multichannel electrocorticography (ECoG) contributes to the understanding of and orientation of epilepsy. The aim of this study was to explore the synchronization in multichannel ECoG recordings from patients with neocortical epilepsy and characterize neural activity inside and outside the onset zone.MethodsFour patients with neocortical epilepsy, who became seizure-free for more than 1 year after surgery guided by ECoG monitoring, were included in this study. ECoG data recorded during pre-surgical evaluation were analyzed. Synchronizations in phase and amplitude of different frequency bands between ECoG channels was analyzed using MATLAB. We generated 100 surrogate data from the original ECoG data using Amplitude Adjusted Fourier Transform to calculate the enhanced synchronization. The relationship between synchronization characteristics and seizure onset zone was analyzed.ResultsWe found synchronization clusters in the 14–30 Hz and 30–80 Hz bands around the onset areas during both interictal and the beginning of ictal periods in all four patients.ConclusionsThe enhanced-synchronization clusters play a central role in epilepsy, and may activate the onset areas and contribute to the spreading of epileptiform activity.

Self-Consistent Learning of Neural Dynamical Systems From Noisy Time Series

We introduce a new method which, for a single noisy time series, provides unsupervised filtering, state space reconstruction, efficient learning of the unknown governing multivariate dynamical system, and deterministic forecasting. We construct both the underlying trajectories and a latent dynamical system using deep neural networks. Under the assumption that the trajectories follow the latent dynamical system, we determine the unknowns of the dynamical system, and filter out stochastic outliers in the measurements. In this sense the method is self-consistent. The embedding dimension is determined iteratively during the training by using the false-nearest-neighbors Algorithm and it is implemented as an attention map to the state vector. This allows for a state space reconstruction without a priori information on the signal. By exploiting the differentiability of the neural solution trajectory, we can define the neural dynamical system locally at each time, mitigating the need for forward and backwards passing through numerical solvers of the canonical adjoint method. On a chaotic time series masked by additive Gaussian noise, we demonstrate that the denoising ability and the predictive power of the proposed method are mainly due to the self-consistency, insensitive to methods used for the state space reconstruction.

The Temporal Dynamics of Emotion Regulation in Subjects With Major Depression and Healthy Control Subjects

BackgroundEmotion regulation (ER) processes help support well-being, but ineffective ER is implicated in several psychiatric disorders. Engaging ER flexibly by going online and offline as needs and capacities shift may be more effective than engaging ER rigidly across time. Here, we sought to observe the neural temporal dynamics of an ER process, reappraisal, during regulation of responses to negative memories in healthy control subjects (n = 33) and subjects with major depressive disorder (n = 36). MethodsTo track the temporal dynamics of reappraisal neural systems, we used a functional magnetic resonance imaging neural decoding approach. In task 1, subjects explicitly engaged reappraisal on instruction in response to aversive images, and we used this task to develop the decoder for detecting reappraisal. In task 2, subjects experienced negative autobiographical memories from a distant (third person, ER condition) or immersed (first person, control condition) perspective. ResultsThe neural decoder, trained to detect reappraisal in task 1, predicted greater reappraisal occurring during the task 2 distance versus immerse trials and was engaged more intensely during memories that were rated as being more negative. Across time, decoder output manifested a temporal dynamic of early engagement followed by disengagement. These results were replicated in an independent subject dataset (n = 59). Relative to healthy control subjects, subjects with major depressive disorder had a comparable initial increase in decoder engagement at the beginning of the trial but an attenuated decrease at the end. ConclusionsSubjects with major depressive disorder evidenced a more rigid neural dynamic of reappraisal compared with healthy control subjects. Rigid ER may indicate diminished ability to flexibly and effectively regulate emotion.

Emergence of sensory attenuation based upon the free-energy principle

The brain attenuates its responses to self-produced exteroceptions (e.g., we cannot tickle ourselves). Is this phenomenon, known as sensory attenuation, enabled innately, or acquired through learning? Here, our simulation study using a multimodal hierarchical recurrent neural network model, based on variational free-energy minimization, shows that a mechanism for sensory attenuation can develop through learning of two distinct types of sensorimotor experience, involving self-produced or externally produced exteroceptions. For each sensorimotor context, a particular free-energy state emerged through interaction between top-down prediction with precision and bottom-up sensory prediction error from each sensory area. The executive area in the network served as an information hub. Consequently, shifts between the two sensorimotor contexts triggered transitions from one free-energy state to another in the network via executive control, which caused shifts between attenuating and amplifying prediction-error-induced responses in the sensory areas. This study situates emergence of sensory attenuation (or self-other distinction) in development of distinct free-energy states in the dynamic hierarchical neural system.

Neural Piecewise-Constant Delay Differential Equations

Continuous-depth neural networks, such as the Neural Ordinary Differential Equations (ODEs), have aroused a great deal of interest from the communities of machine learning and data science in recent years, which bridge the connection between deep neural networks and dynamical systems. In this article, we introduce a new sort of continuous-depth neural network, called the Neural Piecewise-Constant Delay Differential Equations (PCDDEs). Here, unlike the recently proposed framework of the Neural Delay Differential Equations (DDEs), we transform the single delay into the piecewise-constant delay(s). The Neural PCDDEs with such a transformation, on one hand, inherit the strength of universal approximating capability in Neural DDEs. On the other hand, the Neural PCDDEs, leveraging the contributions of the information from the multiple previous time steps, further promote the modeling capability without augmenting the network dimension. With such a promotion, we show that the Neural PCDDEs do outperform the several existing continuous-depth neural frameworks on the one-dimensional piecewise-constant delay population dynamics and real-world datasets, including MNIST, CIFAR10, and SVHN.