Large-scale Network Models Research Articles

Towards an optimised deep brain stimulation using a large-scale computational network and realistic volume conductor model

Objective. Constructing a theoretical framework to improve deep brain stimulation (DBS) based on the neuronal spatiotemporal patterns of the stimulation-affected areas constitutes a primary target. Approach. We develop a large-scale biophysical network, paired with a realistic volume conductor model, to estimate theoretically efficacious stimulation protocols. Based on previously published anatomically defined structural connectivity, a biophysical basal ganglia-thalamo-cortical neuronal network is constructed using Hodgkin–Huxley dynamics. We define a new biomarker describing the thalamic spatiotemporal activity as a ratio of spiking vs. burst firing. The per cent activation of the different pathways is adapted in the simulation to minimise the differences of the biomarker with respect to its value under healthy conditions. Main results. This neuronal network reproduces spatiotemporal patterns that emerge in Parkinson’s disease. Simulations of the fibre per cent activation for the defined biomarker propose desensitisation of pallido-thalamic synaptic efficacy, induced by high-frequency signals, as one possible crucial mechanism for DBS action. Based on this activation, we define both an optimal electrode position and stimulation protocol using pathway activation modelling. Significance. A key advantage of this research is that it combines different approaches, i.e. the spatiotemporal pattern with the electric field and axonal response modelling, to compute the optimal DBS protocol. By correlating the inherent network dynamics with the activation of white matter fibres, we obtain new insights into the DBS therapeutic action.

Transcranial direct current stimulation inhibits epileptic activity propagation in a large-scale brain network model

Transcranial direct current stimulation inhibits epileptic activity propagation in a large-scale brain network model

APapo: An asynchronous parallel optimization method for DNN models

To address the challenges related to segmentation complexity, high memory usage, extended training duration, and low equipment utilization in parallel optimization of large-scale deep neural network (DNN) models, this paper proposes an asynchronous parallel optimization method APapo. Firstly, a multi-iteration asynchronous pipeline parallel scheduling was established for model parallel computing tasks, controlling the specific scheduling process of micro-batch units to address gradient delay updating during asynchronous iteration. Secondly, combined with the given network model and hardware configuration, a dynamic programming strategy for computing resources and model tasks was designed to achieve dynamic segmentation of model computing tasks and optimal matching of computing resources. Finally, an optimization strategy for runtime scheduling of computing resources and model tasks was developed, using improved device streams to maximize the overlap between computing and communication, thus improving the utilization rate of computing resources and reducing training time. Experimental results show that the APapo method achieves fine-grained task segmentation, maximizes the utilization rate of each GPU computing resource, and on average improves the training speed of large-scale deep neural network models by 2.8 times while maintaining the training accuracy of the model compared to existing parallel optimization methods.

A Robustness-Assured White-Box Watermark in Neural Networks

Recently, stealing highly-valuable and large-scale deep neural network (DNN) models becomes pervasive. The stolen models may be re-commercialized, e.g., deployed in embedded devices, released in model markets, utilized in competitions, etc, which infringes the Intellectual Property (IP) of the original owner. Detecting IP infringement of the stolen models is quite challenging, even with the white-box access to them in the above scenarios, since they may have experienced fine-tuning, pruning, functionality-equivalent adjustment to destruct any embedded watermark. Furthermore, the adversaries may also attempt to extract the embedded watermark or forge a similar watermark to falsely claim ownership. In this paper, we propose a novel DNN watermarking solution, named <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$HufuNet$</tex-math></inline-formula> , to detect IP infringement of DNN models against the above mentioned attacks. Furthermore, HufuNet is the first one theoretically proved to guarantee robustness against fine-tuning attacks. We evaluate HufuNet rigorously on four benchmark datasets with five popular DNN models, including convolutional neural network (CNN) and recurrent neural network (RNN). The experiments and analysis demonstrate that HufuNet is highly robust against model fine-tuning/pruning, transfer learning, kernels cutoff/supplement, functionality-equivalent attacks and fraudulent ownership claims, thus highly promising to protect large-scale DNN models in the real world.

Large-scale closed and generalized networks of ribosome flow model with different site sizes

Dynamical systems play a key role in understanding the different biophysical aspects of many complex transport phenomena. The ribosome flow model with different site sizes (RFMD) is a dynamical system relevant to model various biological and physical processes such as phosphorelay, motion of vehicles along an isolated road, and more. However, many transport processes take place concurrently which give rise to networks comprising multiple tracks e.g., vehicular motion, cargo transport along the cytoskeletal highway, etc. Here, we present two large-scale network models: RFMDs network with a pool (RFMDNP), and the generalized network of RFMDs (RFMDN). The RFMDNP is a closed system that can model the simultaneous movement of particles along tracks having sites of different capacities in a resource-limited environment. We prove that the RFMDNP admits a continuum of linearly ordered steady-state points and it always phase-locks with the periodic excitations in the parameters. Furthermore, we also show that increasing any of the parameters in an RFMD leads to an increase in the output rate of this RFMD, and the output rate of other RFMDs all increase or decrease. Next, we analyze a generalized network of RFMDs that models static connections between the RFMDs and prove the important mathematical properties: (a) the network admits a unique steady-state, and (b) the problem of determining the connecting weights to maximize the steady-state network output rate is a convex optimization problem. We believe that these networks can have wide applicability to model various non-linear phenomena since their behavior is predictable and ordered.

How inhibitory and excitatory inputs gate output of the inferior olive.

The inferior olive provides the climbing fibers to Purkinje cells in the cerebellar cortex, where they elicit all-or-none complex spikes and control major forms of plasticity. Given their important role in both short-term and long-term coordination of cerebellum-dependent behaviors, it is paramount to understand the factors that determine the output of olivary neurons. Here, we use mouse models to investigate how the inhibitory and excitatory inputs to the olivary neurons interact with each other, generating spiking patterns of olivary neurons that align with their intrinsic oscillations. Using dual color optogenetic stimulation and whole-cell recordings, we demonstrate how intervals between the inhibitory input from the cerebellar nuclei and excitatory input from the mesodiencephalic junction affect phase and gain of the olivary output at both the sub- and suprathreshold level. When the excitatory input is activated shortly (~50 ms) after the inhibitory input, the phase of the intrinsic oscillations becomes remarkably unstable and the excitatory input can hardly generate any olivary spike. Instead, when the excitatory input is activated one cycle (~150 ms) after the inhibitory input, the excitatory input can optimally drive olivary spiking, riding on top of the first cycle of the subthreshold oscillations that have been powerfully reset by the preceding inhibitory input. Simulations of a large-scale network model of the inferior olive highlight to what extent the synaptic interactions penetrate in the neuropil, generating quasi-oscillatory spiking patterns in large parts of the olivary subnuclei, the size of which also depends on the relative timing of the inhibitory and excitatory inputs.

Interactive visual analytics of parallel training strategies for DNN models

Understanding and optimizing the parallel training strategies in the training of large-scale Deep Neural Network (DNN) models is crucial to enhance training efficiency. Existing works tried to demonstrate the layer-level information of the computational graph to support parallel training strategy selection. Whereas, the overall parallel execution logic is rarely considered by previous methods. In this paper, we proposed a novel visual analytics approach for parallel training strategies, demonstrating the execution logic of the distributed computing from up to bottom via explaining communication operators. Specifically, a computation-communication bipartite construction algorithm is designed for the computational graph visualization. Furthermore, a system is developed to help users easily access the proposed approach and explore the parallel training strategies interactively. With empirical evaluation through a quantitative user study and a qualitative expert interview, the practicality and superiority of the proposed approach is verified.

Astrocytic clearance and fragmentation of toxic proteins in Alzheimer’s disease on large-scale brain networks

The human brain is the most complicated biological structure on the planet. A major challenge of brain network modelling lies in its multi-scale spatio-temporal nature, covering scales from synapses to the whole brain. The coupled multiphysics and biochemical activities which spread through such a complex system shape brain capacity inside a structure-function relationship that requires a particular mathematical framework. Next-generation coupled-based mathematical modelling approaches to brain networks and the analysis of data-driven dynamical systems are needed to advance state-of-the-art therapeutic strategies for treating neurodegenerative diseases (NDDs) that affect millions of people worldwide, such as Alzheimer’s disease (AD) and Parkinson’s disease (PD). Importantly, AD is marked by the presence of amyloid-beta (Aβ) plaques and tau (τ) proteins. Some disease-specific misfolded proteins can interact with healthy proteins to form long chains and aggregates of different sizes that have different transport properties and toxicity. An improved large-scale brain network model is proposed here to understand the pathogenesis of AD, especially the role of astrocytes in the presence of misfolded proteins (Aβ and τ). The idea involves astrocytic clearance, which assists in eliminating toxic Aβ via fragmentation. We use the general Smoluchowski theory of nucleation, aggregation, and fragmentation to predict the development and propagation of aggregates of misfolded proteins in the brain. It has been shown that the developed model leads to different size distributions and propagation along the network. We predicted that astrocytic clearance varies with the aggregate size, which is key to slowing down AD progression. The clearance and fragmentation of toxic proteins span several spatial and temporal scales, and this research will potentially yield new insight into the associated processes and brain networks in health and disease. Detailed multi-scale brain modelling provides a promising approach for consolidating, organizing, and bridging the data sets of data-driven brain network models.

Improving Neural Models for Natural Language Processing in Russian with Synonyms

Large-scale neural network models, including models for natural language processing, require large datasets that could be unavailable for low-resource languages or for special domains. We consider a way to approach the problem of poor variability and small size of available data for training NLP models based on augmenting the data with synonyms. We design a novel augmentation scheme that includes replacing words with synonyms, apply it to the Russian language and report improved results for the sentiment analysis task.

Improving hydrodynamic modeling of river networks by incorporating data assimilation using a particle filter

Numerical modeling is a well-recognized method for studying the hydrodynamic processes in river networks. Multi-source measurements also offer abundant information on the patterns and mechanisms within the processes. Therefore, improving hydrodynamic modeling of river networks through the use of data assimilation techniques has become a hot research topic in recent years. The particle filter (PF) is a commonly used data assimilation method and has been proven to be applicable to various nonlinear and non-Gaussian models. In the current study, an improved numerical hydrodynamic model for large-scale river networks is established by incorporating the advanced PF algorithm. Furthermore, the PF method based on the Gaussian likelihood function (GLF) and the method based on the Cauchy likelihood function (CLF) are compared for a complex river network scenario. The feasibility of the PF-based methods was evaluated through application to the Yangtze-Dongting River-lake Network (YDRN) by assimilating water stage data collected at six hydrometric stations during the entire hydrodynamic process in 2003. Additionally, the parameters used in the likelihood function, which affect the assimilation performance, also were explored in the current study. The study results found that the accuracy of the model-derived water stage data was improved when the PF-based methods are utilized, with improvement not only at the data assimilation (calibration) sites but also at three hydrometric stations not used in the data assimilation (i.e., verification sites). The highest average Nash-Sutcliffe Efficiency result for the six assimilation sites were 0.98 while the lowest summed root-mean-square-error result was 1.801 m. The comparison results also indicated that the CLF-based PF outperformed the GLF-based PF when high-accuracy observed data are available. Specifically, the CLF can effectively resolve the filtering failure problem and the dispersion problem of PFs, and further improve the accuracy of the filtering results for a river network scenario. In summary, the CLF-based PF method along with high-accuracy observation data shows promise to provide reliable reference and technical support for hydrodynamic modeling of large-scale river networks.

Methodology for integrating flexibility into realistic large-scale distribution network planning using Tabu search

A successful energy transition will be firmly based on the effective integration of distributed energy resources and on the integration of new flexibility providers into the energy system. To make this possible, a deep transformation in the design, operation and planning of power distribution systems is required. Currently, a lack of comprehensive planning tools capable of supporting operators in their investment plan options exists. As a result, reinforcements of conventional grid assets are common solutions put in place. This paper proposes a methodology to obtain cost-optimal distribution network expansion plans, by modelling a single-stage distribution network planning tool, using conventional assets as well as flexibility contracting from demand response. A Tabu Search metaheuristic has been implemented in order to solve the optimization problem. A case study based on a realistic large-scale city network model is presented, for a planning horizon of ten years with significant load growth due to electromobility penetration. Results show that, in the case study analysed, the use of load flexibility in combination with conventional reinforcements can reduce the total expansion network cost by about 7.5 %. Furthermore, a sensitivity analysis on the cost of flexibility contracting is undertaken. Remarkably, the methodology presented generalises to further alternative solutions by providing a straightforward financial benchmark between the latter and conventional grid expansion.

Short-term traffic flow prediction model based on a shared weight gate recurrent unit neural network

Accurate traffic flow prediction is critical for enhancing traffic network operational efficiency. With the continuous expansion of traffic networks, providing reliable and efficient multi-step traffic flow prediction for large-scale traffic networks with a large number of sensors deployed has become a challenging issue. In this paper, we propose a multi-step many-to-many traffic prediction model for large-scale traffic networks, called spatio-temporal Shared GRU (STSGRU), which receive inputs from multiple sensors and provides predictions for all sensors simultaneously. First, we model the weekly pattern of traffic flow, using periodicity to explore long-term temporal features and provide smooth traffic flow to reduce the impact of data volatility. Second, different from existing models, we propose a shared weight mechanism to achieve many-to-many prediction without mapping traffic networks to images or graph structures. The proposed model strikes a delicate balance between complexity and accuracy. We validate the effectiveness of the proposed method on the Caltrans Performance Measurement System (PeMS) dataset. The results show that our model achieves similar prediction performance with advanced graph neural networks and has higher flexibility.

Modeling the three-dimensional connectivity of in vitro cortical ensembles coupled to Micro-Electrode Arrays.

Nowadays, in vitro three-dimensional (3D) neuronal networks are becoming a consolidated experimental model to overcome most of the intrinsic limitations of bi-dimensional (2D) assemblies. In the 3D environment, experimental evidence revealed a wider repertoire of activity patterns, characterized by a modulation of the bursting features, than the one observed in 2D cultures. However, it is not totally clear and understood what pushes the neuronal networks towards different dynamical regimes. One possible explanation could be the underlying connectivity, which could involve a larger number of neurons in a 3D rather than a 2D space and could organize following well-defined topological schemes. Driven by experimental findings, achieved by recording 3D cortical networks organized in multi-layered structures coupled to Micro-Electrode Arrays (MEAs), in the present work we developed a large-scale computational network model made up of leaky integrate-and-fire (LIF) neurons to investigate possible structural configurations able to sustain the emerging patterns of electrophysiological activity. In particular, we investigated the role of the number of layers defining a 3D assembly and the spatial distribution of the connections within and among the layers. These configurations give rise to different patterns of activity that could be compared to the ones emerging from real in vitro 3D neuronal populations. Our results suggest that the introduction of three-dimensionality induced a global reduction in both firing and bursting rates with respect to 2D models. In addition, we found that there is a minimum number of layers necessary to obtain a change in the dynamics of the network. However, the effects produced by a 3D organization of the cells is somewhat mitigated if a scale-free connectivity is implemented in either one or all the layers of the network. Finally, the best matching of the experimental data is achieved supposing a 3D connectivity organized in structured bundles of links located in different areas of the 2D network.

Intrinsic neural timescales mediate the cognitive bias of self – temporal integration as key mechanism

Our perceptions and decisions are not always objectively correct as they are featured by a bias related to our self. What are the behavioral, neural, and computational mechanisms of such cognitive bias? Addressing this yet unresolved question, we here investigate whether the cognitive bias is related to temporal integration and segregation as mediated by the brain's Intrinsic neural timescales (INT). Using Signal Detection Theory (SDT), we operationalize the cognitive bias by the Criterion C as distinguished from the sensitivity index d’. This was probed in a self-task based on morphed self- and other faces. Behavioral data demonstrate clear cognitive bias, i.e., Criterion C. That was related to the EEG-based INT as measured by the autocorrelation window (ACW) in especially the transmodal regions dorsolateral prefrontal cortex (dlPFC) and default-mode network (DMN) as distinct from unimodal visual cortex. Finally, simulation of the same paradigm in a large-scale network model shows high degrees of temporal integration of temporally distinct inputs in CMS/DMN and dlPFC while temporal segregation predominates in visual cortex. Together, we demonstrate a key role of INT-based temporal integration in CMS/DMN and dlPFC including its relation to the brain's uni-transmodal topographical organization in mediating the cognitive bias of our self.

Simulations of cortical networks using spatially extended conductance-based neuronal models.

The Hodgkin-Huxley model of action potential generation and propagation, published in the Journal of Physiology in 1952, initiated the field of biophysically detailed computational modelling in neuroscience, which has expanded to encompass a variety of species and components of the nervous system. Here we review the developments in this area with a focus on efforts in the community towards modelling the mammalian neocortex using spatially extended conductance-based neuronal models. The Hodgkin-Huxley formalism and related foundational contributions, such as Rall's cable theory, remain widely used in these efforts to the current day. We argue that at present the field is undergoing a qualitative change due to new very rich datasets describing the composition, connectivity and functional activity of cortical circuits, which are being integrated systematically into large-scale network models. This trend, combined with the accelerating development of convenient software tools supporting such complex modelling projects, is giving rise to highly detailed models of the cortex that are extensively constrained by the data, enabling computational investigation of a multitude of questions about cortical structure and function.

SunSpark: Fusion of time-domain and frequency-domain transformer for accurate identification of DC arc faults

&lt;abstract&gt;&lt;p&gt;Photovoltaic (PV) power generation is pivotal to the energy strategies of various nations, yet it is plagued by significant security challenges. This paper proposes a large-scale neural network model that integrates time-domain and frequency-domain techniques for the detection of arc faults in PV systems. The algorithm leverages sequence decomposition to extract trend information from current signals, and then applies the Fourier transform to convert various encoded data into the frequency domain. Due to the sparsity of frequency-domain information, the computational cost of extracting and processing information in the frequency domain is minimal, resulting in high efficiency. The selectively extracted information is then input into a separate lightweight classifier for classification and recognition. The proposed intelligent framework not only effectively filters out high-frequency noise signals, but also demonstrates strong robustness against various disturbances, yielding exceptional recognition performance with an accuracy rate consistently surpassing 97$ \% $. Code and data are available at this repository: &lt;ext-link ext-link-type="uri" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://github.com/yixizhuimeng?tab = projects"&gt;https://github.com/yixizhuimeng?tab = projects&lt;/ext-link&gt;.&lt;/p&gt;&lt;/abstract&gt;

Adaptive Distributed Parallel Training Method for a Deep Learning Model Based on Dynamic Critical Paths of DAG

AI provides a new method for massive simulated data calculations in molecular dynamics, materials, and other scientific computing fields. However, the complex structures and large-scale parameters of neural network models make them difficult to develop and train. The automatic parallel technology based on graph algorithms is one of the most promising methods to solve this problem, despite the low efficiency in the design, implementation, and execution of distributed parallel policies for large-scale neural network models. In this paper, we propose an adaptive distributed parallel training method based on the dynamic generation of critical DAG (directed acyclic graph) paths, called FD-DPS, to solve this efficiency problem. Firstly, the proposed model splits operators with the dimension of the tensor, which can expand the space available for model parallelism. Secondly, a dynamic critical path generation method is employed to determine node priority changes in the DAG of the neural network models. Finally, the model implements the optimal scheduling of critical paths based on the priority of the nodes, thereby improving the performance of parallel strategies. Our experiments show that FD-DPS can achieve 12.76% and 11.78% faster training on PnasNet_mobile and ResNet_200 models, respectively, compared with the MP-DPS and Fast methods.

Decomposition-Based Bayesian Network Structure Learning Algorithm for Abnormity Diagnosis Model for Coal Mill Process

In the structure learning of the large-scale Bayesian network (BN) model for the coal mill process, taking the view of the problem that the decomposition-based method cannot guarantee the sufficient learning of abnormal state node neighborhood in the diagnosis model, this paper proposes a new BN structure learning method based on decomposition. Firstly, a sketch is constructed based on an improved Markov blanket discovery algorithm and edge thickening and thinning. Second, the node centrality of k-path is used to search the important nodes, and the subgraph decomposition is realized by extracting these important nodes and their neighborhoods from the sketch. Then, through the targeted design of subgraph de-duplication, subgraph learning, and subgraph reorganization methods, the learning of large-scale BN is realized. This method is applied to public data sets, and its advantages and disadvantages are analyzed by comparing them with other methods. The advantage of the BN structure learning method of the abnormal condition diagnosis model is further verified by applying the method to the coal mill process, which is consistent with the original design intention.

A Block-Based and Highly Parallel CNN Accelerator for Seed Sorting

Seed sorting is critical for the breeding industry to improve the agricultural yield. The seed sorting methods based on convolutional neural networks (CNNs) have achieved excellent recognition accuracy on large-scale pretrained network models. However, CNN inference is a computationally intensive process that often requires hardware acceleration to operate in real time. For embedded devices, the high-power consumption of graphics processing units (GPUs) is generally prohibitive, and the field programmable gate array (FPGA) becomes a solution to perform high-speed inference by providing a customized accelerator for a particular user. To date, the recognition speeds of the FPGA-based universal accelerators for high-throughput seed sorting tasks are slow, which cannot guarantee real-time seed sorting. Therefore, a block-based and highly parallel MobileNetV2 accelerator is proposed in this paper. First, a hardware-friendly quantization method that uses only fixed-point operation is designed to reduce resource consumption. Then, the block convolution strategy is proposed to avoid latency and energy consumption increase caused by large-scale intermediate result off-chip data transfers. Finally, two scalable computing engines are explicitly designed for depth-wise convolution (DWC) and point-wise convolution (PWC) to develop the high parallelism of block convolution computation. Moreover, an efficient memory system with a double buffering mechanism and new data reordering mode is designed to address the imbalance between memory access and parallel computing. Our proposed FPGA-based MobileNetV2 accelerator for real-time seed sorting is implemented and evaluated on the platform of Xilinx XC7020. Experimental results demonstrate that our implementation can achieve about 29.4 frames per second (FPS) and 10.86 Giga operations per second (GOPS), and 0.92× to 5.70 × DSP-efficiency compared with previous FPGA-based accelerators.

In pursuit of the epileptogenic zone in focal epilepsy:a dynamical network biomarker approach

The success of resection surgery for drug-resistant epilepsy patients hinges on the correct identification of the epileptogenic zone (EZ) consisting of the subnetwork of brain regions that underlies seizure genesis in focal epilepsy. The dynamical network biomarker (DNB) method is a dynamical systems-based network analysis approach for identifying subnetworks that are the first to exhibit the transition as a complex system undergoes a bifurcation. The approach was devised and validated in the context of complex disease onset where the dynamics is known to be nonlinear and high-dimensional. We here adapt and implement the DNB approach for the identification of the EZ from the analysis of Stereoelectroencephalography (SEEG) data. The method is first successfully tested on simulated data generated with a large-scale brain network model of epilepsy using The Virtual Brain neuroinformatic platform and then applied to clinical SEEG data from focal epilepsy patients. The results are compared with those obtained by expert clinicians that designate the EZ using the Epileptogenicity Index (EI) method. High average precision values are obtained and posit the presented approach as a promising candidate tool for the pursuit of EZ in focal epilepsy.