Neural Information Processing Research Articles

Ising model selection using ℓ 1-regularized linear regression: a statistical mechanics analysis* *This article is an updated version of: Meng X, Obuchi T and Kabashima Y 2021 Ising model selection using ℓ 1-regularized linear regression: a statistical mechanics analysis Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin, P S Liang and J Wortman

We theoretically analyze the typical learning performance of ℓ 1-regularized linear regression (ℓ 1-LinR) for Ising model selection using the replica method from statistical mechanics. For typical random regular graphs in the paramagnetic phase, an accurate estimate of the typical sample complexity of ℓ 1-LinR is obtained. Remarkably, despite the model misspecification, ℓ 1-LinR is model selection consistent with the same order of sample complexity as ℓ 1-regularized logistic regression (ℓ 1-LogR), i.e. , where N is the number of variables of the Ising model. Moreover, we provide an efficient method to accurately predict the non-asymptotic behavior of ℓ 1-LinR for moderate M, N, such as precision and recall. Simulations show a fairly good agreement between theoretical predictions and experimental results, even for graphs with many loops, which supports our findings. Although this paper mainly focuses on ℓ 1-LinR, our method is readily applicable for precisely characterizing the typical learning performances of a wide class of ℓ 1-regularized M-estimators including ℓ 1-LogR and interaction screening.

Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime* *This article is an updated version of: Cui H, Loureiro B, Krzakala F and Zdeborová L 2021 Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin,

In this manuscript we consider kernel ridge regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the features co-variance. These decays were, however, provided for sizeably different setups, namely in the noiseless case with constant regularization and in the noisy optimally regularized case. Intermediary settings have been left substantially uncharted. In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization. In particular, we show the existence of a transition in the noisy setting between the noiseless exponents to its noisy values as the sample complexity is increased. Finally, we illustrate how this crossover can also be observed on real data sets.

Equilibrium and non-equilibrium regimes in the learning of restricted Boltzmann machines* *This article is an updated version of: Decelle A, Furtlehner C and Seoane B 2021 Equilibrium and non-equilibrium regimes in the learning of restricted Boltzmann machines Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin, P S Liang and J Wortman Vaughan (New

Training restricted Boltzmann machines (RBMs) have been challenging for a long time due to the difficulty of precisely computing the log-likelihood gradient. Over the past few decades, many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem: the mixing time, i.e. the number of Monte Carlo iterations needed to sample new configurations from a model. In this work, we show that this mixing time plays a crucial role in the dynamics and stability of the trained model, and that RBMs operate in two well-defined regimes, namely equilibrium and out-of-equilibrium, depending on the interplay between this mixing time of the model and the number of steps, k, used to approximate the gradient. We further show empirically that this mixing time increases with the learning, which often implies a transition from one regime to another as soon as k becomes smaller than this time. In particular, we show that using the popular k (persistent) contrastive divergence approaches, with k small, the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium display faster dynamics, and a smooth convergence to dataset-like configurations during the sampling. Finally, we discuss how to exploit in practice both regimes depending on the task one aims to fulfill: (i) short k can be used to generate convincing samples in short learning times, (ii) large k (or increasingly large) is needed to learn the correct equilibrium distribution of the RBM. Finally, the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization.

Locality defeats the curse of dimensionality in convolutional teacher–student scenarios* *This article is an updated version of: Favero A, Cagnetta F and Wyart M 2021 Locality defeats the curse of dimensionality in convolutional teacher–student scenarios Advances in Neural Information Processing Systems vol 34, ed M Ranzato, A Beygelzimer, Y Dauphin, P S Liang and J Wortman Vaughan (New York: Curran

Convolutional neural networks perform a local and translationally-invariant treatment of the data: quantifying which of these two aspects is central to their success remains a challenge. We study this problem within a teacher–student framework for kernel regression, using ‘convolutional’ kernels inspired by the neural tangent kernel of simple convolutional architectures of given filter size. Using heuristic methods from physics, we find in the ridgeless case that locality is key in determining the learning curve exponent β (that relates the test error ϵ t ∼ P −β to the size of the training set P), whereas translational invariance is not. In particular, if the filter size of the teacher t is smaller than that of the student s, β is a function of s only and does not depend on the input dimension. We confirm our predictions on β empirically. We conclude by proving, under a natural universality assumption, that performing kernel regression with a ridge that decreases with the size of the training set leads to similar learning curve exponents to those we obtain in the ridgeless case.

Particle dual averaging: optimization of mean field neural network with global convergence rate analysis* *This article is an updated version of: Nitanda A, Wu D and Suzuki T 2021 Particle dual averaging: optimization of mean field neural network with global convergence rate analysis Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin, P S

We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally handle nonlinear functional on the probability space. An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain. By adapting finite-dimensional convex optimization theory into the space of measures, we analyze PDA in regularized empirical/expected risk minimization, and establish quantitative global convergence in learning two-layer mean field neural networks under more general settings. Our theoretical results are supported by numerical simulations on neural networks with reasonable size.

Rhythms of human attention and memory: An embedded process perspective.

It remains a dogma in cognitive neuroscience to separate human attention and memory into distinct modules and processes. Here we propose that brain rhythms reflect the embedded nature of these processes in the human brain, as evident from their shared neural signatures: gamma oscillations (30–90 Hz) reflect sensory information processing and activated neural representations (memory items). The theta rhythm (3–8 Hz) is a pacemaker of explicit control processes (central executive), structuring neural information processing, bit by bit, as reflected in the theta-gamma code. By representing memory items in a sequential and time-compressed manner the theta-gamma code is hypothesized to solve key problems of neural computation: (1) attentional sampling (integrating and segregating information processing), (2) mnemonic updating (implementing Hebbian learning), and (3) predictive coding (advancing information processing ahead of the real time to guide behavior). In this framework, reduced alpha oscillations (8–14 Hz) reflect activated semantic networks, involved in both explicit and implicit mnemonic processes. Linking recent theoretical accounts and empirical insights on neural rhythms to the embedded-process model advances our understanding of the integrated nature of attention and memory – as the bedrock of human cognition.

Neural synchrony in cortical networks: mechanisms and implications for neural information processing and coding.

Synchronization of neuronal discharges on the millisecond scale has long been recognized as a prevalent and functionally important attribute of neural activity. In this article, I review classical concepts and corresponding evidence of the mechanisms that govern the synchronization of distributed discharges in cortical networks and relate those mechanisms to their possible roles in coding and cognitive functions. To accommodate the need for a selective, directed synchronization of cells, I propose that synchronous firing of distributed neurons is a natural consequence of spike-timing-dependent plasticity (STDP) that associates cells repetitively receiving temporally coherent input: the “synchrony through synaptic plasticity” hypothesis. Neurons that are excited by a repeated sequence of synaptic inputs may learn to selectively respond to the onset of this sequence through synaptic plasticity. Multiple neurons receiving coherent input could thus actively synchronize their firing by learning to selectively respond at corresponding temporal positions. The hypothesis makes several predictions: first, the position of the cells in the network, as well as the source of their input signals, would be irrelevant as long as their input signals arrive simultaneously; second, repeating discharge patterns should get compressed until all or some part of the signals are synchronized; and third, this compression should be accompanied by a sparsening of signals. In this way, selective groups of cells could emerge that would respond to some recurring event with synchronous firing. Such a learned response pattern could further be modulated by synchronous network oscillations that provide a dynamic, flexible context for the synaptic integration of distributed signals. I conclude by suggesting experimental approaches to further test this new hypothesis.

Electric Love Therapy

AbstractIn the early 1970s, as the industrial economy crumbled around Detroit and its suburbs, Dr. H. C. Tien, an independent electroconvulsive therapist who ran the Michigan Institute of Psychosynthesis in Lansing, MI, advocated a “cybernetic” approach to family psychiatry. In Tien’s practice, one can learn how the liberatory kernel of religion and the truth of sexual difference—key components of moral treatment in nineteenth-century asylum reform—became amplified by emerging paradigms of neural nets and information processing. In Tien’s practice—what he called Electric Love Therapy—electric shock treatment became a technology of conversion and, more precisely, of sexual differentiation and spiritual cultivation. Tien’s is a disturbing example of how the regulation of sexuality and gender as private matters serves as resource and spur to secular demands to proprietize religion as an interior matter.

Visual Information Computing and Processing Model Based on Artificial Neural Network.

This paper analyzes the parallel and serial information processing structure of visual system and proposes a visual information processing model with three layers: visual receptor layer, visual information conduction and relay layer, and information processing layer of visual information computing and processing area. Based on the analysis, abstraction, and simplification of the biological prototype of each layer in the visual system, a framework model of an artificial neural system corresponding to the visual system is proposed. An artificial neural network model is proposed to simulate the mechanism of visual attention. A network model is formed by introducing the saliency mask map as additional information on the benchmark network, and the selective enhancement operation is performed on the extracted features in different regions according to the mask map. The experimental results show that the visual computing processing network model can effectively improve the classification performance of the network when the appropriate saliency mask is used. The visual information computing and processing model network can work effectively for different data sets and different structures of the benchmark network, which is a universal network model. The complexity of visual information computing and processing model network is very small, and the improvement of network performance is not at the cost of increasing model complexity, but in the way of improving network efficiency. The performance of artificial neural network visual information computation and processing model is directly related to the performance of saliency map used as mask map.

The EBRAINS Hodgkin-Huxley Neuron Builder: An online resource for building data-driven neuron models.

In the last decades, brain modeling has been established as a fundamental tool for understanding neural mechanisms and information processing in individual cells and circuits at different scales of observation. Building data-driven brain models requires the availability of experimental data and analysis tools as well as neural simulation environments and, often, large scale computing facilities. All these components are rarely found in a comprehensive framework and usually require ad hoc programming. To address this, we developed the EBRAINS Hodgkin-Huxley Neuron Builder (HHNB), a web resource for building single cell neural models via the extraction of activity features from electrophysiological traces, the optimization of the model parameters via a genetic algorithm executed on high performance computing facilities and the simulation of the optimized model in an interactive framework. Thanks to its inherent characteristics, the HHNB facilitates the data-driven model building workflow and its reproducibility, hence fostering a collaborative approach to brain modeling.

Small-world spiking neural network with anti-interference ability based on speech recognition under interference

The external interference can hamper the normal function of neuromorphic hardware under complex noise environment. Therefore, the study of brain-like models with anti-interference ability is a crucial issue. A bio-brain has the advantage of self-adaptability. The existing brain-like models are lack of bio-interpretability. Drawing from the advantage of bio-brain, the purpose of this paper is to investigate the anti-interference ability of brain-like models with bio-interpretability. In this study, we construct a spiking neural network with a small-world topology (SWSNN) as a brain-like model, in which Izhikevich neuron models and synaptic plasticity models with time-delay are used to represent the nodes and edges of the network, respectively. Then, the anti-interference ability of our SWSNN against different external noise is investigated, and its mechanism is explored. Further, by taking the speech recognition task as the case study, we verify the anti-interference ability of the SWSNN. Our simulation results indicate that: (i) The anti-interference ability of SWSNN is better than that of SNNs with other topologies. (ii) The discussion of neural information processing of the SWSNN under interference implies that the dynamic regulation of synaptic plasticity is an intrinsic factor of the anti-interference, and the topology is a factor that affects the anti-interference at the level of performance. (iii) Taking the speech recognition task as a case study, the SWSNN under different external noises still have almost the same high speech recognition accuracy, compared with the SWSNN without interference, and SWSNN yields better anti-interference ability than SNNs with other topologies in terms of the speech recognition accuracy. In addition, the anti-interference ability of our SWSNN framework is superior to that of existing liquid state machine (LSM) framework. Our simulation results lay a preliminary foundation for the neuromorphic hardware with robustness under complex noise environment.

Mixed vine copula flows for flexible modeling of neural dependencies.

Recordings of complex neural population responses provide a unique opportunity for advancing our understanding of neural information processing at multiple scales and improving performance of brain computer interfaces. However, most existing analytical techniques fall short of capturing the complexity of interactions within the concerted population activity. Vine copula-based approaches have shown to be successful at addressing complex high-order dependencies within the population, disentangled from the single-neuron statistics. However, most applications have focused on parametric copulas which bear the risk of misspecifying dependence structures. In order to avoid this risk, we adopted a fully non-parametric approach for the single-neuron margins and copulas by using Neural Spline Flows (NSF). We validated the NSF framework on simulated data of continuous and discrete types with various forms of dependency structures and with different dimensionality. Overall, NSFs performed similarly to existing non-parametric estimators, while allowing for considerably faster and more flexible sampling which also enables faster Monte Carlo estimation of copula entropy. Moreover, our framework was able to capture low and higher order heavy tail dependencies in neuronal responses recorded in the mouse primary visual cortex during a visual learning task while the animal was navigating a virtual reality environment. These findings highlight an often ignored aspect of complexity in coordinated neuronal activity which can be important for understanding and deciphering collective neural dynamics for neurotechnological applications.

Neural Information Processing and Computations of Two-Input Synapses.

Information processing in artificial neural networks is largely dependent on the nature of neuron models. While commonly used models are designed for linear integration of synaptic inputs, accumulating experimental evidence suggests that biological neurons are capable of nonlinear computations for many converging synaptic inputs via homo- and heterosynaptic mechanisms. This nonlinear neuronal computation may play an important role in complex information processing at the neural circuit level. Here we characterize the dynamics and coding properties of neuron models on synaptic transmissions delivered from two hidden states. The neuronal information processing is influenced by the cooperative and competitive interactions among synapses and the coherence of the hidden states. Furthermore, we demonstrate that neuronal information processing under two-input synaptic transmission can be mapped to linearly nonseparable XOR as well as basic AND/OR operations. In particular, the mixtures of linear and nonlinear neuron models outperform the fashion-MNIST test compared to the neural networks consisting of only one type. This study provides a computational framework for assessing information processing of neuron and synapse models that may be beneficial for the design of brain-inspired artificial intelligence algorithms and neuromorphic systems.

Algorithmic fairness in computational medicine.

SummaryMachine learning models are increasingly adopted for facilitating clinical decision-making. However, recent research has shown that machine learning techniques may result in potential biases when making decisions for people in different subgroups, which can lead to detrimental effects on the health and well-being of specific demographic groups such as vulnerable ethnic minorities. This problem, termed algorithmic bias, has been extensively studied in theoretical machine learning recently. However, the impact of algorithmic bias on medicine and methods to mitigate this bias remain topics of active discussion. This paper presents a comprehensive review of algorithmic fairness in the context of computational medicine, which aims at improving medicine with computational approaches. Specifically, we overview the different types of algorithmic bias, fairness quantification metrics, and bias mitigation methods, and summarize popular software libraries and tools for bias evaluation and mitigation, with the goal of providing reference and insights to researchers and practitioners in computational medicine.

Rapid adaptation of predictive models during language comprehension: Aperiodic EEG slope, individual alpha frequency and idea density modulate individual differences in real-time model updating.

Predictive coding provides a compelling, unified theory of neural information processing, including for language. However, there is insufficient understanding of how predictive models adapt to changing contextual and environmental demands and the extent to which such adaptive processes differ between individuals. Here, we used electroencephalography (EEG) to track prediction error responses during a naturalistic language processing paradigm. In Experiment 1, 45 native speakers of English listened to a series of short passages. Via a speaker manipulation, we introduced changing intra-experimental adjective order probabilities for two-adjective noun phrases embedded within the passages and investigated whether prediction error responses adapt to reflect these intra-experimental predictive contingencies. To this end, we calculated a novel measure of speaker-based, intra-experimental surprisal (“speaker-based surprisal”) as defined on a trial-by-trial basis and by clustering together adjectives with a similar meaning. N400 amplitude at the position of the critical second adjective was used as an outcome measure of prediction error. Results showed that N400 responses attuned to speaker-based surprisal over the course of the experiment, thus indicating that listeners rapidly adapt their predictive models to reflect local environmental contingencies (here: the probability of one type of adjective following another when uttered by a particular speaker). Strikingly, this occurs in spite of the wealth of prior linguistic experience that participants bring to the laboratory. Model adaptation effects were strongest for participants with a steep aperiodic (1/f) slope in resting EEG and low individual alpha frequency (IAF), with idea density (ID) showing a more complex pattern. These results were replicated in a separate sample of 40 participants in Experiment 2, which employed a highly similar design to Experiment 1. Overall, our results suggest that individuals with a steep aperiodic slope adapt their predictive models most strongly to context-specific probabilistic information. Steep aperiodic slope is thought to reflect low neural noise, which in turn may be associated with higher neural gain control and better cognitive control. Individuals with a steep aperiodic slope may thus be able to more effectively and dynamically reconfigure their prediction-related neural networks to meet current task demands. We conclude that predictive mechanisms in language are highly malleable and dynamic, reflecting both the affordances of the present environment as well as intrinsic information processing capabilities of the individual.

Disorder and the Neural Representation of Complex Odors.

Animals smelling in the real world use a small number of receptors to sense a vast number of natural molecular mixtures, and proceed to learn arbitrary associations between odors and valences. Here, we propose how the architecture of olfactory circuits leverages disorder, diffuse sensing and redundancy in representation to meet these immense complementary challenges. First, the diffuse and disordered binding of receptors to many molecules compresses a vast but sparsely-structured odor space into a small receptor space, yielding an odor code that preserves similarity in a precise sense. Introducing any order/structure in the sensing degrades similarity preservation. Next, lateral interactions further reduce the correlation present in the low-dimensional receptor code. Finally, expansive disordered projections from the periphery to the central brain reconfigure the densely packed information into a high-dimensional representation, which contains multiple redundant subsets from which downstream neurons can learn flexible associations and valences. Moreover, introducing any order in the expansive projections degrades the ability to recall the learned associations in the presence of noise. We test our theory empirically using data from Drosophila. Our theory suggests that the neural processing of sparse but high-dimensional olfactory information differs from the other senses in its fundamental use of disorder.

Separating Uncertainty from Surprise in Auditory Processing with Neurocomputational Models: Implications for Music Perception.

Advances in neuroscience have led to the emergence of two complementary theories of neural information processing. First, the “Bayesian brain hypothesis” proposes that the brain actively predicts and represents incoming sensory information as probabilities that are updated in a near-optimal

Neural correlates of risky decision making in Parkinson's disease patients with impulse control disorders.

Some patients with Parkinson's disease (PD) experience impulse control disorders (ICDs), characterized by deficient voluntary control over impulses, drives, or temptations regarding excessive hedonic behavior. The present study aimed to better understand the neural basis of impulsive, risky decision making in PD patients with ICDs by disentangling potential dysfunctions in decision and outcome mechanisms. We collected fMRI data from 20 patients with ICDs and 28 without ICDs performing an information gathering task. Patients viewed sequences of bead colors drawn from hidden urns and were instructed to infer the majority bead color in each urn. With each new bead, they could choose to either seek more evidence by drawing another bead (draw choice) or make an urn-inference (urn choice followed by feedback). We manipulated risk via the probability of bead color splits (80/20 vs. 60/40) and potential loss following an incorrect inference ($10 vs. $0). Patients also completed the Barratt Impulsiveness Scale (BIS) to assess impulsivity. Patients with ICDs showed greater urn choice-specific activation in the right middle frontal gyrus, overlapping the dorsal premotor cortex. Across all patients, fewer draw choices (i.e., more impulsivity) were associated with greater activation during both decision making and outcome processing in a variety of frontal and parietal areas, cerebellum, and bilateral striatum. Our findings demonstrate that ICDs in PD are associated with differences in neural processing of risk-related information and outcomes, implicating both reward and sensorimotor dopaminergic pathways.

A new patterns of self-organization activity of brain: Neural energy coding

According to the basic principles and methods of information theory, the operation way of neural coding is studied and analyzed by using the minimum mutual information and the maximum entropy principle. This paper describes how the principles of minimum mutual information and maximum entropy are used to evaluate the amount of information in neural responses. Its main contribution is as follows: (1) that the expression of neural information is closely related to the utilization of neural energy, and it is found that the highly evolved nervous system strictly follows the two basic principles of economy and efficiency in energy consumption and utilization; (2) In order to verify the relationship between neural information processing and energy utilization, this paper uses the concept of energy-efficiency ratio to measure the economy and high efficiency of the nervous system in term of energy utilization by using the maximum entropy principle; (3) The numerical results show that the energy consumed by the nervous system reflects not only the internal law of neural information conduction and processing, but also the self-organization structure of neural information coding. The results suggest that energy neural coding, a novel neural information processing method, can be used to understand how brain activity works. Such a coding pattern can not only be extended to research the large-scale neuroscience field, but also unify brain models at all levels by use of the energy theory. This will provide a scientific theoretical basis for the exploration of how the brain works and the computational principles of brain-like artificial intelligence.

The Neuromodulatory Role of the Noradrenergic and Cholinergic Systems and Their Interplay in Cognitive Functions: A Focused Review.

The noradrenergic and cholinergic modulation of functionally distinct regions of the brain has become one of the primary organizational principles behind understanding the contribution of each system to the diversity of neural computation in the central nervous system. Decades of work has shown that a diverse family of receptors, stratified across different brain regions, and circuit-specific afferent and efferent projections play a critical role in helping such widespread neuromodulatory systems obtain substantial heterogeneity in neural information processing. This review briefly discusses the anatomical layout of both the noradrenergic and cholinergic systems, as well as the types and distributions of relevant receptors for each system. Previous work characterizing the direct and indirect interaction between these two systems is discussed, especially in the context of higher order cognitive functions such as attention, learning, and the decision-making process. Though a substantial amount of work has been done to characterize the role of each neuromodulator, a cohesive understanding of the region-specific cooperation of these two systems is not yet fully realized. For the field to progress, new experiments will need to be conducted that capitalize on the modular subdivisions of the brain and systematically explore the role of norepinephrine and acetylcholine in each of these subunits and across the full range of receptors expressed in different cell types in these regions.