Neural Representation Research Articles

3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation.

In contrast to the traditional avatar creation pipeline which is a costly process, contemporary generative approaches directly learn the data distribution from photographs. While plenty of works extend unconditional generative models and achieve some levels of controllability, it is still challenging to ensure multi-view consistency, especially in large poses. In this work, we propose a network that generates 3D-aware portraits while being controllable according to semantic parameters regarding pose, identity, expression and illumination. Our network uses neural scene representation to model 3D-aware portraits, whose generation is guided by a parametric face model that supports explicit control. While the latent disentanglement can be further enhanced by contrasting images with partially different attributes, there still exists noticeable inconsistency in non-face areas when animating expressions. We solve this by proposing a volume blending strategy in which we form a composite output by blending dynamic and static areas, with two parts segmented from the jointly learned semantic field. Our method outperforms prior arts in extensive experiments, producing realistic portraits with vivid expression in natural lighting when viewed from free viewpoints. It also demonstrates generalization ability to real images as well as out-of-domain data, showing great promise in real applications.

Piecewise Developable Modeling via Implicit Neural Deformation and Feature-Guided Cutting.

We propose a novel and automatic method to model shapes using a small set of discrete developable patches. Central to our approach is using implicit neural shape representation that makes our algorithm independent of tessellation and allows us to obtain the Gaussian curvature of each point analytically. With this powerful representation, we first deform the input shape to be an almost developable shape with clear and sparse salient feature curves. Then, we convert the deformed implicit field to a triangle mesh, which is further cut to disk topology along parts of the sparse feature curves. Finally, we achieve the resulting piecewise developable mesh by alternatingly optimizing discrete developability, enforcing manufacturability constraints, and merging patches. The feasibility and practicability of our method are demonstrated over various shapes. Compared to the state-of-the-art methods, our method achieves a better tradeoff between the number of developable patches and the approximation error.

A high‐performance asynchronous readout circuit with cascade function for a neural recording micro‐electrode array

AbstractThe data‐driven machine learning technology used for neural decoding emphasizes the requirements of in vivo and in vitro neural signal acquisition with high spatial and temporal resolution. However, micro‐electrode arrays (MEAs) that simultaneously achieve high spatial and temporal resolution neural signal acquisition are not yet available. Meanwhile, the high data bandwidth of large‐scale MEA brings challenges in power consumption, data transmission, storage, and neural signal processing. This research aims to improve the spatial and temporal resolution of MEA, reduce the cost and time of large‐scale MEA customization through cascading, and reduce the bandwidth of large‐scale MEA through spike compression. Firstly, based on in‐pixel spike detection, a row‐based neural spike readout mechanism and related array circuit to improve the neural spike readout performance and temporal resolution of neural signal acquisition is proposed. To further enhance the spatial resolution and reduce the risk of large‐scale MEA fabrication, the cascading capability of the readout circuit is explored. Lastly, spatial correlation‐based neural spike encoding is proposed to reduce the data bandwidth, achieving a 5.2× compression rate. This is a study on implementing large‐scale MEA through cascaded readout circuits and novel study to utilize the spatial correlation between detected neural spikes for further compression.

Wavefield Reconstruction of Distributed Acoustic Sensing: Lossy Compression, Wavefield Separation, and Edge Computing

AbstractDistributed acoustic sensing (DAS) presents challenges and opportunities for seismological research and data management. This study explores wavefield reconstruction using deep learning methods for data compression and wavefield separation. We test various architectures to treat DAS data as two‐dimensional arrays, such as the implicit neural representation (INR) models and the SHallow REcurrent Decoder (SHRED) model. The INR models present better data compression ability but do not generalize over space and time, a major practical limitation. On the other hand, SHRED generalizes over space and time for a single optical fiber with data from 20% decimated channels for the reconstruction. Despite good performance in reconstructing long‐wavelength features, the shallow recurrent decoder does not reconstruct transient earthquake wavefields at shorter wavelengths, limiting its usability for seismic data transmission. Nevertheless, we leverage wavefield reconstruction of ocean waves to separate them from the seismic wavefield and improve seismological use cases for earthquake detection and Earth imaging. In summary, as a lightweight deep learning model, SHRED is well suited for wavefield separation and lossy compression of the DAS data.

Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models

Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models

Foundation model of neural activity predicts response to new stimulus types and anatomy.

The complexity of neural circuits makes it challenging to decipher the brain's algorithms of intelligence. Recent breakthroughs in deep learning have produced models that accurately simulate brain activity, enhancing our understanding of the brain's computational objectives and neural coding. However, these models struggle to generalize beyond their training distribution, limiting their utility. The emergence of foundation models, trained on vast datasets, has introduced a new AI paradigm with remarkable generalization capabilities. We collected large amounts of neural activity from visual cortices of multiple mice and trained a foundation model to accurately predict neuronal responses to arbitrary natural videos. This model generalized to new mice with minimal training and successfully predicted responses across various new stimulus domains, such as coherent motion and noise patterns. It could also be adapted to new tasks beyond neural prediction, accurately predicting anatomical cell types, dendritic features, and neuronal connectivity within the MICrONS functional connectomics dataset. Our work is a crucial step toward building foundation brain models. As neuroscience accumulates larger, multi-modal datasets, foundation models will uncover statistical regularities, enabling rapid adaptation to new tasks and accelerating research.

Neurosimilator for Undergraduate Biophysics and Neurophysiology Courses.

Stringent animal welfare principles are forcing undergraduate instructors to avoid the use of animals. Therefore, many hands-on lab sessions using laboratory animals are progressively replaced by computer simulations. These versatile software simulations permit the observation of the behavior of biological systems under a great variety of experimental conditions. While this versatility is important, computer simulations often work even when a student makes wrong assumptions, a situation that poses its own pedagogical problem. Hands-on learning provides pupils with the opportunity to safely make mistakes and learn organically through trial and error and should therefore still be promoted. We propose an electronic model of an excitable cell composed of different modules representing different parts of a neuron - dendrites, soma, axon and node of Ranvier. We describe a series of experiments that allow students to better understand differences between passive and active cell responses and differences between myelinated and demyelinated axons. These circuits can also be used to demonstrate temporal and spatial summation of signals coming to the neuron via dendrites, as well as the neuron coding by firing frequency. Finally, they permit experimental determination along with theoretical calculations of important biophysical properties of excitable cells, such as rheobase, chronaxie and space constant. This open-source model has been successfully integrated into an undergraduate course of the physiology of excitable cells and student feedback assessment reveals that it helped students to understand important notions of the course. Thus, this neuromorphic circuit could be a valuable tool for biophysics and neuroscience courses in other universities.

Event boundaries drive norepinephrine release and distinctive neural representations of space in the rodent hippocampus.

Episodic memories are temporally segmented around event boundaries that tend to coincide with moments of environmental change. During these times, the state of the brain should change rapidly, or reset, to ensure that the information encountered before and after an event boundary is encoded in different neuronal populations. Norepinephrine (NE) is thought to facilitate this network reorganization. However, it is unknown whether event boundaries drive NE release in the hippocampus and, if so, how NE release relates to changes in hippocampal firing patterns. The advent of the new GRABNE sensor now allows for the measurement of NE binding with sub-second resolution. Using this tool in mice, we tested whether NE is released into the dorsal hippocampus during event boundaries defined by unexpected transitions between spatial contexts and presentations of novel objections. We found that NE binding dynamics were well explained by the time elapsed after each of these environmental changes, and were not related to conditioned behaviors, exploratory bouts of movement, or reward. Familiarity with a spatial context accelerated the rate in which phasic NE binding decayed to baseline. Knowing when NE is elevated, we tested how hippocampal coding of space differs during these moments. Immediately after context transitions we observed relatively unique patterns of neural spiking which settled into a modal state at a similar rate in which NE returned to baseline. These results are consistent with a model wherein NE release drives hippocampal representations away from a steady-state attractor. We hypothesize that the distinctive neural codes observed after each event boundary may facilitate long-term memory and contribute to the neural basis for the primacy effect.

Neuronal firing rate diversity lowers the dimension of population covariability.

Populations of neurons produce activity with two central features. First, neuronal responses are very diverse - specific stimuli or behaviors prompt some neurons to emit many action potentials, while other neurons remain relatively silent. Second, the trial-to-trial fluctuations of neuronal response occupy a low dimensional space, owing to significant correlations between the activity of neurons. These two features define the quality of neuronal representation. We link these two aspects of population response using a recurrent circuit model and derive the following relation: the more diverse the firing rates of neurons in a population, the lower the effective dimension of population trial-to-trial covariability. This surprising prediction is tested and validated using simultaneously recorded neuronal populations from numerous brain areas in mice, non-human primates, and in the motor cortex of human participants. Using our relation we present a theory where a more diverse neuronal code leads to better fine discrimination performance from population activity. In line with this theory, we show that neuronal populations across the brain exhibit both more diverse mean responses and lower-dimensional fluctuations when the brain is in more heightened states of information processing. In sum, we present a key organizational principle of neuronal population response that is widely observed across the nervous system and acts to synergistically improve population representation.

Lightweight High-Speed Photography Built on Coded Exposure and Implicit Neural Representation of Videos

Lightweight High-Speed Photography Built on Coded Exposure and Implicit Neural Representation of Videos

SiSRS: Signed social recommender system using deep neural network representation learning

Most approaches used in social recommender systems concentrate mostly on positive relationshipswithin the social network, frequently ignoring the insightful information that negative relationships can offer. A more thorough understanding of user characteristics and behaviors within a social network can result from integrating positive and negative links to construct a signed graph. However, there are two major challenges in developing signed social recommender systems. Considering negative links in these systems needs to be done carefully to make sure that these relationships are appropriately represented and used. Moreover, it is difficult to predict user interactions and preferences effectively when relationships represented by positive and negative links conflict, known as social inconsistency. Therefore, working with signed graphs requires the management of social inconsistency. To address these issues, in this paper, a signed social recommender system called SiSRS is presented using a deep architecture combined with network representation learning. The SiSRS is composed of three principal components. First, the application of signed graph attention networks is explored to collate and disseminate information across the network via graph motifs, leading to the creation of node embeddings. Second, a deep autoencoder model is employed to assimilate top-k semantic signed social data within the deep learning framework. Third, a novel loss function is formulated to reduce the impact of social inconsistency. The efficacy of the SiSRS model was evaluated through extensive experiments conducted on two datasets in terms of various evaluation metrics. The results indicated a superior performance of this approach compared to existing state-of-the-art methods.

The neural representation of self, close, and famous others: An electrophysiological investigation on the social brain

ABSTRACT It is well established that the Self has a unique representation in the social brain, as evident from the Self-Referential Effect (SRE). However, the timing and neural mechanisms underlying the representation of individuals with varying degrees of closeness and emotional relevance to the Self remain unclear. Twenty-two participants read 260 personality traits and decided whether they described themselves, a close friend, or an admired celebrity. A strong Self-Referential Effect (SRE) was found at behavioral, ERP, and neuroimaging levels. Three anterior ERP components were identified as sensitive to social information: a P200 (250–350 ms) responding to famous others’ traits, a P600 (500–700 ms) responding to self-trait processing, and a late positivity (800–950 ms) responding to self-trait processing and close traits. Source reconstructions revealed partially overlapping but distinct neural sources for each individual. The right precuneus (bodily self) and inferior frontal areas (inner voice) were active only during self-processing, while the right medial prefrontal cortex (BA10) was consistently active across tasks, showing a robust SRE. These findings provide insights into the neural mechanisms underlying the representation of the Self in social contexts.

Weak signal detection capacity of type-II Morris–Lecar neuron system under presynaptic bombardments

The information encoding mechanism of the neuronal system is quite complex and is sensitive to environmental conditions. Neuronal noise, especially synaptic random inputs, is a natural part of the brain. Theoretical and experimental studies have revealed that noise plays a vital role in signal frequency selectivity and weak sensory input perception of the neuronal system. There is a consensus that sufficiently intense noise can trigger rhythmic brain activity, and these noise-induced oscillations could improve signal processing of the neuronal system, particularly when the external signal frequency aligns with the frequency of the noise-induced rhythm. Such behavior in response to noise in the signal-processing mechanism of biological systems is elucidated by stochastic resonance. In this regard, this paper systematically analyzes the stochastic resonance in a Type-II Morris–Lecar neuron under excitatory and inhibitory background spike bombardments. Our results indicate that Morris–Lecar Type II neurons enhance their signal coding capacity by exploiting background synaptic activity. In addition, it has been revealed that the biological properties of the background synaptic connections of neurons that oscillate at different frequencies, such as delta, theta, alpha, beta, and gamma bands, are different. On the other hand, it has been found that neurons adapt to the properties of pre-synaptic neurons to encode weak signals. This study ensures novel insights into the functional role of background synaptic input in neuronal coding mechanisms by demonstrating a biophysical factor of stochastic resonance via numerical analyses. Furthermore, our results show that the properties of the background synaptic inputs significantly determine what kind of signal to encode or not. These results are an important indicator of the signal frequency selectivity of neurons, such as auditory cells.

Effects of connectivity hyperalignment (CHA) on estimated brain network properties: from coarse-scale to fine-scale.

Recent gains in functional magnetic resonance imaging (fMRI) studies have been driven by increasingly sophisticated statistical and computational techniques and the ability to capture brain data at finer spatial and temporal resolution. These advances allow researchers to develop population-level models of the functional brain representations underlying behavior, performance, clinical status, and prognosis. However, even following conventional preprocessing pipelines, considerable inter-individual disparities in functional localization persist, posing a hurdle to performing compelling population-level inference. Persistent misalignment in functional topography after registration and spatial normalization will reduce power in developing predictive models and biomarkers, reduce the specificity of estimated brain responses and patterns, and provide misleading results on local neural representations and individual differences. This study aims to determine how connectivity hyperalignment (CHA)-an analytic approach for handling functional misalignment-can change estimated functional brain network topologies at various spatial scales from the coarsest set of parcels down to the vertex-level scale. The findings highlight the role of CHA in improving inter-subject similarities, while retaining individual-specific information and idiosyncrasies at finer spatial granularities. This highlights the potential for fine-grained connectivity analysis using this approach to reveal previously unexplored facets of brain structure and function.

A new method of rock type identification based on transformer by utilizing acoustic emission.

The characterization and analysis of rock types based on acoustic emission (AE) signals have long been focal points in earth science research. However, traditional analysis methods struggle to handle the influx of big data. While signal processing methods combined with deep learning have found widespread use in various process analyses and state identification, effective feature extraction using progressive fusion technology still faces challenges in the field of intelligent rock type identification. To address this issue, our study proposes a novel framework for rock type identification based on AE and introduces a new signal identification model called 3CTNet. This model integrates convolutional neural networks (CNNs) and Transformer encoder, intelligently identifying AE of different rock fractures by establishing dependencies between adjacent positions within the data and gradually extracting advanced features. Furthermore, we experimentally compare five oversampling methods, ultimately selecting the adaptive synthetic sampling method (ADASYN) to balance the dataset and enhance the model's robustness and generalization ability. Comparison of the internal structure of our model with a series of time series processing models demonstrates the effectiveness of the proposed model structure. Experimental results showcase the high identification accuracy of the intelligent rock type identification model based on 3CTNet, with an overall identification accuracy reaching 98.780%. Our proposed method lays a solid foundation for the efficient and accurate identification of formation rock types in geological exploration and oil and gas development endeavors.

Impaired Dynamics of Positional and Contextual Neural Coding in an Alzheimer's Disease Rat Model.

The hippocampal representation of space, formed by the collective activity of populations of place cells, is considered as a substrate of spatial memory. Alzheimer's disease (AD), a widespread severe neurodegenerative condition of multifactorial origin, typically exhibits spatial memory deficits among its early clinical signs before more severe cognitive impacts develop. To investigate mechanisms of spatial memory impairment in a double transgenic rat model of AD. In this study, we utilized 9-12-month-old double-transgenic TgF344-AD rats and age-matched controls to analyze the spatial coding properties of CA1 place cells. We characterized the spatial memory representation, assessed cells' spatial information content and direction-specific activity, and compared their population coding in familiar and novel conditions. Our findings revealed that TgF344-AD animals exhibited lower precision in coding, as evidenced by reduced spatial information and larger receptive zones. This impairment was evident in maps representing novel environments. While controls instantly encoded directional context during their initial exposure to a novel environment, transgenics struggled to incorporate this information into the newly developed hippocampal spatial representation. This resulted in impairment in orthogonalization of stored activity patterns, an important feature directly related to episodic memory encoding capacity. Overall, the results shed light on the nature of impairment at both the single-cell and population levels in the transgenic AD model. In addition to the observed spatial coding inaccuracy, the findings reveal a significantly impaired ability to adaptively modify and refine newly stored hippocampal memory patterns.

Fixational Eye Movements Enhance the Precision of Visual Information Transmitted by the Primate Retina.

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Real-Time Dense Visual SLAM with Neural Factor Representation

Developing a high-quality, real-time, dense visual SLAM system poses a significant challenge in the field of computer vision. NeRF introduces neural implicit representation, marking a notable advancement in visual SLAM research. However, existing neural implicit SLAM methods suffer from long runtimes and face challenges when modeling complex structures in scenes. In this paper, we propose a neural implicit dense visual SLAM method that enables high-quality real-time reconstruction even on a desktop PC. Firstly, we propose a novel neural scene representation, encoding the geometry and appearance information of the scene as a combination of the basis and coefficient factors. This representation allows for efficient memory usage and the accurate modeling of high-frequency detail regions. Secondly, we introduce feature integration rendering to significantly improve rendering speed while maintaining the quality of color rendering. Extensive experiments on synthetic and real-world datasets demonstrate that our method achieves an average improvement of more than 60% for Depth L1 and ATE RMSE compared to existing state-of-the-art methods when running at 9.8 Hz on a desktop PC with a 3.20 GHz Intel Core i9-12900K CPU and a single NVIDIA RTX 3090 GPU. This remarkable advancement highlights the crucial importance of our approach in the field of dense visual SLAM.

GNN-based multi-source domain prototype representation for cross-subject EEG emotion recognition

Emotion recognition based on electroencephalography (EEG) signals is a major area of affective computing. However, the existence of distributional differences between subjects has greatly hindered the large-scale application of EEG emotion recognition techniques. Most of the existing cross-subject methods primarily concentrate on treating multiple subjects as a single source domain. These methods lead to significant distributional differences within the source domain, which hinder the model’s ability to generalise effectively to target subjects. In this paper, we propose a new method that combines graph neural network-based prototype representation of multiple source domains with clustering similarity loss. It consists of three parts: multi-source domain prototype representation, graph neural network and loss. Multi-source domain prototype representation treats different subjects in the source domain as sub-source domains and extracts prototype features, which learns a more fine-grained feature representation. Graph neural network can better model the association properties between prototypes and samples. In addition, we propose a similarity loss based on clustering idea. The loss makes maximum use of similarity between samples in the target domain while ensuring that the classification performance does not degrade. We conduct extensive experiments on two benchmark datasets, SEED and SEED IV. The experimental results validate the effectiveness of the proposed multi-source domain fusion approach and indicate its superiority over existing methods in cross-subject classification tasks.

Crowdsourced Geospatial Intelligence: Constructing 3D Urban Maps with Satellitic Radiance Fields

In urban planning and research, 3D city maps are crucial for activities such as cellular network design, urban development, and climate research. Traditionally, creating these models has involved costly techniques like manual 3D mapping, interpretation of satellite or aerial images, or the use of sophisticated depth-sensing equipment. In this work, we propose a novel approach to develop 3D urban maps by examining the influence of urban structures on satellite signals, using GPS records crowdsourced from hundreds of smartphones during everyday user movements. We introduce the concept of satellitic radiance fields (SaRF), a novel neural scene representation technique designed to capture the distribution of GPS signals in urban settings. SaRF employs a sparse voxel octree framework to depict voxel-centric implicit fields, capturing physical properties like the density of each voxel. This model is progressively refined using a differentiable ray-marching process, ultimately leading to the reconstruction of 3D urban maps. Our thorough experimental evaluation, which incorporates approximately 27.4 million GPS records, reveals an average reconstruction accuracy of 83.1% in six varied urban scenes.