Complex Spatiotemporal Information Research Articles

MOCNN: A Multiscale Deep Convolutional Neural Network for ERP-Based Brain-Computer Interfaces.

Event-related potentials (ERPs) reflect neurophysiological changes of the brain in response to external events and their associated underlying complex spatiotemporal feature information is governed by ongoing oscillatory activity within the brain. Deep learning methods have been increasingly adopted for ERP-based brain-computer interfaces (BCIs) due to their excellent feature representation abilities, which allow for deep analysis of oscillatory activity within the brain. Features with higher spatiotemporal frequencies usually represent detailed and localized information, while features with lower spatiotemporal frequencies usually represent global structures. Mining EEG features from multiple spatiotemporal frequencies is conducive to obtaining more discriminative information. A multiscale feature fusion octave convolution neural network (MOCNN) is proposed in this article. MOCNN divides the ERP signals into high-, medium- and low-frequency components corresponding to different resolutions and processes them in different branches. By adding mid- and low-frequency components, the feature information used by MOCNN can be enriched, and the required amount of calculations can be reduced. After successive feature mapping using temporal and spatial convolutions, MOCNN realizes interactive learning among different components through the exchange of feature information among branches. Classification is accomplished by feeding the fused deep spatiotemporal features from various components into a fully connected layer. The results, obtained on two public datasets and a self-collected ERP dataset, show that MOCNN can achieve state-of-the-art ERP classification performance. In this study, the generalized concept of octave convolution is introduced into the field of ERP-BCI research, which allows effective spatiotemporal features to be extracted from multiscale networks through branch width optimization and information interaction at various scales.

Temporal–Semantic Aligning and Reasoning Transformer for Audio-Visual Zero-Shot Learning

Zero-shot learning (ZSL) enables models to recognize categories not encountered during training, which is crucial for categories with limited data. Existing methods overlook efficient temporal modeling in multimodal data. This paper proposes a Temporal–Semantic Aligning and Reasoning Transformer (TSART) for spatio-temporal modeling. TSART uses the pre-trained SeLaVi network to extract audio and visual features and explores the semantic information of these modalities through audio and visual encoders. It incorporates a temporal information reasoning module to enhance the capture of temporal features in audio, and a cross-modal reasoning module to effectively integrate audio and visual information, establishing a robust joint embedding representation. Our experimental results validate the effectiveness of this approach, demonstrating outstanding Generalized Zero-Shot Learning (GZSL) performance on the UCF101 Generalized Zero-Shot Learning (UCF-GZSL), VGGSound-GZSL, and ActivityNet-GZSL datasets, with notable improvements in the Harmonic Mean (HM) evaluation. These results indicate that TSART has great potential in handling complex spatio-temporal information and multimodal fusion.

CANet: Comprehensive Attention Network for video-based action recognition

Attention mechanisms play a crucial role in improving action recognition performance. A video, a type of 3D data, can be effectively explored using attention mechanisms from temporal, spatial, and channel dimensions. However, existing methods based on 2D CNN tend to deal with complex spatiotemporal information from one or two of the dimensions, which eventually hampers their overall performance. In this paper, we propose a novel Comprehensive Attention Network (CANet) to model spatiotemporal information in all three dimensions adaptively. CANet is composed of three core plug-and-play components, namely the Global Guided Short-term Motion Module (GG-SMM), the Second-order Guided Long-term Motion Module (SG-LMM), and the Spatial Motion Adaptive Module (SMAM). Specifically, (1) the GG-SMM module is designed to represent local motion clues in the short-term temporal dimension to improve the classification accuracy of fast-tempo actions. (2) The SG-LMM module is designed to jointly motivate fine-grained motion information in the long-term temporal and channel dimensions, thereby facilitating the discrimination of long-term motions. (3) The SMAM module is used to represent motion-sensitive regions in the spatial dimension by learning the spatial object offsets. Extensive experiments have been conducted on four widely used action recognition benchmarks, namely, Something-Something V1, Kinetics-400, UCF-101, and HMDB-51. Experimental results demonstrate that the proposed CANet achieves excellent performance compared with other state-of-the-art methods.

Link prediction by continuous spatiotemporal representation via neural differential equations

With the continuous advancement of data science and machine learning, temporal link prediction has emerged as a crucial aspect of dynamic network analysis, providing significant research and application potential across various domains. While deep learning techniques have achieved remarkable results in temporal link prediction, most existing studies have focused on discrete model frameworks. These frameworks face limitations in capturing deep structural features and effectively aggregating temporal information. To address these limitations, we draw inspiration from neural differential equations to propose a Continuous Temporal Graph Neural Differential Equation (CTGNDE) network model for temporal link prediction. Specifically, we design a spatial graph Ordinary Differential Equation (ODE) to capture the spatial correlations inherent in complex spatiotemporal information. Then we employ Neural Controlled Differential Equation (Neural CDE) to learn complex evolution patterns and effectively aggregate temporal information. Finally, we characterize completely continuous and more accurate hidden state trajectories by coupling spatial and temporal messages. Experiments conducted on 10 real-world network datasets validated the superior performance of the CTGNDE model over the state-of-the-art baselines.

Dual-track spatio-temporal learning for urban flow prediction with adaptive normalization

Robust urban flow prediction is crucial for transportation planning and management in urban areas. Although recent advances in modeling spatio-temporal correlations have shown potential, most models fail to adequately consider the complex spatio-temporal semantic information present in real-world scenarios. We summarize the following three primary limitations in existing models: a) The majority of existing models project overall time periods into the same latent space, neglecting the diverse temporal semantics between different time intervals. b) Existing models tend to capture spatial dependencies from a locale perspective such as surroundings but do not pay attention to the global influence factors. c) Beyond the spatio-temporal properties, the dynamics and instability of the data sequences introduce perturbations to the prediction results, potentially leading to model degradation. To address these issues, we propose a dual-track spatial-temporal learning module named DualST for accurate urban flow inference. To more effectively differentiate semantic information in the time dimension, we assign the overall time scales into closeness and periodicity. The dual-track module, which includes temporal causality inference and temporal contextual inference, simultaneously exploits the dynamic evolutionary trends and periodic traffic patterns, respectively. The proposed DualST captures global spatial features in a self-supervised manner which not only enriches the spatial semantics but also avoids introducing additional prior knowledge. To eliminate the instability caused by dynamics, we first adopt spatio-temporal adaptive normalization to learn appropriate data sequence normalization. We evaluate the proposed DualST on two typical urban flow datasets. The experiment results show that our model not only exhibits a consistent superiority over various state-of-the-art baselines but also has remarkable generalization capability.

The reconstruction of flows from spatiotemporal data by autoencoders

Artificial neural networks have become essential tools in data science for uncovering insights from complex data. However, they are usually seen as black boxes. In this work we explore how an autoencoder processes complex spatiotemporal information. We analyze the topological structure of reconstructed flows in the latent space of an autoencoder for two distinct test cases. The first case involves a synthetic spatiotemporal pattern for the temperature field in a convective problem, illustrating a classic extended system that exhibits low-dimensional chaos. The second case focuses on an experimental recording of the labial oscillations responsible for sound production in an avian vocal organ, as an example of periodic dynamics in a biological system. We find that the state representation in its latent space can be topologically equivalent to the phase space of the problem. Autoencoders thus retain phase space representations of the data hidden in its latent layer.

SAR image classification based on spiking neural network through spike-time dependent plasticity and gradient descent

At present, the Synthetic Aperture Radar (SAR) image classification method based on Convolution Neural Network (CNN) has faced some problems such as poor noise resistance and generalization ability. Spiking Neural Network (SNN) is one of the core components of brain-like intelligence and has good application prospects. This article constructs a complete SAR image classifier based on unsupervised and supervised learning of SNN by using spike sequences with complex spatio-temporal information. We firstly expound on the spiking neuron model, the receptive field of SNN, and the construction of spike sequence. Then we put forward an unsupervised learning algorithm based on STDP and a supervised learning algorithm based on gradient descent in series. The average classification accuracy of single layer and bilayer unsupervised learning SNN in three categories images on MSTAR dataset is 81.1% and 82.9%, respectively. Furthermore, the convergent output spike sequences of unsupervised learning can be used as teaching signals. Based on the TensorFlow framework, a single layer supervised learning SNN is built from the bottom, and the classification accuracy reaches 90.2%. By comparing noise resistance and model parameters between SNNs and CNNs, the effectiveness and outstanding advantages of SNN are verified. Code to reproduce our experiments is available at https://github.com/Jiankun-chen/Supervised-SNN-with-GD.

A 3D-printed neuromorphic humanoid hand for grasping unknown objects

SummaryCompared with conventional von Neumann’s architecture-based processors, neuromorphic systems provide energy-saving in-memory computing. We present here a 3D neuromorphic humanoid hand designed for providing an artificial unconscious response based on training. The neuromorphic humanoid hand system mimics the reflex arc for a quick response by managing complex spatiotemporal information. A 3D structural humanoid hand is first integrated with 3D-printed pressure sensors and a portable neuromorphic device that was fabricated by the multi-axis robot 3D printing technology. The 3D neuromorphic robot hand provides bioinspired signal perception, including detection, signal transmission, and signal processing, together with the biomimetic reflex arc function, allowing it to hold an unknown object with an automatically increased gripping force without a conventional controlling processor. The proposed system offers a new approach for realizing an unconscious response with an artificially intelligent robot.

Directional biases in whole hand motion perception revealed by mid-air tactile stimulation

Many emerging technologies are attempting to leverage the tactile domain to convey complex spatiotemporal information translated directly from the visual domain, such as shape and motion. Despite the intuitive appeal of touch for communication, we do not know to what extent the hand can substitute for the retina in this way. Here we ask whether the tactile system can be used to perceive complex whole hand motion stimuli, and whether it exhibits the same kind of established perceptual biases as reported in the visual domain. Using ultrasound stimulation, we were able to project complex moving dot percepts onto the palm in mid-air, over 30 cm above an emitter device. We generated dot kinetogram stimuli involving motion in three different directional axes (‘Horizontal’, ‘Vertical’, and ‘Oblique’) on the ventral surface of the hand. Using Bayesian statistics, we found clear evidence that participants were able to discriminate tactile motion direction. Furthermore, there was a marked directional bias in motion perception: participants were both better and more confident at discriminating motion in the vertical and horizontal axes of the hand, compared to those stimuli moving obliquely. This pattern directly mirrors the perceptional biases that have been robustly reported in the visual field, termed the ‘Oblique Effect’. These data demonstrate the existence of biases in motion perception that transcend sensory modality. Furthermore, we extend the Oblique Effect to a whole hand scale, using motion stimuli presented on the broad and relatively low acuity surface of the palm, away from the densely innervated and much studied fingertips. These findings highlight targeted ultrasound stimulation as a versatile method to convey potentially complex spatial and temporal information without the need for a user to wear or touch a device.

A model integrating multiple processes of synchronization and coherence for information instantiation within a cortical area

What is the form of dynamic, e.g., sensory, information in the mammalian cortex? Information in the cortex is modeled as a coherence map of a mixed chimera state of synchronous, phasic, and disordered minicolumns. The theoretical model is built on neurophysiological evidence. Complex spatiotemporal information is instantiated through a system of interacting biological processes that generate a synchronized cortical area, a coherent aperture. Minicolumn elements are grouped in macrocolumns in an array analogous to a phased-array radar, modeled as an aperture, a “hole through which radiant energy flows.” Coherence maps in a cortical area transform inputs from multiple sources into outputs to multiple targets, while reducing complexity and entropy. Coherent apertures can assume extremely large numbers of different information states as coherence maps, which can be communicated among apertures with corresponding very large bandwidths. The coherent aperture model incorporates considerable reported research, integrating five conceptually and mathematically independent processes: 1) a damped Kuramoto network model, 2) a pumped area field potential, 3) the gating of nearly coincident spikes, 4) the coherence of activity across cortical lamina, and 5) complex information formed through functions in macrocolumns. Biological processes and their interactions are described in equations and a functional circuit such that the mathematical pieces can be assembled the same way the neurophysiological ones are. The model can be conceptually convolved over the specifics of local cortical areas within and across species. A coherent aperture becomes a node in a graph of cortical areas with a corresponding distribution of information.

Enhanced view-independent representation method for skeleton-based human action recognition

Human action recognition is an important branch of computer vision science. It is a challenging task based on skeletal data because joints have complex spatiotemporal information. In this work, we propose a method for action recognition, which consists of three parts: view-independent representation, combination with cumulative Euclidean distance, and combined model. First, the action sequence becomes view-independent representations independent of the view. Second, these representations are further combined with cumulative Euclidean distances, so the joints more closely associated with the action are emphasised. Then, a combined model is adopted to extract these representation features and classify actions. It consists of two parts, a regular three-layer BLSTM network, and a temporal attention module. Experimental results on two multi-view benchmark datasets, Northwestern-UCLA and NTU RGB + D, demonstrate the effectiveness of our complete method. Despite its simple architecture and the use of only one type of action feature, it can still significantly improve recognition performance and has strong robustness.

Uncertainty Analysis of Object-Based Land-Cover Classification Using Sentinel-2 Time-Series Data

Recently, time-series from optical satellite data have been frequently used in object-based land-cover classification. This poses a significant challenge to object-based image analysis (OBIA) owing to the presence of complex spatio-temporal information in the time-series data. This study evaluates object-based land-cover classification in the northern suburbs of Munich using time-series from optical Sentinel data. Using a random forest classifier as the backbone, experiments were designed to analyze the impact of the segmentation scale, features (including spectral and temporal features), categories, frequency, and acquisition timing of optical satellite images. Based on our analyses, the following findings are reported: (1) Optical Sentinel images acquired over four seasons can make a significant contribution to the classification of agricultural areas, even though this contribution varies between spectral bands for the same period. (2) The use of time-series data alleviates the issue of identifying the “optimal” segmentation scale. The finding of this study can provide a more comprehensive understanding of the effects of classification uncertainty on object-based dense multi-temporal image classification.

View-independent representation with frame interpolation method for skeleton-based human action recognition

Human action recognition is an important branch of computer vision science. It is a challenging task based on skeletal data because of joints’ complex spatiotemporal information. In this work, we propose a method for action recognition, which consists of three parts: view-independent representation, frame interpolation, and combined model. First, the action sequence becomes view-independent representations independent of the view. Second, when judgment conditions are met, differentiated frame interpolations are used to expand the temporal dimensional information. Then, a combined model is adopted to extract these representation features and classify actions. Experimental results on two multi-view benchmark datasets Northwestern-UCLA and NTU RGB+D demonstrate the effectiveness of our complete method. Although using only one type of action feature and a simple architecture combined model, our complete method still outperforms most of the referential state-of-the-art methods and has strong robustness.

Panoramic Video Quality Assessment Based on Non-Local Spherical CNN

Panoramic video and stereoscopic panoramic video are essential carriers of virtual reality content, so it is very crucial to establish their quality assessment models for the standardization of virtual reality industry. However, it is very challenging to evaluate the quality of the panoramic video at present. One reason is that the spatial information of the panoramic video is warped due to the projection process, and the conventional video quality assessment (VQA) method is difficult to deal with this problem. Another reason is that the traditional VQA method is problematic to capture the complex global time information in the panoramic video. In response to the above questions, this paper presents an end-to-end neural network model to evaluate the quality of panoramic video and stereoscopic panoramic video. Compared to other panoramic video quality assessment methods, our proposed method combines spherical convolutional neural networks (CNN) and non-local neural networks, which can effectively extract complex spatiotemporal information of the panoramic video. We evaluate the method in two databases, VRQ-TJU and VR-VQA48. Experiments show the effectiveness of different modules in our method, and our method outperforms state-of-the-art other related methods.

Supervised learning in spiking neural networks: A review of algorithms and evaluations

As a new brain-inspired computational model of the artificial neural network, a spiking neural network encodes and processes neural information through precisely timed spike trains. Spiking neural networks are composed of biologically plausible spiking neurons, which have become suitable tools for processing complex temporal or spatiotemporal information. However, because of their intricately discontinuous and implicit nonlinear mechanisms, the formulation of efficient supervised learning algorithms for spiking neural networks is difficult, and has become an important problem in this research field. This article presents a comprehensive review of supervised learning algorithms for spiking neural networks and evaluates them qualitatively and quantitatively. First, a comparison between spiking neural networks and traditional artificial neural networks is provided. The general framework and some related theories of supervised learning for spiking neural networks are then introduced. Furthermore, the state-of-the-art supervised learning algorithms in recent years are reviewed from the perspectives of applicability to spiking neural network architecture and the inherent mechanisms of supervised learning algorithms. A performance comparison of spike train learning of some representative algorithms is also made. In addition, we provide five qualitative performance evaluation criteria for supervised learning algorithms for spiking neural networks and further present a new taxonomy for supervised learning algorithms depending on these five performance evaluation criteria. Finally, some future research directions in this research field are outlined.

Genetic dissection of rod and cone pathways mediating light responses and receptive fields of ganglion cells in the mouse retina

Retinal ganglion cells (GCs) are important visual neurons which carry complex spatiotemporal information from the retina to higher visual centers in the brain. By taking advantage of pathway-specific knockout/mutant mice and multi-electrode array (MEA) recording techniques, we analyze contributions of rod and cone pathways to responsiveness, kinetics and receptive field profiles of GCs under scotopic and photopic conditions. Our data suggest: (1) Scotopic responses of some GCs require all three rod pathways, some require only the secondary and tertiary rod pathways, and others require only the tertiary rod pathway. (2) There are more responsive GCs in photopic conditions than responsive GCs in scotopic conditions. (3) Gap junctions slow down GCs’ scotopic light responses and increase GCs’ ratio of antagonistic to center inputs. (4) Cone pathways do not affect the kinetics but alter the ratio of antagonistic to center inputs of scotopic GC responses, and they speed up GCs photopic responses and alter the ratio of GCs’ antagonistic to center synaptic inputs and receptive field profiles. (5) Rod bipolar cells shorten response latency of ON GCs and increase the ratio of GCs’ antagonistic to center synaptic inputs. (6) Light adaptation speeds up GCs’ temporal processing and tunes GC photopic responses to higher frequencies, and the tertiary rod pathway plays a significant role in adaptation-induced TTP changes in some GCs. (7) GC RF center sizes are partially mediated by AIIACs and GC-GC coupling. (8) Connexin36 gap junctions and cone pathways alter synaptic circuits underlying antagonistic surround inputs to GCs in photopic conditions.

Complex temporal patterns processing by a neural mass model of a cortical column.

It is well known that neuronal networks are capable of transmitting complex spatiotemporal information in the form of precise sequences of neuronal discharges characterized by recurrent patterns. At the same time, the synchronized activity of large ensembles produces local field potentials that propagate through highly dynamic oscillatory waves, such that, at the whole brain scale, complex spatiotemporal dynamics of electroencephalographic (EEG) signals may be associated to sensorimotor decision making processes. Despite these experimental evidences, the link between highly temporally organized input patterns and EEG waves has not been studied in detail. Here, we use a neural mass model to investigate to what extent precise temporal information, carried by deterministic nonlinear attractor mappings, is filtered and transformed into fluctuations in phase, frequency and amplitude of oscillatory brain activity. The phase shift that we observe, when we drive the neural mass model with specific chaotic inputs, shows that the local field potential amplitude peak appears in less than one full cycle, thus allowing traveling waves to encode temporal information. After converting phase and amplitude changes obtained into point processes, we quantify input-output similarity following a threshold-filtering algorithm onto the amplitude wave peaks. Our analysis shows that the neural mass model has the capacity for gating the input signal and propagate selected temporal features of that signal. Finally, we discuss the effect of local excitatory/inhibitory balance on these results and how excitability in cortical columns, controlled by neuromodulatory innervation of the cerebral cortex, may contribute to set a fine tuning and gating of the information fed to the cortex.

Rethinking multiscale cardiac electrophysiology with machine learning and predictive modelling

We review some of the latest approaches to analysing cardiac electrophysiology data using machine learning and predictive modelling. Cardiac arrhythmias, particularly atrial fibrillation, are a major global healthcare challenge. Treatment is often through catheter ablation, which involves the targeted localised destruction of regions of the myocardium responsible for initiating or perpetuating the arrhythmia. Ablation targets are either anatomically defined, or identified based on their functional properties as determined through the analysis of contact intracardiac electrograms acquired with increasing spatial density by modern electroanatomic mapping systems. While numerous quantitative approaches have been investigated over the past decades for identifying these critical curative sites, few have provided a reliable and reproducible advance in success rates. Machine learning techniques, including recent deep-learning approaches, offer a potential route to gaining new insight from this wealth of highly complex spatio-temporal information that existing methods struggle to analyse. Coupled with predictive modelling, these techniques offer exciting opportunities to advance the field and produce more accurate diagnoses and robust personalised treatment. We outline some of these methods and illustrate their use in making predictions from the contact electrogram and augmenting predictive modelling tools, both by more rapidly predicting future states of the system and by inferring the parameters of these models from experimental observations.

A New Supervised Learning Algorithm Based on Genetic Inheritance for Spiking Neural Networks

Spiking neural networks (SNNs) can perform complex spatio-temporal information computations in precise temporal coding. These networks differ from previous models in that spiking neurons convey information by time rather than rate of spikes. Most existing training algorithms are based on gradient descent with inherent defects, such as local optimum and over-fitting. In this paper, we investigate the performance of the Genetic Algorithm Involving Mechanism of Simulated Annealing, as a supervised training algorithm for SNNs. The key idea is to adopt global search, which effectively avoid local optima and over-fitting. According to the experiment results, this approach has higher accuracy than other learning algorithms on well-known classification problems.

God Is a Porcupine—Brain, Consciousness and Spacetime Physics

Whereas the human body requires a vast numbers of atoms to maintain its intricate anatomical functions, we assert that the human brain requires “something extra” to carry out its higher mental and emotional functions. Recently, neuroscientists are beginning to suspect brain cells are not fast enough, or intricate enough, to correlate complex spatiotemporal information into cognitive understanding. They conclude that spacetime fields may be necessary to assist the brain during neurological processing—in much the same way magnetic and electric fields are essential for the propagation of light. This “something extra,” we argue, is spacetime itself—where structures in the brain, called facilitators (somewhat like Descartes pineal gland), have evolved biologically in such a way, so as to be able to store and retrieve spacetime quanta for the formation and generation of consciousness and memory. In this way, cognition is not a thing complete. Rather it is emergent, and accumulates as discretized spacetime quanta in the brain so rapidly, we perceive our own awareness to be continuous, events spontaneous. In this paper, we consider spacetime to be a field (like all quantum fields), which can be excited into quanta particles called gravitons. We then apply this quanta excitation to help explain the brain’s cognitive processes. If the brain has indeed evolved to interact with discretized spacetime, then with the advent of improved functional imaging equipment, we might be able to map detailed correlations between neural processes, conscious experience and spacetime. In so doing, it might be possible to learn more about the fundamental workings of spacetime itself.