Spatiotemporal Memory Research Articles

GMA: An Improved Framework of Radar Extrapolation Based on Spatiotemporal Sequence Neural Network

AbstractMost previous current spatiotemporal sequence neural networks used for radar extrapolation have difficulties in learning long‐term spatiotemporal memories. This is because the spatiotemporal sequential neural networks only use the information from the previous time step node to update the prediction state, and the networks tend to rely on the convolution layers to capture the spatiotemporal features, which are local and inefficient. In order to capture the long‐term temporal characteristics and local abrupt spatial characteristics of radar echo sequence, we propose a new framework, Global Memory Attention (GMA), which has two contributions. The first is that we establish a global information flow between calculation units, extract the key historical memory from the global information flow, calculate the correlation between the historical key memory and the current time frame prediction state, and determine how much historical memory participates in the prediction state update. It alleviates the current network's difficulty in learning the long‐term spatial and temporal characteristics of radar echo sequences and the problem of short‐term dependence, and reduces the interference of image noise in the process of radar extrapolation prediction. The second is that the GMA module is a flexible additional module that can be applied to most radar extrapolation algorithms based on timing networks. GMA has reached the highest level on radar extrapolation tasks, and we have provided ablation studies to verify the effectiveness of GMA.

Episodic Events as Spatiotemporal Memory: The Sequence of Information in the Episodic Buffer of Working Memory for Language Comprehension.

Memory and language are the two higher-order cognitive abilities intertwined for communication and other cognitive skills. Memory is the storage capacity of all the information we perceive. Where the sensory memory perceives the stimuli, the working memory actively stores the information and passes it to the long-term memory. However, there is a question that how is the continuous perception of stimuli transformed into meaningful information and organized for proper execution and retrieval from the memory? This paper focuses on the episodic memory that perceives information that is spatial and temporal based on our everyday experiences. Though the spatiotemporal information we receive is continuous; the episodic memory arranges the information as to episodes in the working memory before the information is stored for a longer period. The episodic buffer is one of the components of the working memory model which holds the episodic memory that is organized concerning time. To this point, the paper tries to understand the working of the episodic buffer in maintaining the episodic memory and also about the process of episodic events into meaningful units. Further, the paper also concentrates on the hippocampus which is considered to be the location of the episodic buffer.

Who are the 'silent spreaders'?: contact tracing in spatio-temporal memory models.

The COVID-19 epidemic has swept the world for over two years. However, a large number of infectious asymptomatic COVID-19 cases (ACCs) are still making the breaking up of the transmission chains very difficult. Efforts by epidemiological researchers in many countries have thrown light on the clinical features of ACCs, but there is still a lack of practical approaches to detect ACCs so as to help contain the pandemic. To address the issue of ACCs, this paper presents a neural network model called Spatio-Temporal Episodic Memory for COVID-19 (STEM-COVID) to identify ACCs from contact tracing data. Based on the fusion Adaptive Resonance Theory (ART), the model encodes a collective spatio-temporal episodic memory of individuals and incorporates an effective mechanism of parallel searches for ACCs. Specifically, the episodic traces of the identified positive cases are used to map out the episodic traces of suspected ACCs using a weighted evidence pooling method. To evaluate the efficacy of STEM-COVID, a realistic agent-based simulation model for COVID-19 spreading is implemented based on the recent epidemiological findings on ACCs. The experiments based on rigorous simulation scenarios, manifesting the current situation of COVID-19 spread, show that the STEM-COVID model with weighted evidence pooling has a higher level of accuracy and efficiency for identifying ACCs when compared with several baselines. Moreover, the model displays strong robustness against noisy data and different ACC proportions, which partially reflects the effect of breakthrough infections after vaccination on the virus transmission.

Memory propagation in barrier discharge at water interface: suspected Markov states and spatiotemporal memory effects

Barrier discharges are known for strong memory effects which shape their stochastic properties. We study memory propagation in a surface barrier discharge operated at the water interface in atmospheric pressure air using electrical measurements and synchronized intensified CCD imaging. Comprehensive electrical data sets of seemingly chaotic behaviour are evaluated from thousands of subsequent periods recorded with high temporal resolution and large dynamic range. We confirm known memory effects and identify new ones both in between the half-periods as well as between subsequent pulses within one half-period. We find two memoryless states for the first discharges in the positive polarity which are defined by the presence/absence of photoemission-induced collective behaviour. Given the determined probability for entering one of these two suspected Markov states, the system bifurcates and follows one of the two subsequent non-Markovian pathways. The identification of the collective behaviour in these pathways enables us to recognize the spatially resolved property of the system and to determine its probability of occurrence, both solely from the zero-dimensional electrical characterisation.

Asynchronous Spatio-Temporal Memory Network for Continuous Event-Based Object Detection.

Event cameras, offering extremely high temporal resolution and high dynamic range, have brought a new perspective to addressing common object detection challenges (e.g., motion blur and low light). However, how to learn a better spatio-temporal representation and exploit rich temporal cues from asynchronous events for object detection still remains an open issue. To address this problem, we propose a novel asynchronous spatio-temporal memory network (ASTMNet) that directly consumes asynchronous events instead of event images prior to processing, which can well detect objects in a continuous manner. Technically, ASTMNet learns an asynchronous attention embedding from the continuous event stream by adopting an adaptive temporal sampling strategy and a temporal attention convolutional module. Besides, a spatio-temporal memory module is designed to exploit rich temporal cues via a lightweight yet efficient inter-weaved recurrent-convolutional architecture. Empirically, it shows that our approach outperforms the state-of-the-art methods using the feed-forward frame-based detectors on three datasets by a large margin (i.e., 7.6% in the KITTI Simulated Dataset, 10.8% in the Gen1 Automotive Dataset, and 10.5% in the 1Mpx Detection Dataset). The results demonstrate that event cameras can perform robust object detection even in cases where conventional cameras fail, e.g., fast motion and challenging light conditions.

Taking a walk through time: aversive memory re-experiencing may be linked to spatio-temporal distance

ABSTRACT Background Individuals with posttraumatic stress disorder (PTSD) often report intrusive memories that appear to lack the appropriate spatio-temporal context. Objective We examined whether focusing on the spatio-temporal context of aversive autobiographical memories reduces negative emotions, appraisals, and re-experiencing symptoms. Methods We recruited 109 healthy adults and had them rate emotionality, vividness, and re-experiencing of an aversive autobiographical memory. Furthermore, we assessed automatic associations of idiosyncratic memory triggers with the concepts ‘past’ vs. ‘now’, and self-reported sense of memory closure and distancing. To manipulate spatio-temporal memory (re-)organization, the experimental group (n = 53) performed a lifeline exercise in virtual reality (VR), where participants symbolically placed memory triggers along a path representing their own personal life story. The control group (n = 56) completed a non-personalized VR task. Results We found a marked decrease in negative emotions, negative appraisals, and re-experiencing in the following week, but on average, the lifeline exercise was not superior to the control condition. However, those in the lifeline group with stronger trigger-past associations subsequently exhibited a more pronounced reduction in re-experiencing. Also, participants with a higher subjective sense of memory distancing reported less re-experiencing. Conclusions The findings lend tentative support for theoretical assumptions about PTSD, but the potential causal role of automatic associations with spatio-temporal information remains to be clarified.

Effects of a Cc2d1a/Freud-1 Knockdown in the Hippocampus on Behavior, the Serotonin System, and BDNF.

The serotonin 5-HT1A receptor is one of the most abundant and widely distributed brain serotonin (5-HT) receptors that play a major role in the modulation of emotions and behavior. The 5-HT1A receptor gene (Htr1a) is under the control of transcription factor Freud-1 (also known as Cc2d1a/Freud-1). Here, using adeno-associated virus (AAV) constructs in vivo, we investigated effects of a Cc2d1a/Freud-1 knockdown in the hippocampus of C57BL/6J mice on behavior, the brain 5-HT system, and brain-derived neurotrophic factor (BDNF). AAV particles carrying the pAAV_H1-2_shRNA-Freud-1_Syn_EGFP plasmid encoding a short-hairpin RNA targeting mouse Cc2d1a/Freud-1 mRNA had an antidepressant effect in the forced swim test 5 weeks after virus injection. The knockdown impaired spatiotemporal memory as assessed in the Morris water maze. pAAV_H1-2_shRNA-Freud-1_Syn_EGFP decreased Cc2d1a/Freud-1 mRNA and protein levels. Furthermore, the Cc2d1a/Freud-1 knockdown upregulated 5-HT and its metabolite 5-hydroxyindoleacetic acid but not their ratio. The Cc2d1a/Freud-1 knockdown failed to increase mRNA and protein levels of Htr1a but diminished a 5-HT1A receptor functional response. Meanwhile, the Cc2d1a/Freud-1 knockdown reduced Creb mRNA expression and CREB phosphorylation and upregulated cFos mRNA. The knockdown enhanced the expression of a BDNF precursor (proBDNF protein), which is known to play a crucial part in neuroplasticity. Our data indicate that transcription factor Cc2d1a/Freud-1 is implicated in the pathogenesis of depressive disorders not only via the 5-HT1A receptor and transcription factor CREB but also through an influence on BDNF.

A 64K-Neuron 64M-1b-Synapse 2.64pJ/SOP Neuromorphic Chip With All Memory on Chip for Spike-Based Models in 65nm CMOS

Generally, computer-science-oriented artificial neural networks (ANNs) and neuroscience-oriented spiking neural networks (SNNs) are two main approaches to develop brain-spired non von Neumann computing systems. The goal of exploring complex artificial intelligence (AI) systems demands general neuromorphic hardware platforms compatible with both of them. However, as a result of the obvious differences in their fundamental mathematical expression and coding scheme, many neuromorphic platforms or deep neural network (DNN) accelerators accommodate only one of them. This brief presents a reconfigurable scalable neuromorphic chip based on digital leaky integrate-and-fire (LIF) neuron model targeting low-cost large-scale systems. By unifying ANN and SNN paradigms within a LIF neuron framework with point-to-point (P2P) communication, the chip can accommodate most popular neural networks. The chip adopts distributed on-chip memory architecture with a capacity of 64K neurons and 64M synapses. It achieves a peak throughput of 12.29 GSOP/s at 1.2 V, 192MHz and peak energy efficiency of 2.64 pJ/SOP at 890 mV, 24MHz. The results of implementations of a spike-based spatio-temporal memory model and ternary-weight event-based convolutional neural networks (CNNs) demonstrate outstanding compatibility of the chip.

Rapid coordination of effective learning by the human hippocampus.

Although the human hippocampus is necessary for long-term memory, controversial findings suggest that it may also support short-term memory in the service of guiding effective behaviors during learning. We tested the counterintuitive theory that the hippocampus contributes to long-term memory through remarkably short-term processing, as reflected in eye movements during scene encoding. While viewing scenes for the first time, short-term retrieval operative within the episode over only hundreds of milliseconds was indicated by a specific eye-movement pattern, which was effective in that it enhanced spatiotemporal memory formation. This viewing pattern was predicted by hippocampal theta oscillations recorded from depth electrodes and by shifts toward top-down influence of hippocampal theta on activity within visual perception and attention networks. The hippocampus thus supports short-term memory processing that coordinates behavior in the service of effective spatiotemporal learning.

Modeling COVID-19 spreading dynamics and unemployment rate evolution in rural and urban counties of Alabama and New York using fractional derivative models

The COVID-19 pandemic has been affecting the United States (U.S.) since the outbreak documented on 2/29/2020, and understanding its dynamics is critical for pandemic mitigation and economic recovery. This study proposed and applied novel time fractional derivative models (FDMs) to quantify the spatiotemporal dynamics of the COVID-19 pandemic spreading in the states of Alabama and New York, U.S., two states with quite different population compositions, urbanization, and industry structures. Model applications revealed that the pandemic evolving in the two states exhibited an overall similar time-dependent trend with subtle differences in propagation rates. Alabama may have more inter-county communications in rural areas than urban areas, while the opposite may be true for the New York State. Further analysis using the space FDM showed that the COVID-19 pandemic spread in rural/urban areas of the two states by following the tempered stable density distributions with different indexes, while the number of the state’s pandemic epicenters affected the pattern of the COVID-19 pandemic spreading in space. Finally, applications of a novel time FDM revealed that the evolution of the economy, represented by the weekly unemployment insurance claims in the two states, exhibited different spreading and recovery rates, most likely due to their different exposures and responses to the pandemic. Therefore, COVID-19 spreading dynamics exhibited strong and subtly different spatiotemporal memories in rural and urban areas in the Alabama and New York States, motivating the application of FDMs.

Spatiotemporal Memories for Missing Samples Reconstruction.

We develop a systematic theory to reconstruct missing samples in a time series using a spatiotemporal memory based on artificial neural networks. The Markov order of the input process is learned and subsequently used for learning temporal correlations from data difference sequences. We enforce the Lipschitz continuity criterion in our algorithm, leading to a regularized optimization framework for learning. The performance of the algorithm is analyzed using both theory and simulations. The efficacy of the technique is tested on synthetic and real life data sets. Our technique is analytic and uses nonlinear feedback within an optimization setup. Simulation results show that the algorithm presented in this article significantly outperforms the state-of-the-art algorithms for missing samples reconstruction with the same data set and similar training conditions.

Spatiotemporal Metasurfaces: Spatiotemporal Terahertz Metasurfaces for Ultrafast All‐Optical Switching with Electric‐Triggered Bistability (Laser Photonics Rev. 15(3)/2021)

In article number 2000456, Tian Jiang and co-workers propose a novel spatiotemporal terahertz meta-device, which equips temporally ultrafast all-optical terahertz switching with spatially selective nonvolatile resonances. By leveraging both degrees of freedom in space and time domains, this work directly promotes the emergence of highly functional spatiotemporal memory metadevices.

Flexibility as a marker of early cognitive decline in humanized apolipoprotein E ε4 (ApoE4) mice

To test the hypothesis that ApoE4 may be involved in cognitive deficits associated with aging, we investigated the impact of APOE4 status and aging on the flexibility and memory components of spatial learning in mice. Young adult (6 months) and middle-aged (14 months) ApoE4, ApoE3 and C57BL/6 male mice were tested for flexibility in an aquatic Y-maze, and for spatio-temporal memory acquisition in the Starmaze. Our results revealed a flexibility deficit of the 6-month-old ApoE4 mice compared to controls. However, this deficit was not associated with spatio-temporal memory deficit at the same age. Importantly, the ApoE4 flexibility deficit did not increase with age, nor turn into memory deficit, or was able to predict individual variations of memory performance at 14 months. By contrast, control ApoE3 mice showed a decline of flexibility at 14 months resulting in performance similar to that of ApoE4. Overall, our results suggest that ApoE4 could be associated with an acceleration of the flexibility decrease otherwise observed in normal aging.

SA-JSTN: Self-Attention Joint Spatiotemporal Network for Temperature Forecasting

The rapid development of remote sensing technology has brought abundant data support for deep learning based temperature forecasting research. However, recently proposed methods usually focus on the temporal relationship among temperature observation information, whereas ignore the spatial positions of different regions. Motivated by the observation that adjacent regions usually present similar temperature trends, in this article, we consider the temperature forecasting as a spatiotemporal sequence prediction problem, and propose a new deep learning model for temperature forecasting, self-attention joint spatiotemporal network (SA-JSTN), which simultaneously captures the spatiotemporal interdependency information. The kernel component of the SA-JSTN is a newly developed spatiotemporal memory (STM) unit, which describes the temporal and spatial models via a unified memory cell. STM is constructed based on the units of the convolutional long short-term memory (ConvLSTM). Instead of using simple convolutions for spatial information extraction, in STM, we improve ConvLSTM by a self-attention module, which has significantly enhanced the global spatial information representation ability of our proposed network. Compared with other deep learning based temperature forecasting methods, SA-JSTN is able to integrate the global spatial correlation into the temperature series prediction problem, and thus present better performance especially in short-term prediction. We have conducted comparison experiments on two typical temperature datasets to validate the effectiveness of our proposed method.

Rehabilitation of elderly patients at risk of falling: the value of psychophysiological parameters and cognitive-motor training using virtual reality

Due to the slowing of the neurodynamic and cognitive processes, as well as changes in the musculoskeletal system that accompany aging, attention, reaction, and movement coordination are impaired in elderly patients. Decreased overall brain adaptability leads to an increased risk of falls and disability, thus reducing the age of active aging. According to the World Health Organization, 37.3 million falls occur annually that are not fatal but have serious consequences requiring medical attention. These falls are most common among people over the age of 65 years. An objective assessment of psychophysiological characteristics identified a correlation between the duration of simple and complex reactions and the risk of falls and served as a tool for evaluating the effectiveness of balance retraining. Studies have shown that cognitive-motor training improves postural stability and functional performance in daily life. This type of training is widely used to rehabilitate patients with balance disorders, and virtual reality systems are increasingly being used in its implementation. There is a theory that the virtual environment can improve responses to rapid environmental changes, as well as modulate various characteristics of attention, spatiotemporal memory, and planning, which favorably affects postural function. This review describes the changes in psychophysiological parameters in the elderly, as well as balance retraining techniques using cognitive-motor training, including the use of virtual reality technology.

Convolutional Recurrent Reconstructive Network for Spatiotemporal Anomaly Detection in Solder Paste Inspection.

Surface mount technology (SMT) is a process for producing printed-circuit boards. The solder paste printer (SPP), package mounter, and solder reflow oven are used for SMT. The board on which the solder paste is deposited from the SPP is monitored by the solder paste inspector (SPI). If SPP malfunctions due to the printer defects, the SPP produces defective products, and then abnormal patterns are detected by SPI. In this article, we propose a convolutional recurrent reconstructive network (CRRN), which decomposes the anomaly patterns generated by the printer defects, from SPI data. CRRN learns only normal data and detects the anomaly pattern through the reconstruction error. CRRN consists of a spatial encoder (S-Encoder), a spatiotemporal encoder and decoder (ST-Encoder-Decoder), and a spatial decoder (S-Decoder). The ST-Encoder-Decoder consists of multiple convolutional spatiotemporal memories (CSTMs) with a spatiotemporal attention (ST-Attention) mechanism. CSTM is developed to extract spatiotemporal patterns efficiently. In addition, an ST-Attention mechanism is designed to facilitate transmitting information from the spatiotemporal encoder to the spatiotemporal decoder, which can solve the long-term dependency problem. We demonstrate that the proposed CRRN outperforms the other conventional models in anomaly detection. Moreover, we show the discriminative power of the anomaly map decomposed by the proposed CRRN through the printer defect classification.

FORAGING PERFORMANCE OF HONEYBEE (APIS MELLIFERA) AFFECTED BY FOOD RICHNESS AND EXPERIENCE

For honeybees (Apis mellifera), food richness and experience put significant impacts in making foraging decisions. Bees that trace food-rich source they start establishing spatiotemporal memories, which assist them to revisit particular site on following days. There were two aims of conducting present research. Our first aim was to explore whether or not the different levels of food richness or rewards (10%, 30%, and 50% sugar solution) at a feeding source affect the number of forager bees for their visitation once the conditions are switched to unrewarding. While our second aim was to investigate that how the previous experiences affect bees for their foraging duration. For our first aim, we hypothesized that the switching of condition from high to low level of food richness at experienced feeding site (FA) will result gradual reduction in number of bees on particular food source. Moreover, concerning to our second aim, we predicted that the foraging efficiency of bees will increase with the experiences of the previous visits. Regarding our hypothesis, results show that the bees persist visiting those feeding sites that offer food-rich sources. However, the decrease in food richness consequence gradual decline in number of bees at feeding site, but the most remarkable thing is that they persist visiting their foraging trips for several days even if they are unrewarded with food-rich sources. These results also affirm our prediction that foraging efficiency of bees in term of trip duration will increase with the experiences of previous visits. In conclusion, bees exhibit considerable attachment with experienced feeding site that stop providing food anymore, and foraging duration of bees decreases with the experience.

Experimental Effects of Acute High-Intensity Resistance Exercise on Episodic Memory Function: Consideration for Post-Exercise Recovery Period.

BackgroundThe present experiments evaluated the effects of acute high-intensity resistance exercise on episodic memory.MethodsTwo experiments were conducted. For Experiment 1, participants (N = 40; Mage = 21.0 years) were randomized into one of two groups, including an experimental exercise group and a control group (seated for 20 min). The experimental group engaged in an acute bout of resistance exercises (circuit style exercises) for 15 minutes, followed by a 5-min recovery period. Memory function was subsequently assessed using a multiple trial (immediate and delay), word-list episodic memory task (Rey Auditory Verbal Learning Test, RAVLT), and then followed by a comprehensive, computerized assessment of episodic memory (Treasure Hunt task, THT). The THT involved a spatio-temporal assessment of what, where, and when components of episodic memory. Experiment 2 evaluated if altering the recovery period would influence the potential negative effects of high-intensity resistance exercise on episodic memory function. For Experiment 2, participants (N = 51) were randomized into the same acute resistance exercise protocol but either with a 10-min recovery period, 20-min recovery period, or a control group.ResultsFor Experiment 1, for RAVLT, the exercise group performed worse (Fgroup × time = 3.7, p = .001, η2p = .09). Across nearly all THT outcomes, the exercise group had worse spatio-temporal memory than the control group. These results suggest that high-intensity resistance exercise (with a 5-min recovery) may have a detrimental effect on episodic memory function. For Experiment 2, for RAVLT, the exercise with 10-min recovery group performed better (Fgroup × time = 3.1, p = .04, η2p = .11). Unlike Experiment 1, exercise did not impair spatio-temporal memory, with the 20-min exercise recovery group having the best “where” component of episodic memory.ConclusionCollectively, the results from these two experiments suggest that acute high-intensity resistance exercise may impair episodic memory when a short exercise recovery period (e.g., 5-min) is employed, but with a longer recovery period (10+ min), acute high-intensity resistance exercise may, potentially, enhance episodic memory.

ХРОНОТОП – СЕМИОТИКА И БУДУЩЕЕ ГОРОДА

In this paper author presents an interdisciplinary interaction betweeen semiotics and chronotopy. The paper refers to the great cities of Renaissance and the Russian cities like Saint Petersburg. As M. M. Bakhtin formulated, "genre is cultural memory". According to the author, the structural and spatiotemporal memory lies in the core not only of artistic works, but of urban structurestoo.As an instruments of structural and semiotic analysis of city, the terms of chronotope and chronotopy were coined. The followers of M. M. Bakhtin, the structuralists and the semeiologists of the Yu. M. Lotman Semiotic School now agree on this point. In 1990s, one of the founders of Russian semiotics, Yu. M. Lotman came to the conclusion that new spatiotemporal modes can crystallize and spiritually develop citizens. This concept formed the basis of the first part of the paper. The second part considers practical opportunities of semiotics and urban chronotopy in dialogue with students during classes on such humanities subjects as philosophy, global art culture, aestetics e.t.c. According to the author, urban chronotope and urban semiotics are different and complementary instruments of scientific comprehension and development of cities.

Spatio-Temporal Memory Attention for Image Captioning

Visual attention has been successfully applied in image captioning to selectively incorporate the most relevant areas to the language generation procedure. However, the attention in current image captioning methods is only guided by the hidden state of language model, e.g. LSTM (Long-Short Term Memory), indirectly and implicitly, and thus the attended areas are weakly relevant at different time steps. Besides the spatial relationship of attention areas, the temporal relationship in attention is crucial for image captioning according to the attention transmission mechanism of human vision. In this paper, we propose a new spatio-temporal memory attention (STMA) model to learn the spatio-temporal relationship in attention for image captioning. The STMA introduces the memory mechanism to the attention model through a tailored LSTM, where the new cell is used to memorize and propagate the attention information, and the output gate is used to generate attention weights. The attention in STMA transmits with memory adaptively and dependently, which builds strong temporal connections of attentions and learns the spatio-temporal relationship of attended areas simultaneously. Besides, the proposed STMA is flexible to combine with attention-based image captioning frameworks. Experiments on MS COCO dataset demonstrate the superiority of the proposed STMA model in exploring the spatio-temporal relationship in attention and improving the current attention-based image captioning.