Spatiotemporal Signal Research Articles

Space-time modeling of cascading hazards: Chaining wildfires, rainfall and landslide events through machine learning

The current study sets out to explore yearly landslide susceptibility dynamics on slopes regularly affected by fires. To do so, two yearly inventories have been generated, one for the landslides and one for the wildfires, for an area of approximately 2 km2 and for a period of 24 years. It is important to stress that space–time data-driven models employed for susceptibility assessment are relatively new, and their application so far has mostly linked landslide occurrences to the precipitation trigger and the standard morphometric characteristics of the landscape at hand. Here we also consider an additional element of disturbance to the slope equilibrium, in the form of burnt areas tested from one up to three years priors to the reference landslide occurrence time. The relevance of the wildfire spatiotemporal signal is tested as part of a multi-variate modeling procedure. The results highlight at least a 10 % performance increase when these wildfire-related predictors are featured (from an average AUC of 0.75 to 0.85 in a random forest modeling framework). The associated yearly variations in the landslide occurrence probability are translated into individual maps, stressing the extent to which the standard and static definition of susceptibility does not hold especially in the context of multiple hazards and in an urban setting such as the Camaldoli hills (Naples, Italy) we chose for our test site.

Resting state of human brain measured by fMRI experiment is governed more dominantly by essential mode as a global signal rather than default mode network

Resting-state of the human brain has been described by a combination of various basis modes including the default mode network (DMN) identified by fMRI BOLD signals in human brains. Whether DMN is the most dominant representation of the resting-state has been under question. Here, we investigated the unexplored yet fundamental nature of the resting-state. In the absence of global signal regression for the analysis of brain-wide spatial activity pattern, the fMRI BOLD spatiotemporal signals during the rest were completely decomposed into time-invariant spatial-expression basis modes (SEBMs) and their time-evolution basis modes (TEBMs). Contrary to our conventional concept above, similarity clustering analysis of the SEBMs from 166 human brains revealed that the most dominant SEBM cluster is an asymmetric mode where the distribution of the sign of the components is skewed in one direction, for which we call essential mode (EM), whereas the second dominant SEBM cluster resembles the spatial pattern of DMN. Having removed the strong 1/f noise in the power spectrum of TEBMs, the genuine oscillatory behavior embedded in TEBMs of EM and DMN-like mode was uncovered around the low-frequency range below 0.2 Hz.

NOise Reduction with DIstribution Corrected (NORDIC) principal component analysis improves brain activity detection across rodent and human functional MRI contexts.

NOise Reduction with DIstribution Corrected (NORDIC) principal component analysis (PCA) has been shown to selectively suppress thermal noise and improve the temporal signal-to-noise ratio (tSNR) in human functional magnetic resonance imaging (fMRI). However, the feasibility to improve data quality for rodent fMRI using NORDIC PCA remains uncertain. NORDIC PCA may also be particularly beneficial for improving topological brain mapping, as conventional mapping requires precise spatiotemporal signals from large datasets (ideally ~1 hour acquisition) for individual representations. In this study, we evaluated the effects of NORDIC PCA compared with "Standard" processing in various rodent fMRI contexts that range from task-evoked optogenetic fMRI to resting-state fMRI. We also evaluated the effects of NORDIC PCA on human resting-state and retinotopic mapping fMRI via population receptive field (pRF) modeling. In rodent optogenetic fMRI, apart from doubling the tSNR, NORDIC PCA resulted in a larger number of activated voxels and a significant decrease in the variance of evoked brain responses without altering brain morphology. In rodent resting-state fMRI, we found that NORDIC PCA induced a nearly threefold increase in tSNR and preserved task-free relative cerebrovascular reactivity (rCVR) across cortical depth. NORDIC PCA further improved the detection of TGN020-induced aquaporin-4 inhibition on rCVR compared with Standard processing without NORDIC PCA. NORDIC PCA also increased the tSNR for both human resting-state and pRF fMRI, and for the latter also increased activation cluster sizes while retaining retinotopic organization. This suggests that NORDIC PCA preserves the spatiotemporal precision of fMRI signals needed for pRF analysis, and effectively captures small activity changes with high sensitivity. Taken together, these results broadly demonstrate the value of NORDIC PCA for the enhanced detection of neural dynamics across various rodent and human fMRI contexts. This can in turn play an important role in improving fMRI image quality and sensitivity for translational and preclinical neuroimaging research.

Tailoring Space-Time Nonlocality for Event-Based Image Processing Metasurfaces.

Analog computation with passive optical components can enhance processing speeds and reduce power consumption, recently attracting renewed interest thanks to the opportunities enabled by metasurfaces. Basic image processing tasks, such as spatial differentiation, have been recently demonstrated based on engineered nonlocalities in metasurfaces, but next-generation computational schemes require more advanced capabilities. Here, by simultaneously tailoring the nonlocal electromagnetic response of a metasurface in space and time, we demonstrate a passive ultrathin silicon-based device that performs mixed spatiotemporal differentiation of input images, realizing event-based edge detection. The metasurface performs spatial differentiation only when the input image is evolving in time, resulting in spatiotemporal image processing on subpicosecond timescales. Moreover, the metasurface design can be tailored to selectively enhance objects moving at desired speeds. Our results point towards fully passive processing of spatiotemporal signals, for highly compact neuromorphic cameras.

Investigating fish reproduction phenology and essential habitats by identifying the main spatio-temporal patterns of fish distribution

Abstract Fish spawning phenology is a major concern for conservation and fisheries management. New intensive data sources, such as GPS-based tracking data and high-resolution catch declaration data, are becoming increasingly available in the field of marine ecology. These data benefit from high spatiotemporal resolution and open new research avenues for investigating the interannual variability in fish phenology. In this paper, we demonstrate how an integrated species distribution model informed by commercial catch data combined with spatiotemporal dimension reduction methods known as empirical orthogonal functions (EOFs) can be used to synthesize spatiotemporal signals in fish reproduction phenology. Specifically, we address the following questions: (1) Can we identify seasonal spatial patterns that can be interpreted in terms of reproductive phenology and essential habitats? (2) Can we identify changes in reproductive phenology over time? (3) Are these changes related to environmental drivers? The analysis illustrates the reproductive phenology of three key commercial species in the Bay of Biscay (sole, hake, and sea bass). The EOF analysis emphasized strong seasonal spatiotemporal patterns that correspond to reproduction patterns and feeding patterns. Based on this methodology, we identified seasonal variations in the timing of reproduction, and we related these variations to sea surface temperature, a key driver of fish reproduction.

Contrast-Phys+: Unsupervised and Weakly-Supervised Video-Based Remote Physiological Measurement via Spatiotemporal Contrast.

Video-based remote physiological measurement utilizes facial videos to measure the blood volume change signal, which is also called remote photoplethysmography (rPPG). Supervised methods for rPPG measurements have been shown to achieve good performance. However, the drawback of these methods is that they require facial videos with ground truth (GT) physiological signals, which are often costly and difficult to obtain. In this paper, we propose Contrast-Phys+, a method that can be trained in both unsupervised and weakly-supervised settings. We employ a 3DCNN model to generate multiple spatiotemporal rPPG signals and incorporate prior knowledge of rPPG into a contrastive loss function. We further incorporate the GT signals into contrastive learning to adapt to partial or misaligned labels. The contrastive loss encourages rPPG/GT signals from the same video to be grouped together, while pushing those from different videos apart. We evaluate our methods on five publicly available datasets that include both RGB and Near-infrared videos. Contrast-Phys+ outperforms the state-of-the-art supervised methods, even when using partially available or misaligned GT signals, or no labels at all. Additionally, we highlight the advantages of our methods in terms of computational efficiency, noise robustness, and generalization.

Demixing fluorescence time traces transmitted by multimode fibers

Optical methods based on thin multimode fibers (MMFs) are promising tools for measuring neuronal activity in deep brain regions of freely moving mice thanks to their small diameter. However, current methods are limited: while fiber photometry provides only ensemble activity, imaging techniques using of long multimode fibers are very sensitive to bending and have not been applied to unrestrained rodents yet. Here, we demonstrate the fundamentals of a new approach using a short MMF coupled to a miniscope. In proof-of-principle in vitro experiments, we disentangled spatio-temporal fluorescence signals from multiple fluorescent sources transmitted by a thin (200 µm) and short (8 mm) MMF, using a general unconstrained non-negative matrix factorization algorithm directly on the raw video data. Furthermore, we show that low-cost open-source miniscopes have sufficient sensitivity to image the same fluorescence patterns seen in our proof-of-principle experiment, suggesting a new avenue for novel minimally invasive deep brain studies using multimode fibers in freely behaving mice.

Spatio-temporal adaptive algorithm using Hamming code based on 3D-MIMO wireless channel model

Space-time coding is an effective method for constructing communication systems while saving bandwidth, taking into account the shortage of frequency resources and power in fading channels, which is implemented based on the advantages of using adaptive antenna arrays. Currently, WiMAX radio access technology has become widespread; its use is justified in conditions of direct visibility between the base and mobile stations, but in a city with dense buildings and in the presence of organized interference, its use leads to an increase in the likelihood of bit errors. At the same time, the combination of OFDM (orthogonal frequency division multiplexing) and MIMO (spatial coding) technologies based on adaptive antenna arrays significantly reduces the probability of bit errors, and therefore increases the throughput of the communication system. The use of adaptive space-time devices at the outputs of antenna arrays makes it possible to select a signal path with maximum power and with a minimum error in receiving OFDM pilot subcarriers, even in the absence of direct visibility between the base and mobile stations and in conditions of active interference. In addition, the use of a 3D model of the communication channel will allow us to evaluate the effectiveness of the communication system model in real operating conditions. The goal of the work is to solve the problem of additionally increasing noise immunity in communication systems in conditions of numerous reflections caused by dense buildings in urban environments, using a combination of Hamming code, OFDM and an algorithm for adaptive spatio-temporal signal processing in receivers such as SISO (Singl Input – Singl Output), and MIMO (Multiply Input – Multiply Output), built on adaptive antenna arrays. The work calculated the weight coefficients of the adaptation algorithm using the theory of eigenvalues and eigenvectors of the spatial correlation matrix of signals at the outputs of the adaptive antenna arrays. Modeling of algorithms and a 3D model of the communication channel in Matlab using the Hamming code was carried out, as a result, an increase in the efficiency of error correction for the Hamming code was demonstrated, as well as a significant reduction in the probability of code errors and an increase in the performance of the communication system when using an adaptive algorithm. At the same time, the article uses the term “multipath” channel in contrast to the generally accepted, but incorrect “multibeam” channel, since multibeam can be a characteristic of an antenna system. The presented results confirm that when processing signals in a receiving system in a multipath channel with dense urban development, the use of adaptive spatiotemporal algorithms provides increased noise immunity and radio communication capacity, especially for image transmission when using unmanned aerial vehicles.

Identification and prediction method for acoustic emission and electromagnetic radiation signals of rock burst based on deep learning

With the deep development of underground rock engineering, the threat of rock burst disasters is increasing. At present, the identification and prediction of rock burst mostly rely on the experience of field staff to determine the critical value and development trend, and there is a lack of efficient and intelligent methods for the utilization of massive data. Therefore, this paper constructs a rock burst signal recognition and prediction model based on deep learning methods to solve the above problems. In this paper, the acoustic emission (AE) and electromagnetic radiation (EMR) data of the site are first marked and input into the long-short-term memory-fully connected neural network model to realize the identification of rock burst danger signals. Then, the graph data of the AE and EMR sensor monitoring networks are constructed and input into the spatiotemporal graph convolutional network signal prediction model to predict future monitoring data. Finally, this paper uses the same dataset to compare and analyze several other commonly used deep learning models. The results show that the model constructed in this paper has the best performance in the identification and prediction of AE and EMR signals with rockburst risk. This study can provide theoretical reference for intelligent monitoring and early warning of rock burst in underground rock engineering.

Multimodal identification of the mouse brain using simultaneous Ca 2+ imaging and fMRI.

Individual differences in neuroimaging are of interest to clinical and cognitive neuroscientists based on their potential for guiding the personalized treatment of various heterogeneous neurological conditions and diseases. Despite many advantages, the workhorse in this arena, BOLD (blood-oxygen-level-dependent) functional magnetic resonance imaging (fMRI) suffers from low spatiotemporal resolution and specificity as well as a propensity for noise and spurious signal corruption. To better understand individual differences in BOLD-fMRI data, we can use animal models where fMRI, alongside complementary but more invasive contrasts, can be accessed. Here, we apply simultaneous wide-field fluorescence calcium imaging and BOLD-fMRI in mice to interrogate individual differences using a connectome-based identification framework adopted from the human fMRI literature. This approach yields high spatiotemporal resolution cell-type specific signals (here, from glia, excitatory, as well as inhibitory interneurons) from the whole cortex. We found mouse multimodal connectome- based identification to be successful and explored various features of these data.

Fast, Accurate, and Versatile Data Analysis Platform for the Quantification of Molecular Spatiotemporal Signals.

Optical recording of intricate molecular dynamics is becoming an indispensable technique for biological studies, accelerated by the development of new or improved biosensors and microscopy technology. This creates major computational challenges to extract and quantify biologically meaningful spatiotemporal patterns embedded within complex and rich data sources, many of which cannot be captured with existing methods. Here, we introduce Activity Quantification and Analysis (AQuA2), a fast, accurate, and versatile data analysis platform built upon advanced machine learning techniques. It decomposes complex live imaging-based datasets into elementary signaling events, allowing accurate and unbiased quantification of molecular activities and identification of consensus functional units. We demonstrate applications across a wide range of biosensors, cell types, organs, animal models, and imaging modalities. As exemplar findings, we show how AQuA2 identified drug-dependent interactions between neurons and astroglia, and distinct sensorimotor signal propagation patterns in the mouse spinal cord.

Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition

Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-γ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.

Spatially distinct epithelial and mesenchymal cell subsets along progressive lineage restriction in the branching embryonic mammary gland

How cells coordinate morphogenetic cues and fate specification during development remains a fundamental question in organogenesis. The mammary gland arises from multipotent stem cells (MaSCs), which are progressively replaced by unipotent progenitors by birth. However, the lack of specific markers for early fate specification has prevented the delineation of the features and spatial localization of MaSC-derived lineage-committed progenitors. Here, using single-cell RNA sequencing from E13.5 to birth, we produced an atlas of matched mouse mammary epithelium and mesenchyme and reconstructed the differentiation trajectories of MaSCs toward basal and luminal fate. We show that murine MaSCs exhibit lineage commitment just prior to the first sprouting events of mammary branching morphogenesis at E15.5. We identify early molecular markers for committed and multipotent MaSCs and define their spatial distribution within the developing tissue. Furthermore, we show that the mammary embryonic mesenchyme is composed of two spatially restricted cell populations, and that dermal mesenchyme-produced FGF10 is essential for embryonic mammary branching morphogenesis. Altogether, our data elucidate the spatiotemporal signals underlying lineage specification of multipotent MaSCs, and uncover the signals from mesenchymal cells that guide mammary branching morphogenesis.

Basal actomyosin pulses expand epithelium coordinating cell flattening and tissue elongation

Actomyosin networks constrict cell area and junctions to alter cell and tissue shape. However, during cell expansion under mechanical stress, actomyosin networks are strengthened and polarized to relax stress. Thus, cells face a conflicting situation between the enhanced actomyosin contractile properties and the expansion behaviour of the cell or tissue. To address this paradoxical situation, we study late Drosophila oogenesis and reveal an unusual epithelial expansion wave behaviour. Mechanistically, Rac1 and Rho1 integrate basal pulsatile actomyosin networks with ruffles and focal adhesions to increase and then stabilize basal area of epithelial cells allowing their flattening and elongation. This epithelial expansion behaviour bridges cell changes to oocyte growth and extension, while oocyte growth in turn deforms the epithelium to drive cell spreading. Basal pulsatile actomyosin networks exhibit non-contractile mechanics, non-linear structures and F-actin/Myosin-II spatiotemporal signal separation, implicating unreported expanding properties. Biophysical modelling incorporating these expanding properties well simulates epithelial cell expansion waves. Our work thus highlights actomyosin expanding properties as a key mechanism driving tissue morphogenesis.

Design of spatiotemporal variability index for climatic variables

Spatiotemporal signals are currently gaining wide popularity owing to their richness in terms of information content from both spatial and temporal aspects. Climatic studies involve the usage of such signals from different spatial locations to understand the climatic conditions, anomalies and for prediction of similar climatic zones. Measurement of the complex inter-relationships in the spatiotemporal processes is of vital importance in this regard. The predominant frequency components associated with annual, semi-annual, and quarterly seasonal variations exhibit interchangeability across spatial and temporal dimensions, thereby contributing significantly to the variability within the spatiotemporal region under investigation. Employing frequency domain analysis enables the examination of these crucial seasonal components, even amidst noise, provided effective frequency domain-based techniques are employed. This paper is based on the measurement of the variability of spatiotemporal climatic variables considered from different geographic zones using multivariate variational mode decomposition. The main advantage of the proposed index lies in the ability to measure the variability of the signals acquired from either spatially close or far-off locations on Earth. Also, the index harnesses the hidden frequency components in the spatiotemporal signals under test by comparing the aligned periodic components across different zones. The correctness of the proposed index is tested through exhaustive simulation studies and also against the weather measurements like the Earth skin temperature, humidity and precipitation acquired from spatially close and far-off locations over a period of 40 years.

Electrocorticogram (ECoG): Engineering Approaches and Clinical Challenges for Translational Medicine

AbstractElectrocorticogram (ECoG) is an electrophysiological signal that results from the summation of neuronal activity near the cortical surface. To record ECoG signals, the scalp and skull are surgically opened and electrodes are placed on the cortical surface, either epidurally or subdurally. Owing to its improved spatiotemporal resolution and signal quality compared with electroencephalography, it is widely used to diagnose and treat neurological disorders in clinical settings for several decades, despite the invasiveness of ECoG. Recently, ECoG is applied in research to explore brain functions and connectivity, brain‐computer interfaces, and brain‐machine interfaces. In addition to the need for ECoG in neuroscience research, ECoG devices have advanced in terms of materials, fabrication, and function to overcome the limitations of commercially available ECoG arrays. Here, the conventional use of ECoG in clinical medicine, the new applications of ECoG in basic neuroscience research, and the future challenges in translating recent developments in ECoG devices for clinical use are described.

Thin-film transistor for temporal self-adaptive reservoir computing with closed-loop architecture.

Reservoir computing is a powerful neural network-based computing paradigm for spatiotemporal signal processing. Recently, physical reservoirs have been explored based on various electronic devices with outstanding efficiency. However, the inflexible temporal dynamics of these reservoirs have posed fundamental restrictions in processing spatiotemporal signals with various timescales. Here, we fabricated thin-film transistors with controllable temporal dynamics, which can be easily tuned with electrical operation signals and showed excellent cycle-to-cycle uniformity. Based on this, we constructed a temporal adaptive reservoir capable of extracting temporal information of multiple timescales, thereby achieving improved accuracy in the human-activity-recognition task. Moreover, by leveraging the former computing output to modify the hyperparameters, we constructed a closed-loop architecture that equips the reservoir computing system with temporal self-adaptability according to the current input. The adaptability is demonstrated by accurate real-time recognition of objects moving at diverse speed levels. This work provides an approach for reservoir computing systems to achieve real-time processing of spatiotemporal signals with compound temporal characteristics.

Spatiotemporal signal space separation for regions of interest: Application for extracting neuromagnetic responses evoked by deep brain stimulation.

Magnetoencephalography (MEG) recordings are often contaminated by interference that can exceed the amplitude of physiological brain activity by several orders of magnitude. Furthermore, the activity of interference sources may spatially extend (known as source leakage) into the activity of brain signals of interest, resulting in source estimation inaccuracies. This problem is particularly apparent when using MEG to interrogate the effects of brain stimulation on large-scale cortical networks. In this technical report, we develop a novel denoising approach for suppressing the leakage of interference source activity into the activity representing a brain region of interest. This approach leverages spatial and temporal domain projectors for signal arising from prespecified anatomical regions of interest. We apply this denoising approach to reconstruct simulated evoked response topographies to deep brain stimulation (DBS) in a phantom recording. We highlight the advantages of our approach compared to the benchmark-spatiotemporal signal space separation-and show that it can more accurately reveal brain stimulation-evoked response topographies. Finally, we apply our method to MEG recordings from a single patient with Parkinson's disease, to reveal early cortical-evoked responses to DBS of the subthalamic nucleus.

Fully Memristive Elementary Motion Detectors for a Maneuver Prediction.

Insects can efficiently perform object motion detection via a specialized neural circuit, called an elementary motion detector (EMD). In contrast, conventional machine vision systems require significant computational resources for dynamic motion processing. Here, a fully memristive EMD (M-EMD) is presented that implements the Hassenstein-Reichardt (HR) correlator, a biological model of the EMD. The M-EMD consists of a simple Wye (Y) configuration, including a static resistor, a dynamic memristor, and a Mott memristor. The resistor and dynamic memristor introduce different signal delays, enabling spatio-temporal signal integration in the subsequent Mott memristor, resulting in a direction-selective response. In addition, a neuromorphic system is developed employing the M-EMDs to predict a lane-changing maneuver by vehicles on the road. The system achieved a high accuracy (> 87%) in predicting future lane-changing maneuvers on the Next Generation Simulation (NGSIM) dataset while reducing the computational cost by 92.9% compared to the conventional neuromorphic system without the M-EMD, suggesting its strong potential for edge-level computing.

Shape-changing electrode array for minimally invasive large-scale intracranial brain activity mapping

Large-scale brain activity mapping is important for understanding the neural basis of behaviour. Electrocorticograms (ECoGs) have high spatiotemporal resolution, bandwidth, and signal quality. However, the invasiveness and surgical risks of electrode array implantation limit its application scope. We developed an ultrathin, flexible shape-changing electrode array (SCEA) for large-scale ECoG mapping with minimal invasiveness. SCEAs were inserted into cortical surfaces in compressed states through small openings in the skull or dura and fully expanded to cover large cortical areas. MRI and histological studies on rats proved the minimal invasiveness of the implantation process and the high chronic biocompatibility of the SCEAs. High-quality micro-ECoG activities mapped with SCEAs from male rodent brains during seizures and canine brains during the emergence period revealed the spatiotemporal organization of different brain states with resolution and bandwidth that cannot be achieved using existing noninvasive techniques. The biocompatibility and ability to map large-scale physiological and pathological cortical activities with high spatiotemporal resolution, bandwidth, and signal quality in a minimally invasive manner offer SCEAs as a superior tool for applications ranging from fundamental brain research to brain-machine interfaces.