Spatiotemporal Modulation Research Articles

Photoswitchable Detergents for Light-Controlled Liposome Lysis and Channel Gating.

Modulation of membrane properties via photoswitchable lipids has attracted attention due to the unparalleled spatiotemporal resolution of their functional control. Beside lipids, detergents are another prominent class for selective membrane perturbations owing to their ease of handling and spontaneous insertion in lipid bilayers. Herein, we describe the synthesis and characterization of three classes of visible light-sensitive surfactants with various azobenzene tail chain lengths. The photoswitchable detergents show water-solubility and micellization as well as undergo reversible isomerization under blue-/green light illumination. We demonstrate that the light-induced structural change of azobenzene can lead to vesicle rupture, making them a tool for controlled cargo release from vehicles. Via spontaneous insertion into the plasma membrane of mammalian cells transiently transfected with MscL, we used the azobenzene-derived detergents to optically activate the transmembrane mechanosensitive channel. This led to the rapid controlled uptake of membrane-impermeable molecules. Since detergents are extensively used in biochemistry and biotechnology, we propose that the photoswitchable detergents will be useful tools for the spatiotemporal modulation of membrane properties. Additionally, our work provides a design strategy for new detergents in membrane (protein) research.

Microgels With Electrostatically Controlled Molecular Affinity to Direct Morphogenesis.

Concentration gradients of soluble signaling molecules-morphogens-determine the cellular organization in tissue development. Morphogen-releasing microgels have shown potential to recapitulate this principle in engineered tissue constructs, however, with limited control over the molecular cues in space and time. Inspired by the functionality of sulfated glycosaminoglycans (sGAGs) in morphogen signaling in vivo, a library of sGAG-based microgels is developed and designated as µGel Units to Instruct Development (µGUIDEs). Adjustment of the microgel's sGAG sulfation patterns and concentration enabled the programming of electrostatic affinities that control the release of morphogens. Based on computational analyses of molecular transport processes, µGUIDEs provided unprecedented precision in the spatiotemporal modulation of vascular endothelial growth factor (VEGF) gradients in a microgel-in-gel vasculogenesis model and kidney organoid cultures. The versatile approach offers new options for creating morphogen signaling centers to advance the understanding of tissue and organ development.

Four-phase CT lesion recognition based on multi-phase information fusion framework and spatiotemporal prediction module

Multiphase information fusion and spatiotemporal feature modeling play a crucial role in the task of four-phase CT lesion recognition. In this paper, we propose a four-phase CT lesion recognition algorithm based on multiphase information fusion framework and spatiotemporal prediction module. Specifically, the multiphase information fusion framework uses the interactive perception mechanism to realize the channel-spatial information interactive weighting between multiphase features. In the spatiotemporal prediction module, we design a 1D deep residual network to integrate multiphase feature vectors, and use the GRU architecture to model the temporal enhancement information between CT slices. In addition, we employ CT image pseudo-color processing for data augmentation and train the whole network based on a multi-task learning framework. We verify the proposed network on a four-phase CT dataset. The experimental results show that the proposed network can effectively fuse the multi-phase information and model the temporal enhancement information between CT slices, showing excellent performance in lesion recognition.

Remote detection of polluted gases based on thermal infrared spectral imaging in the field

This paper reports the new progress of long wave infrared remote sensing in the field, focusing on the implementation process of window scanning spatiotemporal modulation Fourier spectroscopic imaging technology. Utilizing the self-developed CHIPED-1 device, the spatial modulation interference was achieved through the use of a cone mirror Michelson interferometer. By combining it with a cooled long-wave infrared focal plane detector component and applying data processing steps such as acquisition, recombination, calibration, etc., high-spectral resolution imaging in long-wave infrared region was accomplished. The detection sensitivity index nesr (Noise Equivalent Spectral Radiance) of the self-developed CHIPED-1 long wave infrared hyperspectral imaging principle experimental device reaches 5.6 × 10−8w/(cm−1.sr.cm2) in single pixel，Equivalent to commercial time modulation interferometric hyperspectral imagers; It reflects the progressiveness of the technology, and leaves much room for improvement.By testing the transmittance curve of polypropylene film, the spectral response range of CHIPED-1 infrared hyperspectral imaging principle experimental device reached 11.5 μm.The article also studied the hyperspectral imaging detection method for two-dimensional distributed chemical gas VOCs, taking the detection experiments of field high-rise buildings and ether gas as examples.Under complex backgrounds and low experimental concentrations, the presence of ether vapor cannot be observed from the infrared spectrum slices at the same wave number. However, after differential spectral processing, the spatial distribution of ether vapor can be clearly seen.The hyperspectral method is applied in the field of infrared detection of organic vapor VOCs, which has many advantages over wide-band thermal imaging methods, such as high sensitivity, strong anti-interference ability, and wide recognition range.

Deep spatial-temporal information fusion dynamic graph convolutional network for traffic flow prediction

Abstract Predicting traffic flow is vital for advancing intelligent transportation systems, but achieving precise forecasts remains challenging. To deeply explore the complex and diverse dynamic spatio-temporal correlation of traffic data, this study proposes a multi-information fusion spatio-temporal dynamic graph convolution model for traffic flow prediction. In the model construction, firstly, a dynamic graph generation module is designed to fully capture the complex spatial correlation of the traffic network and model the changing spatio-temporal interaction; secondly, the spatial self-attention mechanism is used to dynamically focus on the spatial relationship between different nodes, and combined with the dynamic graph convolution network to extract the spatial dynamic correlation features of the road network to study deep spatiotemporal information; Finally, existing models rarely consider other traffic information related to traffic flow, and a spatiotemporal information fusion module is designed to enhance the ability of the model to perceive information, thereby improving the performance of traffic flow prediction models. Experiments are conducted on four real traffic datasets using three performance evaluation metrics and 18 baseline models. The results show that the model has higher prediction accuracy.

Mechanotransduction-Piloted Whole-Cell Vaccines for Spatiotemporal Modulation of Postoperative Antitumor Immunity.

Whole tumor cell vaccines hold promise by presenting a broader spectrum of autologous-origin tumor antigens to combat postoperative recurrence and metastasis. However, challenges such as intractable adjuvant modification and obscure interactions with antigen-presenting cells in the postoperative microenvironment impede their translation into effective personalized immunotherapies. In this study, we propose cancer vaccines derived from manganese oxide-immobilized resected tumor cells, featuring whole tumor antigens and adjustable stiffness to modulate interactions with antigen-presenting cells in the postoperative microenvironment. These vaccines effectively stimulate dendritic cell phagocytosis and function through sequential stiffness-mediated mechanotransduction and interferon signaling. We evaluated their efficacy using an orthotopic triple-negative breast cancer mouse model and found that combining the vaccines with radiotherapy effectively inhibits postoperative tumor recurrence and metastasis. Our study underscores the potential of utilizing mechanotransduced adjuvants alongside directly inactivated whole-cell vaccines as a universal solution for preventing postoperative tumor recurrence.

Dimensional resonance of intrinsic stimulated picosecond emission while it induces a photonic crystal and electron population oscillations in heterostructure Al<sub>x</sub>Ga<sub>1–x</sub>As–GaAs–Al<sub>x</sub>Ga<sub>1–x</sub>As

Powerful picosecond optical pumping of the GaAs heterostructure layer causes the generation of stimulated picosecond emission in it. Due to its high intensity, the emission induces a Bragg grating of the electron population in the active region of the layer, making the latter an active photonic crystal. In the emission field, the inverse population of electrons oscillates with time, which should lead to spatiotemporal modulation of the emission and this population. It has been discovered that if the distance Y between the end of the heterostructure and the center of the active medium and the geometric parameters of the indicated modulation and movement of emission in the photonic crystal satisfy certain conditions, then dimensional resonance occurs - a maximum of modulation of the dependence of the energy of emission emerging from the end on Y and on pump energy appears locally.

Graph Convolution Network Combining Spatiotemporal Self-Attention and Mutual Information for Pedestrian-Trajectory Prediction

Accurate pedestrian-trajectory prediction is significant and challenging for intelligent driving. There are two challenges in this field: most existing methods ignore pedestrian–vehicle interaction in the multiple pedestrian and vehicle scenario; and research on interaction modes and objects during the interaction process remains relatively limited in depth and scope. To tackle these challenges, a novel graph convolution network combining spatiotemporal self-attention and mutual information (STSAMI-GCN) is designed for predicting pedestrian trajectories. First, to model the spatiotemporal interaction between pedestrians and vehicles, we build a spatiotemporal graph interaction module that integrates graph convolutional networks and self-attention mechanisms. Second, to filter redundant interactions and optimize feature extraction in the spatiotemporal graph interaction module, a mutual information graph network module is designed as an auxiliary network, which can reduce information redundancy and simplify graph complexity. In addition, to address the problem of propagation and accumulation of prediction errors, we design a trajectory decoder to generate future trajectories. Finally, a suitable dataset for our research scenario is built to validate the proposed model. We also evaluate it on two publicly available datasets. Experimental results show that, in the multiple pedestrian and vehicle scenario, STSAMI-GCN can be used to predict the future trajectories of multiple pedestrians, and to predict socially acceptable future trajectories. Specifically, STSAMI-GCN achieves 0.25 and 0.27 average and final deviation error, respectively, and outperforms various mainstream methods. Furthermore, ablation studies demonstrate the effectiveness of the spatiotemporal dual self-attention mechanism and mutual information graph network in improving pedestrian-trajectory prediction accuracy.

Adversarial Spatiotemporal Contrastive Learning for Electrocardiogram Signals.

Extracting invariant representations in unlabeled electrocardiogram (ECG) signals is a challenge for deep neural networks (DNNs). Contrastive learning is a promising method for unsupervised learning. However, it should improve its robustness to noise and learn the spatiotemporal and semantic representations of categories, just like cardiologists. This article proposes a patient-level adversarial spatiotemporal contrastive learning (ASTCL) framework, which includes ECG augmentations, an adversarial module, and a spatiotemporal contrastive module. Based on the ECG noise attributes, two distinct but effective ECG augmentations, ECG noise enhancement, and ECG noise denoising, are introduced. These methods are beneficial for ASTCL to enhance the robustness of the DNN to noise. This article proposes a self-supervised task to increase the antiperturbation ability. This task is represented as a game between the discriminator and encoder in the adversarial module, which pulls the extracted representations into the shared distribution between the positive pairs to discard the perturbation representations and learn the invariant representations. The spatiotemporal contrastive module combines spatiotemporal prediction and patient discrimination to learn the spatiotemporal and semantic representations of categories. To learn category representations effectively, this article only uses patient-level positive pairs and alternately uses the predictor and the stop-gradient to avoid model collapse. To verify the effectiveness of the proposed method, various groups of experiments are conducted on four ECG benchmark datasets and one clinical dataset compared with the state-of-the-art methods. Experimental results showed that the proposed method outperforms the state-of-the-art methods.

Hierarchically structured nanofibrous scaffolds spatiotemporally mediate the osteoimmune micro-environment and promote osteogenesis for periodontitis-related alveolar bone regeneration

Periodontitis suffer from inflammation-induced destruction of periodontal tissues, resulting in the serious loss of alveolar bone. Controlling inflammation and promoting bone regeneration are two crucial aspects for periodontitis-related alveolar bone defect treatment. Herein, we developed a hierarchically structured nanofibrous scaffold with a nano-embossed sheath and a bone morphogenetic protein 2-loaded core to match the periodontitis-specific features that spatiotemporally modulated the osteoimmune environment and promoted periodontal bone regeneration. We investigated the potential of this unique scaffold to treat periodontitis-related alveolar bone defects in vivo and in vitro. The results demonstrated that the hierarchically structured scaffold effectively reduced the inflammatory levels in macrophages and enhanced the osteogenic potential of bone mesenchymal stem cells in an inflammatory microenvironment. Moreover, in vivo experiments revealed that the hierarchically structured scaffold significantly ameliorated inflammation in the periodontium and inhibited alveolar bone resorption. Notably, the hierarchically structured scaffold also exhibited a prolonged effect on promoting alveolar bone regeneration. These findings highlight the significant therapeutic potential of hierarchically structured nanofibrous scaffolds for the treatment of periodontitis, and their promising role in the field of periodontal tissue regeneration. Statement of Significance: We present a novel hierarchically structured nanofibrous scaffold of coupling topological and biomolecular signals for precise spatiotemporal modulation of the osteoimmune micro-environment. Specifically, the scaffold was engineered via coaxial electrospinning of the poly(ε-caprolactone) sheath and a BMP-2/polyvinyl alcohol core, followed by surface-directed epitaxial crystallization to generate cyclic nano-lamellar embossment on the sheath. With this unique hierarchical structure, the cyclic nano-lamellar sheath provided a direct nano-topographical cue to alleviate the osteoimmune environment, and the stepwise release of BMP-2 from the core provided a biological cue for bone regeneration. This research underscores the potential of hierarchically structured nanofibrous scaffolds as a promising therapeutic approach for periodontal tissue regeneration and highlights their role in advancing periodontal tissue engineering.

AONet: Attention network with optional activation for unsupervised video anomaly detection

AbstractAnomaly detection in video surveillance is crucial but challenging due to the rarity of irregular events and ambiguity of defining anomalies. We propose a method called AONet that utilizes a spatiotemporal module to extract spatiotemporal features efficiently, as well as a residual autoencoder equipped with an attention network for effective future frame prediction in video anomaly detection. AONet utilizes a novel activation function called OptAF that combines the strengths of the ReLU, leaky ReLU, and sigmoid functions. Furthermore, the proposed method employs a combination of robust loss functions to address various aspects of prediction errors and enhance training effectiveness. The performance of the proposed method is evaluated on three widely used benchmark datasets. The results indicate that the proposed method outperforms existing state‐of‐the‐art methods and demonstrates comparable performance, achieving area under the curve values of 97.0%, 86.9%, and 73.8% on the UCSD Ped2, CUHK Avenue, and ShanghaiTech Campus datasets, respectively. Additionally, the high speed of the proposed method enables its application to real‐time tasks.

STFM: Accurate Spatio-Temporal Fusion Model for Weather Forecasting

Meteorological prediction is crucial for various sectors, including agriculture, navigation, daily life, disaster prevention, and scientific research. However, traditional numerical weather prediction (NWP) models are constrained by their high computational resource requirements, while the accuracy of deep learning models remains suboptimal. In response to these challenges, we propose a novel deep learning-based model, the Spatiotemporal Fusion Model (STFM), designed to enhance the accuracy of meteorological predictions. Our model leverages Fifth-Generation ECMWF Reanalysis (ERA5) data and introduces two key components: a spatiotemporal encoder module and a spatiotemporal fusion module. The spatiotemporal encoder integrates the strengths of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), effectively capturing both spatial and temporal dependencies. Meanwhile, the spatiotemporal fusion module employs a dual attention mechanism, decomposing spatial attention into global static attention and channel dynamic attention. This approach ensures comprehensive extraction of spatial features from meteorological data. The combination of these modules significantly improves prediction performance. Experimental results demonstrate that STFM excels in extracting spatiotemporal features from reanalysis data, yielding predictions that closely align with observed values. In comparative studies, STFM outperformed other models, achieving a 7% improvement in ground and high-altitude temperature predictions, a 5% enhancement in the prediction of the u/v components of 10 m wind speed, and an increase in the accuracy of potential height and relative humidity predictions by 3% and 1%, respectively. This enhanced performance highlights STFM’s potential to advance the accuracy and reliability of meteorological forecasting.

Multi-area short-term load forecasting based on spatiotemporal graph neural network

Short term power load forecasting can accurately evaluate the overall power load changes and provide accurate reference for power system operation decision-making. To address the limitations of traditional load forecasting methods, which are unable to capture spatial correlations and simultaneously predict load changes in multiple areas, this paper proposes a load forecasting model based on the spatiotemporal attention convolutional mechanism. The proposed model is composed of two key components: a spatiotemporal attention module and a spatiotemporal convolution module. First, the dynamic spatiotemporal correlations between different electricity load areas are captured and analyzed by using the spatiotemporal attention mechanism. Secondly, the spatial pattern and temporal features of the load data sequence are effectively obtained by spatiotemporal convolutional layers. Finally, this paper verifies the accuracy and effectiveness of the proposed method through a real electricity consumption load dataset. The experimental results demonstrate that, in comparison to various benchmark prediction methods, the proposed method can fully explore and utilize the spatiotemporal correlation between different electricity load areas, thereby improving the accuracy of load prediction.

Spatiotemporally modulated full-polarized light emission for multiplexed optical encryption

Spatiotemporal control of full freedoms of polarized light emission is crucial in multiplexed optical computing, encryption and communication. Although recent advancements have been made in active emission or passive conversion of polarized light through solution-processed nanomaterials or metasurfaces, these design paths usually encounter limitations, such as small polarization degrees, low light utilization efficiency, limited polarization states, and lack of spatiotemporal control. Here, we addressed these challenges by integrating the spatiotemporal modulation of the LED device, the precise control and efficient polarization emission through nanomaterial assembly, and the programmable patterning/positioning using 3D printing. We achieved an extremely high degree of polarization for both linearly and circularly polarized emission from ultrathin inorganic nanowires and quantum nanorods thanks to the shear-force-induced alignment effect during the protruding of printing filaments. Real-time polarization modulation covering the entire Poincaré sphere can be conveniently obtained through the programming of the on-off state of each LED pixel. Further, the output polarization states can be encoded by an ordered chiral plasmonic film. Our device provides an excellent platform for multiplexing spatiotemporal polarization information, enabling visible light communication with an exceptionally elevated level of physical layer security and multifunctional encrypted displays that can encode both 2D and 3D information.

Novel spatio-temporal attention causal convolutional neural network for multi-site PM2.5 prediction

Multi-site PM2.5 prediction has emerged as a crucial approach, given that the accuracy of prediction models based solely on data from a single monitoring station may be constrained. However, existing multi-site PM2.5 prediction methods predominantly rely on recurrent networks for extracting temporal dependencies and overlook the domain knowledge related to air quality pollutant dispersion. This study aims to explore whether a superior prediction architecture exists that not only approximates the prediction performance of recurrent networks through feedforward networks but also integrates domain knowledge of PM2.5. Consequently, we propose a novel spatio-temporal attention causal convolutional neural network (Causal-STAN) architecture for predicting PM2.5 concentrations at multiple sites in the Yangtze River Delta region of China. Causal-STAN comprises two components: a multi-site spatio-temporal feature integration module, which identifies temporal local correlation trends and spatial correlations in the spatio-temporal data, and extracts inter-site PM2.5 concentrations from the directional residual block to delineate directional features of PM2.5 concentration dispersion between sites; and a temporal causal attention convolutional network that captures the internal correlation information and long-term dependencies in the time series. Causal-STAN was evaluated using one-year data from 247 sites in mainland China. Compared to six state-of-the-art baseline models, Causal-STAN achieves optimal performance in 6-hour future predictions, surpassing the recurrent network model and reducing the prediction error by 8%–10%.

Vehicle trajectory prediction based on cross-attention and multilevel spatio-temporal features

Accurately predicting vehicle trajectories is crucial for motion planning in autonomous driving. Some existing trajectory prediction models employ Long Short-Term Memory (LSTM) networks to encode trajectory information, but this approach may lead to information loss. Additionally, the sparsity of the grid representation may limit the model’s ability to extract vehicle interaction features. To address these issues, this paper proposes a vehicle trajectory prediction model based on cross-attention and multilevel spatio-temporal features. The proposed model incorporates a spatio-temporal feature extraction module designed to capture the dynamic evolution of vehicle states across both time and space. Furthermore, it employs a temporal cross-attention module that reuses hidden states to select spatio-temporal features closely related to the target vehicle’s historical state, thus compensating for the temporal information loss during the encoding process. Additionally, a spatial cross-attention module, informed by social context, is deployed to analyze the dynamic interactions among vehicles. This module filters spatio-temporal interaction features with a significant impact on the target vehicle’s trajectory. The model is trained and tested using the NGSIM dataset and the highD dataset. Compared with other models, the model presented in this paper shows higher prediction accuracy in long-term trajectory prediction.

Attention-Guided and Topology-Enhanced Shift Graph Convolutional Network for Skeleton-Based Action Recognition

Graph Convolutional Networks (GCNs) have emerged as a game-changer in skeleton-based action recognition. However, most previous works are resource-heavy, with large FLoating-number OPerations (FLOPs) limiting the model’s potential. A recent work involving shift operators to GCN (Shift-GCN) has successfully introduced a lightweight GCN, but there is still a performance gap compared to previous results. Inspired by Shift-GCN, we propose an innovative and novel model named attention-guided and topology-enhanced shift graph convolutional network (AT-Shift-GCN), which continues the lightweight benchmark and provides a more powerful performance. We employ a topological transfer operation to aggregate the information flow of different channels and extract spatial information. In addition, to extract temporal information across scales, we apply attention to interacting with shift convolution kernels of different lengths. Furthermore, we integrate an ultralight spatiotemporal attention module to fuse spatiotemporal details and provide robust neighborhood representation. In summary, AT-Shift-GCN is a breakthrough in skeleton-based action recognition that provides a lightweight model with enhanced performance on three datasets.

MSTAN: multi-scale spatiotemporal attention network with adaptive relationship mining for remaining useful life prediction in complex systems

Abstract In the era of smart manufacturing and advanced industrial systems, the high degree of integration and intelligence of equipment demands higher reliability and safety from systems. Existing methods often rely on historical data for Remaining Useful Life (RUL) prediction to achieve Prognostic and Health Management (PHM). However, the internal units of complex equipment exhibit significant spatial correlation and temporal diversity, making PHM for complex equipment a multidimensional challenge involving both temporal and spatial information, thereby severely limits the effectiveness of RUL prediction for complex systems. Addressing these challenges, this study introduces a multi-scale spatiotemporal attention network with adaptive relationship mining, specifically designed for the remaining useful life (RUL) prediction of such equipment. The core of the proposed method lies in the multi-scale feature perception module, which adeptly extracts varied scale features from multidimensional sensor data. Following this, an innovative adaptive relationship mining module is integrated to uncover multi-order coupling relationships between diverse sensors, enhancing the model’s predictive accuracy. Furthermore, a spatiotemporal attention module is employed to discern and emphasize crucial spatiotemporal correlations. To validate the effectiveness and superiority of the proposed method, the Commercial Modular Aero-propulsion System Simulation (C-MAPSS) dataset is employed for comprehensive performance evaluation, the IEEE 2012 PHM bearing dataset is also adopted to demonstrate the generalization and robustness of the proposed method. The results not only show a notable improvement over existing methods but also offer a more intuitive understanding through visual representations, marking a significant stride in enhancing the safety and efficiency of complex systems.

Nonreciprocal conversion based on line trajectory near exceptional points

Spatio-temporal permittivity modulation can simultaneously impart wave vector and frequency shifts during an indirect photonic transition process where nonreciprocal responses are accomplished. Here, we combine the nonreciprocity with exceptional points (EPs) by employing line parameter trajectory to a spatio-temporally modulated waveguide. The instantaneous state evolutions on direction-dependent Riemann sheets are thus different in forward and backward directions. Light propagates through the structure is nonreciprocal. Forwardly, an arbitrary incident wave will convert to a combination of the two waveguide modes with same intensity. Backwardly, the waves almost entirely convert to one specific mode. The conversion efficiency is more than 80% and robust to the line design. The operating wavelength is in the telecommunications band, conducive to integration with other chips. Compared to the widely used loop path, the line path has the merit of easy preparation in experiments as only one parameter is changed. The research sheds light on the optical devices such as isolator and amplifier.

A Non-Uniform Grid Graph Convolutional Network for Sea Surface Temperature Prediction

Sea surface temperature (SST) is an important factor in the marine environment and has significant impacts on climate, ecology, and maritime activities. Most existing SST prediction methods consider the ocean as a uniform field and use a uniform grid to predict SST. However, the marine environment is a complex system, and factors such as solar radiation, differences in land and sea thermal properties, and ocean circulation lead to uneven spatial distributions of SSTs. We propose a non-uniform grid construction method based on an SST spatial gradient to encode SST data, as well as a Non-uniform Grid Graph Convolutional Network (NGGCN) model. The NGGCN consists of two spatiotemporal modules, each of which extracts spatial features from the GCN module, captures temporal correlations through the GRU module, and performs feature restoration and output results through the fully connected module. We selected data from the Yellow Sea and Bohai Sea to validate the effectiveness of the NGGCN in predicting SST at different time scales and prediction steps. The results indicate that our model shows a significant improvement in prediction performance compared to other models.