Temporal Modulation Research Articles

ASF-Net: Robust video deraining via temporal alignment and online adaptive learning

In recent times, learning-based methods for video deraining have demonstrated commendable results. However, there are two critical challenges that these methods are yet to address: exploiting temporal correlations among adjacent frames and ensuring adaptability to unknown real-world scenarios. To overcome these challenges, we explore video deraining from a paradigm design perspective to learning strategy construction. Specifically, we propose a new computational paradigm, Alignment-Shift-Fusion Network (ASF-Net), which incorporates a temporal shift module. This module is novel to this field and provides deeper exploration of temporal information by facilitating the exchange of channel-level information within the feature space. To fully discharge the model’s characterization capability, we further construct a LArge-scale RAiny video dataset (LARA) which also supports the development of this community. On the basis of the newly-constructed dataset, we explore the parameters learning process by developing an innovative re-degraded learning strategy. This strategy bridges the gap between synthetic and real-world scenes, resulting in stronger scene adaptability. Our approach exhibits superior performance in three benchmarks and compelling visual quality in real-world scenarios, underscoring its efficacy. The code is available at https://github.com/vis-opt-group/ASF-Net.

Hierarchical stacked spatiotemporal self-attention network for sea surface temperature forecasting

Sea surface temperature (SST) is a highly complex spatiotemporal variable, which stems from its susceptibility to non-linear dynamical processes and substantial spatiotemporal variability. In particular, accurately forecasting small-scale SST is a formidable challenge due to the compounded effects of diverse physical processes spanning across various scales. In this study, we employ deep learning methods to mine the ocean’s evolutionary patterns, as the ocean’s dynamic mechanisms are inherently embedded in spatiotemporal data. We propose a hierarchical stacked spatiotemporal self-attention mechanism (HSSSA) network architecture. The hierarchical stacked encoder–decoder architecture provides the capability for feature fusion and extraction at different scales. The spatial self-attention and temporal self-attention modules simultaneously focus on information from different spatial locations and time steps, allowing the exploration of spatiotemporal patterns in the complex dynamics of the ocean. The experiments are conducted on a high-resolution East China Sea dataset (1/10°×1/10°) to demonstrate the forecast performance of the proposed model for refined ocean variables. The 15-day forecasts indicate that the HSSSA method outperforms the EOF-ARIMA and CNN-Transformer methods.

Temporal multilayer structures in discrete physical systems towards arbitrary-dimensional non-Abelian Aharonov-Bohm interferences

Temporal modulation recently draws great attentions in wave manipulations, with which one can introduce the concept of temporal multilayer structure, a temporal counterpart of spatially multilayer configurations. This kind of multilayer structure holds temporal interfaces in the time domain, which provides additional flexibility in temporal operations. Here we take this opportunity and propose to simulate a non-Abelian gauge field with a temporal multilayer structure in the discrete physical system. Two basic temporal operations, i.e., the folding/unfolding operation and the phase shift operation are used to design such a temporal multilayer structure, which hence can support noncommutative operations to realize the non-Abelian Aharonov-Bohm interference in the time domain. A two-/three-dimensional non-Abelian gauge field can be built, which may be further extended to higher dimensions. Our work therefore provides a unique platform enabling generalization of non-Abelian physics to arbitrary dimensions and offers a method for wave manipulations with temporal band engineering.

Gait recognition for farm workers via multi-scale temporal feature perception

The monitoring and management of farm workers play a crucial role in agricultural production. The real-time monitoring of farm workers can effectively identify them, facilitating standardized management and operation of the farm. Gait, as a unique biometric feature that can be recognized from a distance, is more suitable for identifying individual farm workers walking on the farm at a distance compared to facial and iris features. Despite the increasing research on gait recognition techniques, existing gait recognition models are still lacking in temporal modeling. The gait of farm workers walking through farmland is greatly influenced by the environment, making it crucial to model the temporal information of gait. In this study, we captured videos of 50 farm workers from 11 different perspectives walking in real agricultural fields and formed the Farm Worker Gait Dataset (FWGD). In comparison with existing gait recognition models, we present a multi-scale temporal feature modeling method based on the GaitBase model and obtain a performance improvement on the FWGD dataset. Specifically, we propose an Adaptive Temporal Self-Attention (ATSA) mechanism to extract the most representative features based on temporal contextual temporal relations. Additionally, we utilize a dilated convolutional layer to extract discrete temporal features and employ a Multi-scale Temporal Perception Module (MSTP) with three branches to perceive and aggregate temporal features. With the assistance of MSTP and ATSA, our model significantly improves the discriminability of temporal features, thus enhancing the performance of farm worker gait recognition. Extensive experiments demonstrated that our proposed method outperformed state-of-the-art gait recognition methods on the FWGD dataset.

A reconfigurable FPGA-based spiking neural network accelerator

The spiking neural network (SNN) is suitable for the intelligent edge computing applications because of its low-power characteristic. This work designs a reconfigurable spiking neural network accelerator supporting the spatiotemporal backpropagation (STBP) training method. The reconfigurable architecture is proposed between the spatial convolution module and the temporal accumulation module of the SNN accelerator. A sparse zero-hopping mechanism is designed to exploit the input sparsity of SNN datasets, and a mask mechanism is introduced between the forward inference computation and the backward training computation to exploit the output sparsity. During the training process, the peak and average performances of the SNN accelerator are 5.57 TOPS and 4.96 TOPS respectively, the power consumption is 6.124 W and the energy efficiency is 0.81 TOPS/W. The peak and average performances of the SNN accelerator are 5.98 TOPS and 5.14 TOPS respectively, the power consumption is 6.943 W and the energy efficiency is 0.74 TOPS/W, during the inference process.

Neighbor-Enhanced Representation Learning for Link Prediction in Dynamic Heterogeneous Attributed Networks

Dynamic link prediction aims to predict future connections among unconnected nodes in a network. It can be applied for friend recommendations, link completion, and other tasks. Network representation learning algorithms have demonstrated considerable effectiveness in various prediction tasks. However, most network representation learning algorithms are based on homogeneous networks and static networks for link prediction that do not consider rich semantic and dynamic information. Additionally, existing dynamic network representation learning methods neglect the neighborhood interaction structure of the node. In this work, we design a neighbor-enhanced dynamic heterogeneous attributed network embedding method (NeiDyHNE) for link prediction. In light of the impressive achievements of the heuristic methods, we learn the information of common neighbors and neighbors’ interaction in heterogeneous networks to preserve the neighbors proximity and common neighbors proximity. NeiDyHNE encodes the attributes and neighborhood structure of nodes as well as the evolutionary features of the dynamic network. More specifically, NeiDyHNE consists of the hierarchical structure attention module and the convolutional temporal attention module. The hierarchical structure attention module captures the rich features and semantic structure of nodes. The convolutional temporal attention module captures the evolutionary features of the network over time in dynamic heterogeneous networks. We evaluate our method and various baseline methods on the dynamic link prediction task. Experimental results demonstrate that our method is superior to baseline methods in terms of accuracy.

Well-Defined Synthetic Copolymers with Pendant Aldehydes Form Biocompatible Strain-Stiffening Hydrogels and Enable Competitive Ligand Displacement.

Dynamic hydrogels are attractive platforms for tissue engineering and regenerative medicine due to their ability to mimic key extracellular matrix (ECM) mechanical properties like strain-stiffening and stress relaxation while enabling enhanced processing characteristics like injectability, 3D printing, and self-healing. Systems based on imine-type dynamic covalent chemistry (DCvC) have become increasingly popular. However, most reported polymers comprising aldehyde groups are based on either end-group-modified synthetic or side-chain-modified natural polymers; synthetic versions of side-chain-modified polymers are noticeably absent. To facilitate access to new classes of dynamic hydrogels, we report the straightforward synthesis of a water-soluble copolymer with a tunable fraction of pendant aldehyde groups (12-64%) using controlled radical polymerization and their formation into hydrogel biomaterials with dynamic cross-links. We found the polymer synthesis to be well-controlled with the determined reactivity ratios consistent with a blocky gradient microarchitecture. Subsequently, we observed fast gelation kinetics with imine-type cross-linking. We were able to vary hydrogel stiffness from ≈2 to 20 kPa, tune the onset of strain-stiffening toward a biologically relevant regime (σc ≈ 10 Pa), and demonstrate cytocompatibility using human dermal fibroblasts. Moreover, to begin to mimic the dynamic biochemical nature of the native ECM, we highlight the potential for temporal modulation of ligands in our system to demonstrate ligand displacement along the copolymer backbone via competitive binding. The combination of highly tunable composition, stiffness, and strain-stiffening, in conjunction with spatiotemporal control of functionality, positions these cytocompatible copolymers as a powerful platform for the rational design of next-generation synthetic biomaterials.

Innovative approaches in imaging photoplethysmography for remote blood oxygen monitoring

Peripheral Capillary Oxygen Saturation (SpO2) has received increasing attention during the COVID-19 pandemic. Clinical investigations have demonstrated that individuals afflicted with COVID-19 exhibit notably reduced levels of SpO2 before the deterioration of their health status. To cost-effectively enable individuals to monitor their SpO2, this paper proposes a novel neural network model named “ITSCAN” based on Temporal Shift Module. Benefiting from the widespread use of smartphones, this model can assess an individual’s SpO2 in real time, utilizing standard facial video footage, with a temporal granularity of seconds. The model is interweaved by two distinct branches: the motion branch, responsible for extracting spatiotemporal data features and the appearance branch, focusing on the correlation between feature channels and the location information of feature map using coordinate attention mechanisms. Accordingly, the SpO2 estimator generates the corresponding SpO2 value. This paper summarizes for the first time 5 loss functions commonly used in the SpO2 estimation model. Subsequently, a novel loss function has been contributed through the examination of various combinations and careful selection of hyperparameters. Comprehensive ablation experiments analyze the independent impact of each module on the overall model performance. Finally, the experimental results based on the public dataset (VIPL-HR) show that our model has obvious advantages in MAE (1.10%) and RMSE (1.19%) compared with related work, which implies more accuracy of the proposed method to contribute to public health.

Recurrent feature supplementation network for video super-resolution

ABSTRACT Efficient aggregation of temporal information is the basis for achieving video super-resolution. Most researchers have employed alignment or propagation to exploit the temporal information of consecutive frames. However, they frequently overlook the centrality of the reference frame in the model reconstruction when using temporal features. Thus, in this paper, we design a novel recurrent feature supplementation network. We divide the temporal information into three parts: surrounding, back propagation and forward propagation, and extract and fuse them separately. A new grouping approach is proposed for extracting features from the reference frame and its surroundings. The backward temporal fusion module and the forward temporal fusion module are designed to aggregate the backward and forward temporal information at a distance. The temporal fusion module is designed to aggregate temporal information from different parts. Moreover, we propose a feature supplementation mechanism to improve the stability of the model. The feature supplement module is devised to improve the utilization of input features and the stability of the model. Experiments demonstrate that our model achieves the state-of-the-art performance.

FTMLP: MLP with Feature-Temporal Block for multivariate time series forecasting

Multivariate time series (MTS) forecasting is extensively applied in real-world scenarios. Recent studies introduced the concept of channel independence, which has achieved significant results. However, this approach often misses the intricate interrelations among features and lacks physical interpretation. To address this problem, we propose a new framework, the Feature-Temporal block, designed to extract both temporal and feature information. Each block contains two modules: a feature module using a gating mechanism to understand competitive interactions among features, and a temporal module using learnable filters for frequency domain filtering. This innovative design allows FTMLP to combine information across feature and sequence dimensions, as well as time and frequency domains. The computational efficiency of FTMLP is enhanced due to its exclusive use of Multi-Layer Perceptron (MLP). Comprehensive experiments on several real-world datasets demonstrate that FTMLP achieves state-of-the-art results in terms of overall performance. Code is available at https://github.com/whxlearning/FTMLP.

ST-TDCN: A two-channel tree-structure spatial–temporal convolutional network model for traffic velocity prediction

In the development of intelligent transport systems which aims to provide safe, convenient, and comfortable transport and effectively addresses issues with traffic congestion and pollution, accurate collection of spatial–temporal data is of great significance. Based on the study of the topology and spatial characteristics of the transportation network, data with spatial–temporal attributed can be predicted by relevant models. Thus, the purpose of providing additional information for decision making and labor cost saving is achieved. This study proposed a more accurate traffic velocity prediction model, named Spatial-Temporal Tree-structure Dual-channel Convolution Network. The model designs a spatial tree convolution module for capturing the spatial features of nodes in the traffic network represented by the tree structure and applies a dual-channel temporal convolutional network module to capture the temporal information of velocity data more accurately. Comparative tests have demonstrated that the model proposed in this study has the advantages of higher accuracy, better convergence and more flexibility in responding to dynamic changes of traffic velocity.

WITHDRAWN: Perfect asymmetric transmission for linear polarization in background reflection-free bilayer dielectric metasurfaces

In spite of the importance of asymmetric transmission (AT) with near-unity value and significantly high contrast of transmission, such a perfect AT has not yet been realized in reciprocal systems. For single resonance, it is already known that the upper limit of AT is identical to the background amplitude reflection coefficient. Nevertheless, if multiple resonances simultaneously contribute to AT, their coherent coupling enables us to obtain perfect AT, even when background reflection completely vanishes. Taking bilayer dielectric metasurfaces composed of orthogonal silicon nanorods embedded in silica glass as examples, here we show that perfect AT can be realized for linear polarization with characteristics comparable with bulky optical isolators based on Faraday effect: AT of 0.9994 with dramatically high contrast 35.1dB at telecom wavelength, the spectral position of which is tunable by properly changing the structural parameters. The analytical model based on the quasinormal-mode expansion well agrees with numerical results, revealing the concurrent contributions of multiple resonances. The presented perfect AT has the advantages of excellent performances of AT and compact structures without bulky devices for field bias or temporal modulations, enabling practical applications for optical computing, high-contrast polarization multiplexing and conversion.

Feasibility exploration of cortical auditory evoked potentials evoked by spectro-temporal modulation signals

Background An objective measurement of speech perception would be valuable in hearing-impaired patients who are unable to perform auditory tasks reliably. Objectives To investigate the feasibility of cortical auditory evoked potentials (CAEPs) evoked by different spectro-temporal modulation (STM) signals and provide reference for the further exploration of acoustic change complex (ACC) in hearing-impaired patients. Method 29 normal hearing (NH) adults were recruited and stimulated randomly by STM signals at 6 spectral modulation rates: 0, 1, 2, 4, 8, 16 cycles/octave, at each of 4 temporal modulation rates: 0, 2, 4, 8 Hz, to elicit ACC response. Results The change of spectral modulation rates ≤ 16 cycles/octave at 0 Hz temporal modulation rate had different impacts on latency. 0, 1 cycles/octave, and 4, 8 Hz made a difference to the P2-N1 amplitude easily. 8, 16 cycles/octave were more unstable in rejection rate. No significant difference existed in waveform repetition rate among different STM rates. No significant interaction effect existed in spectral and temporal modulation. Conclusions STM as a stimulus signal to induce ACC response had its feasibility. 2 cycles/octave at spectral modulation rate and 2 Hz at temporal modulation rate had better stability and waveform repeatability in NH adults.

Investigating acoustic startle habituation and prepulse inhibition with silent functional MRI and electromyography in young, healthy adults.

Startle habituation and prepulse inhibition (PPI) are distinct measures of different sensory information processes, yet both result in the attenuation of the startle reflex. Identifying startle habituation and PPI neural mechanisms in humans has mostly evolved from acoustic-focused rodent models. Human functional magnetic resonance imaging (fMRI) studies have used tactile startle paradigms to avoid the confounding effects of gradient-related acoustic noise on auditory paradigms and blood-oxygen-level-dependent (BOLD) measures. This study aimed to examine the neurofunctional basis of acoustic startle habituation and PPI in humans with silent fMRI. Using silent fMRI and simultaneous electromyography (EMG) to measure startle, the neural correlates of acoustic short-term startle habituation and PPI [stimulus onset asynchronies (SOA) of 60 ms and 120 ms] were investigated in 42 healthy adults (28 females). To derive stronger inferences about brain-behaviour correlations at the group-level, models included EMG-assessed measures of startle habituation (regression slope) or PPI (percentage) as a covariate. A linear temporal modulator was modelled at the individual-level to characterise functional changes in neural activity during startle habituation. Over time, participants showed a decrease in startle response (habituation), accompanied by decreasing thalamic, striatal, insula, and brainstem activity. Startle habituation was associated with the linear temporal modulation of BOLD response amplitude in several regions, with thalamus, insula, and parietal lobe activity decreasing over time, and frontal lobe, dorsal striatum, and posterior cingulate activity increasing over time. The paradigm yielded a small amount of PPI (9-13%). No significant neural activity for PPI was detected. Startle habituation was associated with the thalamus, putamen, insula, and brainstem, and with linear BOLD response modulation in thalamic, striatal, insula, parietal, frontal, and posterior cingulate regions. These findings provide insight into the mediation and functional basis of the acoustic primary startle circuit. Instead, whilst reduced compared to conventional MRI, scanner noise may have disrupted prepulse detection and processing, resulting in low PPI and impacting our ability to map its neural signatures. Our findings encourage optimisation of the MRI environment for acoustic PPI-based investigations in humans. Combining EMG and functional neuroimaging methods shows promise for mapping short-term startle habituation in healthy and clinical populations.

Untangling light in noisy luminous environments

Electric light sources, including displays and luminaires, often exhibit rapid light fluctuations called temporal light modulation (TLM). TLM can be seen as their electrical footprint. This work demonstrates that TLM can be remotely detected and used to perform in situ photometric and spectroradiometric measurements of very low light levels superimposed to intense backgrounds. Illuminance levels about 500 000 times smaller than the background were reliably measured by lock-in detection, allowing photometric measurements to be performed during daytime, even with sunny conditions. Using a new type of optical lock-in spectrometer, it becomes possible to retrieve spectral quantities through the TLM of the light source. The spectral distribution of an LED lamp was measured against a daylit background more than 1500 times brighter. Lock-in detection is an excellent technique to perform in situ measurements of electric light, even at very low level, in the presence of daylight and other background lights, opening interesting opportunities to characterize anthropogenic light pollution.

Driver fatigue detection method based on temporal–spatial adaptive networks and adaptive temporal fusion module

The reduction of traffic accidents by determining the driver’s state through fatigue detection is a worthy research issue. Most of the current fatigue driving detection method fails to fully utilize the temporal features of fatigue. To address this problem, this paper proposes a driver fatigue detection method combined with temporal–spatial adaptive networks (TSNet) and adaptive temporal fusion module (ATFM). First, a frame sequence of T frames is obtained by strided sampling of the input video and data enhancement. Subsequently, the temporal–spatial adaptive module (TSAM) is used as the core module and incorporated into Efficientnet-v2 to construct TSNet, which adaptively extracts temporal features according to different videos, adds attention weights to discriminative spatial and channel features, and fully extracts fatiguing temporal–spatial features of videos. Finally, ATFM is utilized to learn the weights between the fatigue classification scores of each frame in the frame sequence and adaptively fuses the fatigue classification scores of individual frames to obtain fatigue prediction results, increasing the extent of the influence of keyframes on the fatigue prediction results. In this paper, the proposed method achieves an accuracy of 89.42% on the NTHU-DDD dataset, which is better than other state-of-the-art methods, and the number of parameters of the proposed method is 24.70M, which is smaller than most of the methods. Through a series of comparative experiments, TSNet and ATFM alone also outperform models and modules with similar functionality.

PolypNextLSTM: a lightweight and fast polyp video segmentation network using ConvNext and ConvLSTM

PurposeCommonly employed in polyp segmentation, single-image UNet architectures lack the temporal insight clinicians gain from video data in diagnosing polyps. To mirror clinical practices more faithfully, our proposed solution, PolypNextLSTM, leverages video-based deep learning, harnessing temporal information for superior segmentation performance with least parameter overhead, making it possibly suitable for edge devices.MethodsPolypNextLSTM employs a UNet-like structure with ConvNext-Tiny as its backbone, strategically omitting the last two layers to reduce parameter overhead. Our temporal fusion module, a Convolutional Long Short Term Memory (ConvLSTM), effectively exploits temporal features. Our primary novelty lies in PolypNextLSTM, which stands out as the leanest in parameters and the fastest model, surpassing the performance of five state-of-the-art image and video-based deep learning models. The evaluation of the SUN-SEG dataset spans easy-to-detect and hard-to-detect polyp scenarios, along with videos containing challenging artefacts like fast motion and occlusion.ResultsComparison against 5 image-based and 5 video-based models demonstrates PolypNextLSTM’s superiority, achieving a Dice score of 0.7898 on the hard-to-detect polyp test set, surpassing image-based PraNet (0.7519) and video-based PNS+ (0.7486). Notably, our model excels in videos featuring complex artefacts such as ghosting and occlusion.ConclusionPolypNextLSTM, integrating pruned ConvNext-Tiny with ConvLSTM for temporal fusion, not only exhibits superior segmentation performance but also maintains the highest frames per speed among evaluated models. Code can be found here: https://github.com/mtec-tuhh/PolypNextLSTM.

Transformer models for Land Cover Classification with Satellite Image Time Series

In this paper we address the task of pixel-wise land cover (LC) classification using satellite image time series (SITS). For that purpose, we use a supervised deep learning model and focus on combining spatial and temporal features. Our method is based on the Swin Transformer and captures global temporal features by using self-attention and local spatial features by convolutions. We extend the architecture to receive multi-temporal input to generate one output label map for every input image. In our experiments we focus on the application of pixel-wise LC classification from Sentinel‑2 SITS over the whole area of Lower Saxony (Germany). The experiments with our new model show that by using convolutions for spatial feature extraction or a temporal weighting module in the skip connections the performance improves and is more stable. The combined usage of both adaptations results in the overall best performance although this improvement is only minimal. Compared to a fully convolutional neural network without any self-attention layers our model improves the results by 2.1% in the mean F1-Score on a corrected test dataset. Additionally, we investigate different types of temporal position encoding, which do not have a significant impact on the performance.

Multi-perspective feedback-attention coupling model for continuous-time dynamic graphs

Abstract Representation learning over graph networks has recently gained popularity, with many models showing promising results. However, several challenges remain: (1) most methods are designed for static or discrete-time dynamic graphs; (2) existing continuous-time dynamic graph algorithms focus on a single evolving perspective; and (3) many continuous-time dynamic graph approaches necessitate numerous temporal neighbors to capture long-term dependencies. In response, this paper introduces a Multi-Perspective Feedback-Attention Coupling (MPFA) model. MPFA incorporates information from both evolving and original perspectives to effectively learn the complex dynamics of dynamic graph evolution processes. The evolving perspective considers the current state of historical interaction events of nodes and uses a temporal attention module to aggregate current state information. This perspective also makes it possible to capture long-term dependencies of nodes using a small number of temporal neighbors. Meanwhile, the original perspective utilizes a feedback attention module with growth characteristic coefficients to aggregate the original state information of node interactions. Experimental results on one dataset organized by ourselves and seven public datasets validate the effectiveness and competitiveness of our proposed model.

Spatio-temporal feature learning for enhancing video quality based on screen content characteristics

With the rising demands for remote desktops and online meetings, screen content videos have drawn significant attention. Different from natural videos, screen content videos often exhibit scene switches where the content abruptly changes from one frame to the next. These scene switches result in obvious distortions in compressed videos. Besides, frame freezing, where the content remains unchanged for a certain duration, is also very common in screen content videos. Existing alignment-based models struggle to effectively enhance scene switch frames and lack efficiency when dealing with frame freezing situations. Therefore, we propose a novel alignment-free method that effectively handles both scene switches and frame freezing. In our approach, we develop a spatial and temporal feature extraction module that compresses and extracts spatio-temporal information from three groups of frame inputs. This enables efficient handling of scene switches. In addition, an edge aware block is proposed for extracting edge information, which guides the model to focus on restoring the high-frequency components in frame freezing situations. The fusion module is then designed to adaptively fuse the features from three groups, considering different positions of video frames, to enhance frames during scene switch and frame freezing scenarios. Experimental results demonstrate the significant advancements achieved by the proposed edge aware with spatio-temporal information fusion network (EAST) in enhancing the quality of compressed videos, surpassing the current state-of-the-art methods.