Temporal Modulation Research Articles

SAPPNet: students’ academic performance prediction during COVID-19 using neural network

A variety of reasons have made it more difficult for educators and tutors to anticipate students’ performance. Numerous researchers have used various predictive models to identify students who may be at-risk of dropping out early. Additionally, these methods were used to forecast final semester grades based on various datasets. However, these prediction models still fall short of meeting educational management requirements. In this paper, we propose the deep learning (DL) based model named students academic performance prediction network (SAPPNet) to predict the students’ grades. We consider the questionnaire-based Jordan University dataset which contains demographic information, usage of digital tools before and after COVID-19, sleep times before and after COVID-19, social interaction, psychological state, and academic performance. SAPPNet consists of spatial convolution modules which are designed to extract spatial dependencies includes categorical and numerical attributes that represent static features (gender, level/year, age, digital tools used before and after COVID-19, psychological condition using prolonged e-learning tools) and temporal module for temporal dependencies involves sequences that capture changes before and after COVID-19. Additionally, we also try to implement classical machine learning (ML) models including support vector machine, k nearest neighbor, decision tree, and random forest, and DL models named artificial neural network, convolutional neural network, long short-term memory, and students learning prediction network. Simulation results show that SAPPNet achieved the best performance compared to state-of-the-art methods, with an accuracy, precision, recall, and an F1-score of 93%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$93\\%$$\\end{document}. The proposed model with spatial and temporal modules improves the prediction performance, and it implies new aspect of the educational dataset.

Agile manipulation of the time-frequency distribution of high-speed electromagnetic waves

Controlling the temporal evolution of an electromagnetic (EM) wave’s frequency components, the so-called time-frequency (TF) distribution, in a versatile and real-time fashion remains very challenging, especially at the high speeds (> GHz regime) required in contemporary communication, imaging, and sensing applications. We propose a general framework for manipulating the TF properties of high-speed EM waves. Specifically, the TF distribution is continuously mapped along the time domain through phase-only processing, enabling its user-defined manipulation via widely-available temporal modulation techniques. The time-mapping operations can then be inverted to reconstruct the TF-processed signal. Using off-the-shelf telecommunication components, we demonstrate arbitrary control of the TF distribution of EM waves over a full bandwidth approaching 100 GHz with nanosecond-scale programmability and MHz-level frequency resolution. We further demonstrate applications for mitigating rapidly changing frequency interference terms and the direct synthesis of fast waveforms with customized TF distributions. The reported method represents a significant advancement in TF processing of EM waves and it fulfills the stringent requirements for many modern and emerging applications.

Generation of patterns and higher harmonics in 1D quantum droplets in tilted and driven quasi-periodic confinements

The generation of patterns by breaking the spatial symmetry in external confinement is a captivating area of physics. The emergence of patterns is a fundamental inquiry spanning various disciplines such as nonlinear optics, condensed matter physics, and fluid dynamics. The article investigates the generation of a variety of patterns in a one-dimensional binary mixture of Bose–Einstein condensate forming quantum droplets. By solving the extended Gross–Pitaevskii equation in the presence of tilted and driven engineered bi-chromatic optical lattices (BOL), the out-of-equilibrium dynamics of droplets under strong dc and ac fields are illustrated. Under the influence of a dc field, a stripe-like pattern emerges in the temporal domain, while the scenario with ac fields demonstrates temporal periodic and bi-periodic oscillations of density waves. The width and period of formed patterns are directly correlated with the strength of ac and dc fields. Moreover, temporal modulation of the BOL potential depth yields various harmonics in the oscillations of the condensate density pattern. Through Fast Fourier Transform (FFT) analysis, it is confirmed that these harmonics encompass multiple and combinational frequencies, suggesting potential applications in generating desired frequency combs within quantum droplets. We have also carried out a thorough numerical stability check of the obtained solutions and found them sufficiently stable.

Temporally enhanced abnormal behavior detection based on multi-channel coupling

Abnormal behavior detection in surveillance videos based on deep learning has becomea hot topic in current research. However, due to the complexity and variability of crowd movement and external environment, detecting abnormal behavior faces great challenges. Current abnormal behavior recognition models have limitations in feature extraction and pay insufficient attention to dynamic temporal features. To address these problems, a spatiotemporal enhancement anomaly detection method based on multi-channel coupling is proposed. Based on the SlowFast network, a multi-channel coupled temporal enhancement module and a spatial enhancement module are introduced respectively to extract more discriminative static and dynamic feature information. A large number of experiments are carried out on three benchmark datasets (Violent Flow, Hockey Fight, and Real-life Violence Situations). The results show that the prediction accuracy of the proposed abnormal behavior recognition method reaches 95.3%、97.3%and respectively 94%, thus verifying the effectiveness of the method.

Short-Term Prediction of Origin–Destination Passenger Flow in Urban Rail Transit Systems with Multi-Source Data: A Deep Learning Method Fusing High-Dimensional Features

Short-term origin–destination (OD) passenger flow forecasting is crucial for urban rail transit enterprises aiming to optimise transportation products and increase operating income. As there are large-scale OD pairs in an urban rail transit system, OD passenger flow cannot be obtained in real time (temporal hysteresis). Additionally, the distribution characteristics are also complex. Previous studies mainly focus on passenger flow prediction at metro stations, while few methods solve the OD passenger flow prediction problems of an urban rail transit system. In view of this, we propose a novel deep learning method fusing high-dimensional features (HDF-DL) with multi-source data. The HDF-DL method is combined with three modules. The temporal module incorporates the time-varying, trend, and cyclic characteristics of OD passenger flow, while the latest OD passenger flow time sequence (within 1 h) is excluded from the time-varying characteristics. In the spatial module, the K-means and K-shape algorithms are used to classify OD pairs from multiple perspectives and capture the spatial features, reducing the difficulty of OD passenger flow predictions with large-scale and complex characteristics. Weather factors are considered in the external feature module. The HDF-DL method is tested on a large-scale metro system in China, in which eight baseline models are designed. The results show that the HDF-DL method achieves high prediction accuracy across multiple time granularities, with a mean absolute percentage error of about 10%. OD passenger flow in every departure time interval can be predicted with high and stable accuracy, effectively capturing temporal characteristics. The modular design of HDF-DL, which fuses high-dimensional features and employs appropriate neural networks for different data types, significantly reduces prediction errors and outperforms baseline models.

Role of chirping in pulsed single-photon spectroscopy

We investigate the precision of estimating the interaction strength between a two-level system (TLS) and a single-photon pulse when the latter is subject to chirping. We consider linear, quadratic, and sinusoidal temporal phases applied to Gaussian and exponential temporal profiles. At the asymptotic time, when the TLS has fully decayed to its ground state, the fundamental precision depends solely on the magnitude of its spectral amplitude. For quadratically phase-modulated Gaussian pulses, this is entirely determined by the spectral bandwidth. We provide expressions for evaluating the fundamental precision for general temporal profiles and phase modulations. Finally, we show that experimentally feasible mode-resolved measurements are optimal, or close to it, for chirped, pulsed single-photon spectroscopy. Published by the American Physical Society 2024

Spatial–Temporal-Correlation-Constrained Dynamic Graph Convolutional Network for Traffic Flow Forecasting

Accurate traffic flow prediction in road networks is essential for intelligent transportation systems (ITS). Since traffic data are collected from the road network with spatial topological and time series sequences, the traffic flow prediction is regarded as a spatial–temporal prediction task. With the powerful ability to model the non-Euclidean data, the graph convolutional network (GCN)-based models have become the mainstream framework for traffic forecasting. However, existing GCN-based models either use the manually predefined graph structure to capture the spatial features, ignoring the heterogeneity of road networks, or simply perform 1-D convolution with fixed kernel to capture the temporal dependencies of traffic data, resulting in insufficient long-term temporal feature extraction. To solve those issues, a spatial–temporal correlation constrained dynamic graph convolutional network (STC-DGCN) is proposed for traffic flow forecasting. In STC-DGCN, a spatial–temporal embedding encoder module (STEM) is first constructed to encode the dynamic spatial relationships for road networks at different time steps. Then, a temporal feature encoder module with heterogeneous time series correlation modeling (TFE-HCM) and a spatial feature encoder module with dynamic multi-graph modeling (SFE-DCM) are designed to generate dynamic graph structures for effectively capturing the dynamic spatial and temporal correlations. Finally, a spatial–temporal feature fusion module based on a gating fusion mechanism (STM-GM) is proposed to effectively learn and leverage the inherent spatial–temporal relationships for traffic flow forecasting. Experimental results from three real-world traffic flow datasets demonstrate the superior performance of the proposed STC-DGCN compared with state-of-the-art traffic flow forecasting models.

Magnetospheric Line Radiation: Temporal Modulation Corresponding to a Bouncing Wave

AbstractMagnetospheric line radiation (MLR) is a peculiar type of whistler‐mode emissions formed by several similarly spaced spectral lines, whose frequencies may vary with time. These emissions typically occur at frequencies between about 1 and 8 kHz and are observed in both ground‐based and satellite measurements. However, their origin and source locations are not yet understood, as their detection by spacecraft instruments is challenging due to the limited frequency resolution of onboard wave measurements. We use high‐resolution multicomponent wave data obtained by the Van Allen Probes to demonstrate that, in addition to the frequency modulation, MLR events possess a temporal modulation with periods on the order of seconds, which may be related to the wave bouncing between hemispheres. Observed modulation periods are used to estimate a tentative source L‐shell. The modulation periods are shorter for events with larger frequency spacing, and the estimated source L‐shells correlate with model plasmapause locations.

Holistic and in situ calibration method for a spectral-temporally modulated Mueller spectropolarimeter

The spectral-temporally modulated Mueller spectropolarimeter (STMSP) offers advantages of broader band limitation, better resolution, and faster detection speed. However, the current STMSP calibration method separates the polarization state generator and analyzer, necessitating subsequent recombination, which is inefficient and unstable. In this paper, a holistic and in situ calibration method for STMSP is proposed. It only requires insertion of a polarizer as a reference sample, eliminating the need for separate calibration and recombination. The STMSP is calibrated in situ as a whole, addressing the misalignment error of the spectral modulation module, the total polarimetric errors of the temporal modulation module, and the spectral modulation transfer function of the spectrometer. Experimental results demonstrate high accuracy, with a root mean square error (RMSE) of 0.0004, which is an order of magnitude lower than that of the dual-rotating retarder spectropolarimeter (DRRSP) after eigenvalue calibration. This demonstrates its potential for enabling faster and more accurate acquisition of the Mueller spectra.

Error-robust hybrid atom-photon entangling gate via Gaussian soft control

Hybrid atom-photon gates play an important role for the realization of a quantum interface capable of mapping atomic states to photons for communication across quantum networks. Here, we propose a feasible theoretical scheme for implementing a hybrid atom-photon controlled-Z gate between an atom and a microwave photon in a superconducting coplanar waveguide resonator based on the Gaussian soft control technique. The gate protocol employs a classical auxiliary field that induces an atomic transition between one state of the atomic qubit and Rydberg states for obtaining strong coupling of the atom and microwave resonator. By tailoring the amplitude of this field with Gaussian temporal modulation, the gate performances are improved in various aspects. Numerical simulations demonstrate that the controlled-Z gate based on Gaussian soft control is resilient to the variation of the atom-photon coupling strength and deviation in the gate time, and it is less sensitive to the Rydberg level shifts caused by stray electric fields.

Dynamic control of Lamb modes in phononic crystals

This study explores the possibilities of temporal control of one-dimensional phononic crystal consisting of a homogeneous piezoelectric plate on which metallic electrodes are structured. Previous works [1] have shown that Lamb-type guided modes in these periodically structured piezoelectric ceramic plates exhibit a tunable frequency dispersion that can be controlled by external electric circuits connected to the electrodes. In this study we develop a device enabling changes in the electric boundary conditions (EBC) from one condition to another by making use of switches driven by a microcontroller. This device agility is exploited to perform dynamic control of the crystal properties. To our knowledge, this is the first study devoted to the dynamic modulation of Lamb-type guided waves. Here, we do not simply change the EBCs as a function of time, but we create a "wave" of modulation of properties. Broadband translation of Lamb-wave dispersion curves in the frequency-wavenumber space (f, k) are observed, depending on the modulation speed. Experimentally, the translation of the S0 mode is detected. The relationship between the position of static bands and the position of bands after temporal modulation obeys a very simple vector rule in the (f, k) space. [1] Phys. Rev. B 99, 094302 (2019).

Endogenous pain modulation after sleep restriction in migraine: a blinded crossover study

BackgroundPatients with migraine are vulnerable to insufficient sleep, but the impact of sleep restriction is largely unknown. In addition, the importance of sleep may be different in patients with migraine who mostly have attack onsets during sleep, so called sleep-related migraine, compared to patients with non-sleep-related migraine. In this study we investigate the effect of sleep restriction on endogenous pain modulation in patients with migraine and healthy controls. We also compared the effect of sleep restriction in sleep-related and in non-sleep-related migraine.MethodsMeasurements were conducted in 39 patients with migraine between attacks and 31 controls, once after habitual sleep and once after two consecutive nights of partial sleep restriction. There were 29 and 10 patients with non-sleep-related and sleep-related migraine respectively. Test stimulus was 2-min tonic noxious heat to the left volar forearm. Temporal summation was calculated as the regression coefficient for rated pain in the late part of this 2-min stimulation. Conditioning stimulus was right hand-immersion in 7 °C water. Conditioned pain modulation was defined as the difference in rated pain with and without the conditioning stimulus and was calculated for temporal summation and mean rated pain for the test stimulus. The effect of sleep restriction on temporal summation and conditioned pain modulation was compared in migraine subjects and controls using two-level models with recordings nested in subjects.ResultsConditioned pain modulation for temporal summation of heat pain tended to be reduced after sleep restriction in patients with migraine compared to controls (p = 0.060) and, in an exploratory analysis, was reduced more after sleep restriction in sleep-related than in non-sleep-related migraine (p = 0.017). No other differences between groups after sleep restriction were found for temporal summation or conditioned pain modulation.ConclusionPatients with migraine may have a subtly altered endogenous pain modulation system. Sleep restriction may have an increased pronociceptive effect on this system, suggesting a mechanism for vulnerability to insufficient sleep in migraine. This effect seems to be larger in sleep-related migraine than in non-sleep-related migraine.

Multimodal vehicle trajectory prediction and integrated threat assessment algorithm based on adaptive driving intention

As autonomous driving and connected communication technologies advance swiftly, vehicle trajectory prediction has become increasingly significant. The motion of a vehicle is contingent not only on its historical trajectory but is also subject to the influence of surrounding vehicles, thereby exhibiting intricate social and temporal interdependencies. Furthermore, the inherent randomness and uncertainty in driver behavior render vehicle trajectory prediction inherently multimodal, a factor that is frequently neglected in current research. Against this backdrop, a multimodal vehicle trajectory prediction (MTP) model based on an encoder-decoder architecture is proposed to hierarchically extract historical features of vehicles. The model consists of five key components: temporal feature encoder module, spatial interaction module, spatial-temporal dependence module, driving intention fusion module and multimodal trajectory output module. Experiments on the NGSIM dataset show that the predictive performance of the model has been improved to varying degrees, especially at 3–5 s, where the improvement is more significant. Compared with state-of-the-art models, the Root Mean Square Error (RMSE) error at 5 s time horizon is 3.38 m on NGSIM dataset, which represents a 25 % improvement. To measure the safety of predicted trajectories, we propose a comprehensive threat assessment model that combines collision time (TTC), headway (TH) and time to lateral collision (TLC) metrics based on safe distance theory. This model not only evaluates the longitudinal collision threat in the following state, but also evaluates the lateral collision threat during driving maneuvers in multi lane scenarios, thereby comprehensively improving the safety of vehicle driving. This research also offers new perspectives and insights for the development of autonomous driving.

HOI-V: One-stage human-object interaction detection based on multi-feature fusion in videos

Effective detection of Human-Object Interaction (HOI) is important for machine understanding of real-world scenarios. Nowadays, image-based HOI detection has been abundantly investigated, and recent one-stage methods strike a balance between accuracy and efficiency. However, it is difficult to predict temporal-aware interaction actions from static images since limited temporal context information is introduced. Meanwhile, due to the lack of early large-scale video HOI datasets and the high computational cost of spatial-temporal HOI model training, recent exploratory studies mostly follow a two-stage paradigm, but independent object detection and interaction recognition still suffer from computational redundancy and independent optimization. Therefore, inspired by the one-stage interaction point detection framework, a one-stage spatial-temporal HOI detection baseline is proposed in this paper, in which the short-term local motion features and long-term temporal context features are obtained by the proposed temporal differential excitation module (TDEM) and DLA-TSM backbone. Complementary visual features between multiple clips are then extracted by multi-feature fusion and fed into the parallel detection branches. Finally, a video dataset containing only actions with reduced data size (HOI-V) is constructed to motivate further research on end-to-end video HOI detection. Extensive experiments are also conducted to verify the validity of our proposed baseline.

Time of day affects MrgD-dependent modulation of cardiomyocyte contractility.

The renin-angiotensin system (RAS) is composed of a series of peptides, receptors, and enzymes that play a pivotal role in maintaining cardiovascular homeostasis. Among the most important players in this system are the angiotensin-II and angiotensin-(1-7) peptides. Our group has recently demonstrated that alamandine (ALA), a peptide with structural and functional similarities to angiotensin-(1-7), interacts with cardiomyocytes, enhancing contractility via the Mas-related G protein-coupled receptor member D (MrgD). It is currently unknown whether this modulation varies along the distinct phases of the day. To address this issue, we assessed the ALA-induced contractility response of cardiomyocytes from mice at four Zeitgeber times (ZTs). At ZT2 (light phase), ALA enhanced cardiomyocyte shortening in an MrgD receptor-dependent manner, which was associated with nitric oxide (NO) production. At ZT14 (dark phase), ALA induced a negative modulation on the cardiomyocyte contraction. β-Alanine, an MrgD agonist, reproduced the time-of-day effects of ALA on myocyte shortening. NG-nitro-l-arginine methyl ester, an NO synthase inhibitor, blocked the increase in fractional shortening induced by ALA at ZT2. No effect of ALA on myocyte shortening was observed at ZT8 and ZT20. Our results show that ALA/MrgD signaling in cardiomyocytes is subject to temporal modulation. This finding has significant implications for pharmacological approaches that combine chronotherapy for cardiac conditions triggered by disruption of circadian rhythms and hormonal signaling.NEW & NOTEWORTHY Alamandine, a member of the renin-angiotensin system, serves critical roles in cardioprotection, including the modulation of cardiomyocyte contractility. Whether this effect varies along the day is unknown. Our results provide evidence that alamandine via receptor MrgD exerts opposing actions on cardiomyocyte shortening, enhancing, or reducing contraction depending on the time of day. These findings may have significant implications for the development and effectiveness of future cardiac therapies.

Dynamic Reorganization Patterns of Brain Modules after Stroke Reflecting Motor Function.

Advancements in neuroimaging technologies have significantly deepened our understanding of the neural physiopathology associated with stroke. Nevertheless, the majority of studies ignored the characteristics of dynamic changes in brain networks. The relationship between dynamic changes in brain networks and the severity of motor dysfunction after stroke needs further investigation. From the perspective of multilayer network module reconstruction, we aimed to explore the dynamic reorganization of the brain and its relationship with motor function in subcortical stroke patients. We recruited 35 healthy individuals and 50 stroke patients with unilateral limb motor dysfunction (further divided into mild-moderate group and severe group). Using dynamic multilayer network modularity analysis, we investigated changes in the dynamic modular reconfiguration of brain networks. Additionally, we assessed longitudinal clinical scale changes in stroke patients. Correlation and regression analyses were employed to explore the relationship between characteristic dynamic indicators and impairment and recovery of motor function, respectively. We observed increased temporal flexibility in the Default Mode Network (DMN) and decreased recruitment of module reconfiguration in the Attention Network (AN), Sensorimotor Network (SMN), and DMN after stroke. We also observed reduced module loyalty following stroke. Additionally, correlation analysis showed that hyper-flexibility of the DMN was associated with better lower limb motor function performance in stroke patients with mild-to-moderate impairment. Regression analysis indicated that increased flexibility within the DMN and decreased recruitment coefficient within the AN may predict good lower limb function prognosis in patients with mild to moderate motor impairment. Our study revealed more frequent modular reconfiguration and hyperactive interaction of brain networks after stroke. Notably, dynamic modular remodeling was closely related to the impairment and recovery of motor function. Understanding the temporal module reconfiguration patterns in multilayer networks after stroke can provide valuable information for more targeted treatments.

A complex-valued convolutional fusion-type multi-stream spatiotemporal network for automatic modulation classification

Automatic Modulation Classification (AMC) is crucial in non-cooperative communication systems as it facilitates the identification of interference signals with minimal prior knowledge. Although there have been significant advancements in Deep Learning (DL) within the field of AMC, leveraging the inherent relationships between In-phase (I) and Quadrature-phase (Q) components, and enhance recognition accuracy under low signal-to-noise ratio (SNR) conditions remains a challenge. This study introduces a complex-valued convolutional fusion-type multi-stream spatiotemporal network (CC-MSNet) for AMC, which combines spatial and temporal feature extraction modules for modulation recognition. Experimental results demonstrate that the CC-MSNet performs well on three benchmark datasets, RML2016.10a, RML2016.10b, and RML2016.04c, with average recognition accuracy of 62.86%, 65.08%, and 71.12%. It also performs excellently in low SNR environments below 0dB, significantly outperforming other networks.

Convergence and divergence of storm waves induced by multi-scale currents: Observations and coupled wave-current modeling

Current effects on waves (CEW) are among the most intricate physical processes in wave evolution. In this study, we used a coupled wave-tide-circulation model for the Northwest Atlantic to investigate current effects on storm waves during Hurricane Igor in 2010. Validated with extensive buoy and altimeter data, the inclusion of CEW in the model significantly improves the accuracy in simulating significant wave heights (Hs) by up to 21.3% for a wave buoy. Storm waves experience significant temporal and spatial modulation by multi-scale currents. Storm-driven currents have the most pronounced impact to the right of the storm track, which typically align with wave propagation and reduce Hs by up to 12.1%. The subsequent near-inertial oscillations induce temporal fluctuations of wave convergence and divergence at near-inertial frequencies, which also occurs in regions with strong tidal currents but at tidal frequencies. Furthermore, storm waves are modulated by the Gulf Stream, Labrador Current and associated mesoscale eddies. Overall, these multi-scales yield strong effects on storm waves (Hs > 3.0 m), significantly modulating Hs (−25.2%–+55.4%) and mean wave periods (−14.9%–+15.7%). The mean wave energy power shows more significant modulation by multi-scale currents, reflecting the combined effects of changing wave states and current-induced transport of wave energy. CEW are governed by the interactive dynamic and kinematic effects. The relative wind effect is the primary mechanism for lower storm waves by reducing energy input to waves and influences downstream wave states. Among kinematic effects, current-induced wave refraction typically plays a dominant role in redistributing wave energy. This study systematically quantified the modulation of storm waves by multi-scale currents and revealed the underlying mechanisms, providing a comprehensive understanding of extreme wave states under coupled ocean dynamics.

Depression detection based on the temporal-spatial-frequency feature fusion of EEG

Depression is a prevalent affective psychiatric disorder projected to be the leading contributor to the world’s disease burden by 2030. Due to its high prevalence and low recognition rate, an objective and effective detection method is urgently needed. Deep learning methods based on electroencephalography (EEG) have shown significant potential in depression detection. However, excessive channels can increase redundancy and computational complexity in EEG, while irrelevant channels may reduce accuracy. Additionally, existing models often overlook the complementarity between the temporal-, spatial-, and frequency-domain features of EEG, limiting their detection capabilities. To address these issues, we propose a method that fuses the temporal, spatial, and frequency domain features of EEG to enhance the detection accuracy while eliminating redundant channels. We introduce an EEG channel selection method based on frequency domain weighting that automatically adjusts the channel weights to select the EEG channels that best capture spatial information across the delta, theta, alpha, beta, and gamma bands, thereby optimizing the extraction of spatial-frequency features. In addition, we designed a multiscale spatiotemporal convolutional attention network to extract the spatiotemporal features of EEG. In this network, the multiscale convolutional attention module enhanced the model’s ability to capture spatial features, whereas the temporal trend-aware self-attention module extracted long-term temporal features by analyzing global correlations across different time points. Experimental results on the MODMA dataset show that our method achieved a 97.24% detection accuracy, surpassing current state-of-the-art models. This study offers a novel approach for constructing depression detection models, providing a foundation for future research and application.

Bidirectional temporal and frame-segment attention for sparse action segmentation of figure skating

Temporal action segmentation is a task for understanding human activities in long-term videos. Most of the efforts have been focused on dense-frame action, which relies on strong correlations between frames. However, in the figure skating scene, technical actions are sparsely shown in the video. This brings new challenges: a large amount of redundant temporal information leads to weak frame correlation. To end this, we propose a Bidirectional Temporal and Frame-Segment Attention Module (FSAM). Specifically, we propose an additional reverse-temporal input stream to enhance frame correlation, learned by fusing bidirectional temporal features. In addition, the proposed FSAM contains a Multi-stage segment-aware GCN and decoder interaction module, aiming to learn the correlation between segment features across time domains and integrate embeddings between frame and segment representations. To evaluate our approach, we propose the Figure Skating Sparse Action Segmentation (FSSAS) dataset: The dataset comprises 100 samples of the Olympic figure skating final and semi-final competition, with more than 50 different men and women athletes. Extensive experiments show that our method achieves an accuracy of 87.75 and an edit score of 90.18 on the FSSAS dataset.