Spatial-temporal Joint Research Articles

Spatial–Temporal Joint Design and Optimization of Phase-Coded Waveform for MIMO Radar

By simultaneously transmitting multiple different waveform signals, a multiple-input multiple-output (MIMO) radar possesses higher degrees of freedom and potential in many aspects compared to a traditional phased-array radar. The spatial–temporal characteristics of waveforms are the key to determining their performance. In this paper, a transmitting waveform design method based on spatial–temporal joint (STJ) optimization for a MIMO radar is proposed, where waveforms are designed not only for beam-pattern matching (BPM) but also for minimizing the autocorrelation sidelobes (ACSLs) of the spatial synthesis signals (SSSs) in the directions of interest. Firstly, the STJ model is established, where the two-step strategy and least squares method are utilized for BPM, and the L2p-Norm of the ACSL is constructed as the criterion for temporal characteristics optimization. Secondly, by transforming it into an unconstrained optimization problem about the waveform phase and using the gradient descent (GD) algorithm, the hard, non-convex, high-dimensional, nonlinear optimization problem is solved efficiently. Finally, the method’s effectiveness is verified through numerical simulation. The results show that our method is suitable for both orthogonal and partial-correlation MIMO waveform designs and efficiently achieves better spatial–temporal characteristic performances simultaneously in comparison with existing methods.

Traffic prediction based on spatial-temporal disentangled generative models

Understanding the underlying spatial-temporal patterns in a traffic network is crucial for accurate predictions. However, current deep learning techniques often struggle to effectively identify the generative factors hidden in spatial-temporal traffic data, leading to a lack of interpretability. To address this challenge, we have developed an innovative generative model called Spatial-Temporal Disentangled Neural Relational Inference (STDNRI) for traffic prediction and disentangling interpretable generative factors within observed traffic network data. In this study, we assume that spatial-temporal data is dominated by three generative factors: time-correlated factors, spatial-correlated factors and spatial-temporal joint correlation factors. Based on information bottleneck theory, we establish a novel objective function to maximize the disentanglement of different factors. To model the probability distributions of three factors, we propose three graph neural network encoders to represent time-correlated factors using a graph containing only self-loops, and denote spatial-correlated factors as well as spatial-temporal joint correlation factors adopting a fully-connected graph without self-loops. Then, graph neural network decoder with a spatial-temporal message passing mechanism is used to generate the final predictions. Several comprehensive experiments were conducted based on two real datasets from NYC Yellow Taxi and SHMetro. The results indicate that our STDNRI outperforms ten existing competitive models in terms of high prediction accuracy and provides excellent interpretability for generative factors. Furthermore, the disentangled interpretable factors can afford valuable insights for planners, urban developers, and other stakeholders to gain a deeper understanding of the dynamics of the traffic network, thus enabling more informed decision-making.

Spatial–Temporal Analysis-Based Video Quality Assessment: A Two-Stream Convolutional Network Approach

In system processing, video inevitably suffers from distortion, which leads to quality degradation and affects the user experience. Therefore, it is of great importance to design an accurate and effective objective video quality assessment (VQA) method. In this paper, by considering the multi-dimensional characteristics for video and visual perceptual mechanism, a two-stream convolutional network for VQA is proposed based on spatial–temporal analysis, named TSCNN-VQA. Specifically, for feature extraction, TSCNN-VQA first extracts spatial and temporal features by two different convolutional neural network branches, respectively. After that, the spatial–temporal joint feature fusion is constructed to obtain the joint spatial–temporal features. Meanwhile, the TSCNN-VQA also integrates an attention module to guarantee that the process conforms to the mechanism that the visual system perceives video information. Finally, the overall quality is obtained by non-linear regression. The experimental results in both the LIVE and CSIQ VQA datasets show that the performance indicators obtained by TSCNN-VQA are higher than those of existing VQA methods, which demonstrates that TSCNN-VQA can accurately evaluate video quality and has better consistency with the human visual system.

STA-Net: A Spatial–Temporal Joint Attention Network for Driver Maneuver Recognition, Based on In-Cabin and Driving Scene Monitoring

Next-generation advanced driver-assistance systems (ADASs) are a promising direction for intelligent transportation systems. To achieve intelligent security monitoring, it is imperative that vehicles possess the ability to accurately comprehend driver maneuvers amidst diverse driver behaviors and complex driving scenarios. Existing CNN-based and transformer-based driver maneuver recognition methods face challenges in effectively capturing global and local features across temporal and spatial dimensions. This paper proposes a Spatial–Temporal Joint Attention Network (STA-Net) to realize high-efficient temporal and spatial feature extractions in driver maneuver recognition. First, we introduce a two-stream architecture for a concurrent analysis of in-cabin driver behaviors and out-cabin environmental information. Second, we propose a Multi-Scale Transposed Attention (MSTA) module and Multi-Scale Feedforward Network (MSFN) to extract features at multiple scales, addressing receptive field inadequacies and combining high-level and low-level information. Third, to address the information redundancy in multi-scale features, we propose a Cross-Spatial Attention Module (CSAM) and Multi-Scale Cross-Spatial Fusion Module (MCFM) to select essential features. Additionally, we introduce an asymmetric loss function to effectively tackle the issue of sample imbalance across diverse categories of driving maneuvers. The proposed method demonstrates a remarkable accuracy of 90.97% and an F1 score of 89.37% on the Brain4Cars dataset, surpassing the performance of the methods compared. These results substantiate the fact that our approach effectively enhances driver maneuver recognition.

Attention Mechanism with Spatial-Temporal Joint Deep Learning Model for the Forecasting of Short-Term Passenger Flow Distribution at the Railway Station

Accurate understanding of passenger flow distribution is crucial for effective station crowd management. However, due to the complexity and randomness of passenger flow and the unclear spatial-temporal correlation between functional areas within the station, predicting the spatiotemporal distribution dynamics of inflow and future short-term distribution trends is challenging. Emerging deep learning models offer valuable insights for accurately predicting passenger flow distribution. Thus, we propose a deep learning architecture, named “ST-Bi-LSTM,” which combines a bidirectional long short-term memory network with a spatial-temporal attention mechanism. Initially, we outline the methodologies of Bi-LSTM, the DeepWalk-based spatial attention mechanism, and the temporal attention mechanism. The spatial attention mechanism is employed to extract station spatial network topology information and enhance the representation of passenger flow characteristics in highly correlated areas during the forecasting process. Simultaneously, the temporal attention Bi-LSTM is utilized for capturing temporal correlations. The architecture comprises four branches dedicated to station real-time video monitoring data, spatial network topology, function area attributes, and train timetables. Subsequently, leveraging in-station CCTV data, passenger travel behavior data, and train timetables, we apply the architecture to the Tianjin West High-Speed Railway Station. We conduct a comparative analysis of the prediction performance and time complexity of the proposed architecture against existing baseline models, demonstrating superior performance and robustness exhibited by the ST-Bi-LSTM model (achieving a reduction in RMSE of over 10%). This study facilitates the transition of station management from passive response to active prediction of station passenger flow dynamics.

Skeleton action recognition via graph convolutional network with self-attention module

<abstract><p>Skeleton-based action recognition is an important but challenging task in the study of video understanding and human-computer interaction. However, existing methods suffer from two deficiencies. On the one hand, most methods usually involve manually designed convolution kernel which cannot capture spatial-temporal joint dependencies of complex regions. On the other hand, some methods just use the self-attention mechanism, ignoring its theoretical explanation. In this paper, we proposed a unified spatio-temporal graph convolutional network with a self-attention mechanism (SA-GCN) for low-quality motion video data with fixed viewing angle. SA-GCN can extract features efficiently by learning weights between joint points of different scales. Specifically, the proposed self-attention mechanism is end-to-end with mapping strategy for different nodes, which not only characterizes the multi-scale dependencies of joints, but also integrates the structural features of the graph and an ability of self-learning fusion features. Moreover, the attention mechanism proposed in this paper can be theoretically explained by GCN to some extent, which is usually not considered in most existing models. Extensive experiments on two widely used datasets, NTU-60 RGB+D and NTU-120 RGB+D, demonstrated that SA-GCN significantly outperforms a series of existing mainstream approaches in terms of accuracy.</p></abstract>

Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint

INTRODUCTION: Video inpainting is a very important task in computer vision, and it’s a key component of various practical applications. It also plays an important role in video occlusion removal, traffic monitoring and old movie restoration technology. Video inpainting is to obtain reasonable content from the video sequence to fill the missing region, and maintain time continuity and spatial consistency.OBJECTIVES: In previous studies, due to the complexity of the scene of video inpainting, there are often cases of fast motion of objects in the video or motion of background objects, which will lead to optical flow failure. So the current video inpainting algorithm hasn’t met the requirements of practical applications. In order to avoid the problem of optical flow failure, this paper proposes a transformer-guided video inpainting model based on local Spatial-temporal joint.METHODS: First, considering the rich Spatial-temporal relationship between local flows, a Local Spatial-Temporal Joint Network (LSTN) including encoder, decoder and transformer module is designed to roughly inpaint the local corrupted frames, and the Deep Flow Network is used to calculate the local bidirectional corrupted flows. Then, the local corrupted optical flow map is input into the Local Flow Completion Network (LFCN) with pseudo 3D convolution and attention mechanism to obtain a complete set of bidirectional local optical flow maps. Finally, the roughly inpainted local frame and the complete bidirectional local optical flow map are sent to the Spatial-temporal transformer and the inpainted video frame is output.RESULTS: Experiments show that the algorithm achieves high quality results in the video target removal task, and has a certain improvement in indicators compared with advanced technologies.CONCLUSION: Transformer-Guided Video Inpainting Algorithm Based on Local Spatial-Temporal joint can obtain high-quality optical flow information and inpainted result video.

JAMSNet: A Remote Pulse Extraction Network Based on Joint Attention and Multi-Scale Fusion

Remote photoplethysmography (rPPG) has been an active research topic in recent years. While most existing methods are focusing on eliminating motion artifacts in the raw traces obtained from single-scale region-of-interest (ROI), it is worth noting that there are some noise signals that cannot be effectively separated in single-scale space but can be separated more easily in multi-scale space. In this paper, we analyze the distribution of pulse signal and motion artifacts in different layers of a Gaussian pyramid. We propose a method that combines multi-scale analysis and neural network for pulse extraction in different scales, and a layer-wise attention mechanism to adaptively fuse the features according to signal strength. In addition, we propose spatial-temporal joint attention module and channel-temporal joint attention module to learn and exaggerate pulse features in the joint spaces, respectively. The proposed remote pulse extraction network is called Joint Attention and Multi-Scale fusion Network (JAMSNet). Extensive experiments have been conducted on two publicly available datasets and one self-collected dataset. The results show that the proposed JAMSNet shows better performance than state-of-the-art methods.

A Spatial–Temporal Joint Radar-Communication Waveform Design Method with Low Sidelobe Level of Beampattern

A joint radar-communication (JRC) system utilizes the integrated transmit waveform and a single platform to perform radar and communication functions simultaneously. Admittedly, the multibeam waveform design approach could transmit the assigned waveforms in different beams with the aid of spatial and temporal degrees of freedom. However, a high sidelobe level (SLL) in the beampattern reduces energy efficiency and expands exposure probability. In this study, we propose a novel spatial–temporal joint waveform design method based on the beamforming algorithm to form a low SLL beampattern. Waveform synthesis constraints are considered to synthesize desired radar and communication waveforms at designated directions. Furthermore, we impose the constant modulus constraint to lessen the impact of the high peak-to-average ratio (PAPR). The optimization process of the whole model can be summarized as two stages. First, the covariance matrix is created by convex optimization with respect to the minimum SLL. Second, the integrated transmit waveform is tuned through an alternating projection algorithm. Based on the simulation findings, we demonstrate that the proposed method outperforms the traditional methods in terms of low SLL and waveform synthesis. Meanwhile, we validate the effectiveness of the proposed method using semi-physical experiment results.

Attention Mechanism With Spatial-Temporal Joint Model for Traffic Flow Speed Prediction

Intelligent transportation system (ITS) plays an important role in solving today’s transportation problems, and short-term traffic flow prediction is at its core. Deep learning can extract and capture abstract high-order features, and introducing attention mechanism to improve the performance of deep learning algorithm has been verified in many fields. Due to the complexity and randomness of traffic flow, accurate traffic flow prediction is not a simple task. Reasonable use of deep learning to predict traffic flow is of great significance to the whole transportation system. In this paper, the reason of choosing recurrent neural network (RNN) as the basic network for traffic flow prediction is explained. Aiming at the problem of gradient disappearance in practical application, the long short-term memory network (LSTM) is introduced to improve the model, and the model framework, algorithm and training process are described in detail. Attention mechanism is introduced into LSTM-RNN to build a short-term traffic flow prediction model. Applying the proposed model to observed traffic flow data, we found that the proposed model has higher prediction accuracy and model efficiency.

Swarm of micro flying robots in the wild.

Aerial robots are widely deployed, but highly cluttered environments such as dense forests remain inaccessible to drones and even more so to swarms of drones. In these scenarios, previously unknown surroundings and narrow corridors combined with requirements of swarm coordination can create challenges. To enable swarm navigation in the wild, we develop miniature but fully autonomous drones with a trajectory planner that can function in a timely and accurate manner based on limited information from onboard sensors. The planning problem satisfies various task requirements including flight efficiency, obstacle avoidance, and inter-robot collision avoidance, dynamical feasibility, swarm coordination, and so on, thus realizing an extensible planner. Furthermore, the proposed planner deforms trajectory shapes and adjusts time allocation synchronously based on spatial-temporal joint optimization. A high-quality trajectory thus can be obtained after exhaustively exploiting the solution space within only a few milliseconds, even in the most constrained environment. The planner is finally integrated into the developed palm-sized swarm platform with onboard perception, localization, and control. Benchmark comparisons validate the superior performance of the planner in trajectory quality and computing time. Various real-world field experiments demonstrate the extensibility of our system. Our approach evolves aerial robotics in three aspects: capability of cluttered environment navigation, extensibility to diverse task requirements, and coordination as a swarm without external facilities.

Local Spatial-Temporal Matching Method for Space-Based Infrared Aerial Target Detection.

The feature of a space-based infrared signal is that the intensity of clutter is much stronger than that of an aerial target. Such a feature poses a great challenge to aerial target detection since the existing infrared target detection methods are prone to enhance clutter but ignore the real target, which results in missed detection and false alarms. To tackle the challenge, we propose a concise method based on local spatial–temporal matching (LSM). Specifically, LSM mainly consists of local normalization, local direction matching, spatial–temporal joint model, and inverse matching. Local normalization aims to enhance the target to the same strength as the clutter, so that the weak target will not be ignored. After normalization, a direction-matching step is applied to estimate the moving direction of the background between the basic frame and referenced frame. Then the spatial–temporal joint model is constructed to enhance the target and suppress strong clutter. Similarly, inverse matching is conducted to further enhance the target. Finally, a salience map is obtained, on which the aerial target is extracted by the adaptive threshold segmentation. Experiments conducted on four space-based infrared datasets indicate that LSM handles the above challenge and outperforms seven state-of-the-art methods in space-based infrared aerial target detection.

Security-constrained economic dispatch exploiting the operational flexibility of transmission networks

Dynamic line rating (DLR) and flexible ac transmission system (FACTS) are two technologies to enhance the utilization of existing transmission lines and exploit the inherent transmission-side flexibility. This paper coordinates DLR and FACTS in a two-stage security-constrained economic dispatch (SCED) model to enhance the power system flexibility in N-K contingency management and reduce operational cost. To utilize DLR while handling its uncertainty, spatial–temporal joint chance constraints (JCCs) are used. A scalable reformulation method for these JCCs is developed, which enables considering the spatial–temporal correlations between DLRs and co-optimizing the DLR utilization with system dispatch. To leverage the flexibility of FACTS, the adjustments of FACTS set points before and after the occurrence of any N-K line outage are modelled. The proposed SCED model is reformulated as a triple-level robust model, which avoids enumerating all possible N-K outages in the SCED model and obtains a more reasonable model size. To handle the presence of binary variables in the lowest level of the robust model, a nested column-and-constraint generation solution structure is designed. Case studies on IEEE 14 and 118-bus systems are conducted to verify the effectiveness of the proposed SCED model.

Spatial-temporal joint zinc flotation froth image denoising based on bubble dynamic features

The environment of the flotation plant is harsh, and the noise pollution of the collected froth images is severe. It is challenging to obtain real-time and high-quality froth images by common denoising methods. A spatial-temporal joint denoising method based on bubble dynamic features is proposed. This method first analyses the dynamic features of the bubble and detects the stability of each reference image sub-blocks; then selects the corresponding filtering to denoising according to the different states of the froth sub-blocks; and finally obtains a weighted average result of temporal filtering and spatial filtering. This method uses a modified and noise-robust cross-correlation method to estimate the sub-blocks' motion, which improves the denoising performance of time-domain filtering. The experiments show that the method can obtain a high signal-to-noise ratio and strong structural similarity result for non-static background images. It has also been well verified in the application of zinc flotation froth images.

Spatial‐temporal slowfast graph convolutional network for skeleton‐based action recognition

In skeleton-based action recognition, the graph convolutional network (GCN) has achieved great success. Modelling skeleton data in a suitable spatial-temporal way and designing the adjacency matrix are crucial aspects for GCN-based methods to capture joint relationships. In this study, we propose the spatial-temporal slowfast graph convolutional network (STSF-GCN) and design the adjacency matrices for the skeleton data graphs in STSF-GCN. STSF-GCN contains two pathways: (1) the fast pathway is in a high frame rate, and joints of adjacent frames are unified to build ‘small’ spatial-temporal graphs. A new spatial-temporal adjacency matrix is proposed for these ‘small’ spatial-temporal graphs. Ablation studies verify the effectiveness of the proposed adjacency matrix. (2) The slow pathway is in a low frame rate, and joints from all frames are unified to build one ‘big’ spatial-temporal graph. The adjacency matrix for the ‘big’ spatial-temporal graph is obtained by computing self-attention coefficients of each joint. Finally, outputs from two pathways are fused to predict the action category. STSF-GCN can efficiently capture both long-range and short-range spatial-temporal joint relationships. On three datasets for skeleton-based action recognition, STSF-GCN can achieve state-of-the-art performance with much less computational cost.

A Survey on OD Data Visualization

<p indent=0mm>OD data is a type of data that provides origin and destination positions, and sparsely describes the movement trajectory of objects. Being the research hotspots in data visualization and geospatial analysis, visual analysis of OD data contributes to discovering group movement patterns, exploring spatial anomalies and hidden relationships, and in-depth analysis of various statistics attributes. It is started from the pair-point structure of OD data in geospatial space, and classical OD data visualization methods and principles are elaborated, such as OD matrix, OD map, OD flow, etc. Then from the perspective of spatial-temporal feature exploration, spatial and spatial-temporal joint dimensions, the visual analysis of OD data is performed, while its application scenarios and enhanced significance of feature expression, perception, extraction and presentation are researched. To solve the problem of visual clutter in large-scale OD data visualization, the visual simplification methods such as filtering, aggregation, and sampling are sorted out in the research methods of visual analysis of OD data. Finally, the applications and specific cases of OD data visual analysis technology are investigated in different domains, the challenging problems in the research of OD data visual analysis are summarized, and the trend of its future development is prospected.

Model-based spatial-temporal mapping of opisthorchiasis in endemic countries of Southeast Asia.

Opisthorchiasis is an overlooked danger to Southeast Asia. High-resolution disease risk maps are critical but have not been available for Southeast Asia. Georeferenced disease data and potential influencing factor data were collected through a systematic review of literatures and open-access databases, respectively. Bayesian spatial-temporal joint models were developed to analyze both point- and area-level disease data, within a logit regression in combination of potential influencing factors and spatial-temporal random effects. The model-based risk mapping identified areas of low, moderate, and high prevalence across the study region. Even though the overall population-adjusted estimated prevalence presented a trend down, a total of 12.39 million (95% Bayesian credible intervals [BCI]: 10.10-15.06) people were estimated to be infected with O. viverrini in 2018 in four major endemic countries (i.e., Thailand, Laos, Cambodia, and Vietnam), highlighting the public health importance of the disease in the study region. The high-resolution risk maps provide valuable information for spatial targeting of opisthorchiasis control interventions.

Spatial-temporal joint zinc flotation froth image denoising based on bubble dynamic features

Spatial-temporal joint zinc flotation froth image denoising based on bubble dynamic features

First- And Third-Person Video Co-Analysis By Learning Spatial-Temporal Joint Attention.

Recent years have witnessed a tremendous increase of first-person videos captured by wearable devices. Such videos record information from different perspectives than the traditional third-person view, and thus show a wide range of potential usages. However, techniques for analyzing videos from different views can be fundamentally different, not to mention co-analyzing on both views to explore the shared information. In this paper, we take the challenge of cross-view video co-analysis and deliver a novel learning-based method. At the core of our method is the notion of "joint attention", indicating the shared attention regions that link the corresponding views, and eventually guide the shared representation learning across views. To this end, we propose a multi-branch deep network, which extracts cross-view joint attention and shared representation from static frames with spatial constraints, in a self-supervised and simultaneous manner. In addition, by incorporating the temporal transition model of the joint attention, we obtain spatial-temporal joint attention that can robustly capture the essential information extending through time. Our method outperforms the state-of-the-art on the standard cross-view video matching tasks on public datasets. Furthermore, we demonstrate how the learnt joint information can benefit various applications through a set of qualitative and quantitative experiments.

Compressive Subspace Learning With Antenna Cross-Correlations for Wideband Spectrum Sensing

Compressive subspace learning (CSL) with the exploitation of space diversity has found a potential performance improvement for wideband spectrum sensing (WBSS). However, previous works mainly focus on either exploiting antenna auto-correlations or adopting a multiple-input multiple-output (MIMO) channel without considering the spatial correlations, which will degrade their performances. In this paper, we consider a spatially correlated MIMO channel and propose two CSL algorithms (i.e., mCSLSACC and vCSLACC) which exploit antenna cross-correlations, where the mCSLSACC utilizes an antenna averaging temporal decomposition, and the vCSLACC uses a spatial-temporal joint decomposition. For both algorithms, the conditions of statistical covariance matrices (SCMs) without noise corruption are derived. Through establishing the singular value relation of SCMs in statistical sense between the proposed and traditional CSL algorithms, we show the superiority of the proposed CSL algorithms. By further depicting the receiving correlation matrix of MIMO channel with the exponential correlation model, we give important closed-form expressions for the proposed CSL algorithms in terms of the amplification of singular values over traditional CSL algorithms. Such expressions provide a possibility to determine optimal algorithm parameters for high system performances in an analytical way. Simulations validate the correctness of this work and its performance improvement over existing works in terms of WBSS performance.