Past Frames Research Articles

Scalable video transformer for full-frame video prediction

Vision Transformers (ViTs) have shown success in many low-level computer vision tasks. However, existing ViT models are limited by their high computation and memory cost when generating high-resolution videos for tasks like video prediction. This paper presents a scalable video transformer for full-frame video predication. Specifically, we design a backbone transformer block for our video transformer. This transformer block decouples the temporal and channel features to reduce the computation cost when processing large-scale spatial–temporal video features. We use transposed attention to focus on the channel dimension instead of the spatial window to further reduce the computation cost. We also design a Global Shifted Multi-Dconv Head Transposed Attention module (GSMDTA) for our transformer block. This module is built upon two key ideas. First, we design a depth shift module to better incorporate the cross-channel or temporal information from video features. Second, we introduce a global query mechanism to capture global information to handle large motion for video prediction. This new transformer block enables our video transformer to predict a full frame from multiple past frames at the resolution of 1024 × 512 with 12 GB VRAM. Experiments on various video prediction benchmarks demonstrate that our method with only RGB input outperforms state-of-the-art methods that require additional data, like segmentation maps and optical flows. Our method exceeds the state-of-the-art RGB-only methods by a large margin (1.2 dB) in PSNR. Our method is also faster than state-of-the-art video prediction transformers.

Video object detection via space–time feature aggregation and result reuse

AbstractWhen detecting the objects in videos, motion always leads to object deterioration, like blurring and occlusion, as well as the strange state of the object's shape and posture. Consequently, the detection of video frames will lead to a decline in accuracy by using the image object detection model. This paper proposes an online video object detection method based on the one‐stage detector YOLOx. First, the module for space–time feature aggregation is given, which uses the space–time information of past frames to enhance the feature quality of the current frame. Then, the module for result reuse is given, which incorporates the detection results of past frames to improve the detection stability of the current frame. By these two modules, the trade‐off between accuracy and speed of video object detection could be achieved. Experimental results on the ImageNet VID show the improvement of speed and accuracy of the proposed method.

Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations

Spatiotemporal predictive learning is a paradigm that empowers models to learn spatial and temporal patterns by predicting future frames from past frames in an unsupervised manner. This method typically uses recurrent units to capture long-term dependencies, but these units often come with high computational costs and limited performance in real-world scenes. This paper presents an innovative Wavelet-based SpatioTemporal (WaST) framework, which extracts and adaptively controls both low and high-frequency components at image and feature levels via 3D discrete wavelet transform for faster processing while maintaining high-quality predictions. We propose a Time-Frequency Aware Translator uniquely crafted to efficiently learn short- and long-range spatiotemporal information by individually modeling spatial frequency and temporal variations. Meanwhile, we design a wavelet-domain High-Frequency Focal Loss that effectively supervises high-frequency variations. Extensive experiments across various real-world scenarios, such as driving scene prediction, traffic flow prediction, human motion capture, and weather forecasting, demonstrate that our proposed WaST achieves state-of-the-art performance over various spatiotemporal prediction methods.

Generalizable Fourier Augmentation for Unsupervised Video Object Segmentation

The performance of existing unsupervised video object segmentation methods typically suffers from severe performance degradation on test videos when tested in out-of-distribution scenarios. The primary reason is that the test data in real- world may not follow the independent and identically distribution (i.i.d.) assumption, leading to domain shift. In this paper, we propose a generalizable fourier augmentation method during training to improve the generalization ability of the model. To achieve this, we perform Fast Fourier Transform (FFT) over the intermediate spatial domain features in each layer to yield corresponding frequency representations, including amplitude components (encoding scene-aware styles such as texture, color, contrast of the scene) and phase components (encoding rich semantics). We produce a variety of style features via Gaussian sampling to augment the training data, thereby improving the generalization capability of the model. To further improve the cross-domain generalization performance of the model, we design a phase feature update strategy via exponential moving average using phase features from past frames in an online update manner, which could help the model to learn cross-domain-invariant features. Extensive experiments show that our proposed method achieves the state-of-the-art performance on popular benchmarks.

An Anarchist Archaeology of Equality: Pasts and Futures Against Hierarchy

Scholars of the past frame the ‘origins’ or evolution of inequality, usually using archaeological or anthropological evidence as a basis for their arguments, as an intentional, inevitable, important step towards the development of states, implicitly framed as the pinnacle of human political and economic achievement. Anarchist archaeologies reject the idea of hierarchy as a positive or inevitable evolutionary outcome underlying the path to civilization. We argue instead for a radical reorientation towards archaeologies of equality. We propose a prefigurative archaeology that celebrates the myriad ways that human beings have actively undermined and resisted hierarchical social arrangements. We aim to reorient archaeology's focus towards societies that purposefully prevented or constrained the emergence of inequality. To demonstrate the potential of archaeologies of equality we present case examples from Oceania, Britain, West Asia and the American Southwest. Highlighting the accomplishments of societies of equals in the past demonstrates the contingency and problematic nature of present forms of inequality. It allows us to explore a different set of pasts and thus enact different presents as we imagine different futures.

Cascaded Inner-Outer Clip Retformer for Ultrasound Video Object Segmentation.

Computer-aided ultrasound (US) imaging is an important prerequisite for early clinical diagnosis and treatment. Due to the harsh ultrasound (US) image quality and the blurry tumor area, recent memory-based video object segmentation models (VOS) achieve frame-level segmentation by performing intensive similarity matching among the past frames which could inevitably result in computational redundancy. Furthermore, the current attention mechanism utilized in recent models only allocates the same attention level among whole spatial-temporal memory features without making distinctions, which may result in accuracy degradation. In this paper, we first build a larger annotated benchmark dataset for breast lesion segmentation in ultrasound videos, then we propose a lightweight clip-level VOS framework for achieving higher segmentation accuracy while maintaining the speed. The Inner-Outer Clip Retformer is proposed to extract spatialtemporal tumor features in parallel. Specifically, the proposed Outer Clip Retformer extracts the tumor movement feature from past video clips to locate the current clip tumor position, while the Inner Clip Retformer detailedly extracts current tumor features that can produce more accurate segmentation results. Then a Clip Contrastive loss function is further proposed to align the extracted tumor features along both the spatial-temporal dimensions to improve the segmentation accuracy. In addition, the Global Retentive Memory is proposed to maintain the complementary tumor features with lower computing resources which can generate coherent temporal movement features. In this way, our model can significantly improve the spatial-temporal perception ability without increasing a large number of parameters, achieving more accurate segmentation results while maintaining a faster segmentation speed. Finally, we conduct extensive experiments to evaluate our proposed model on several video object segmentation datasets, the results show that our framework outperforms state-of-theart segmentation methods.

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation.

Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Efficient Long-Short Temporal Attention network for unsupervised Video Object Segmentation

Unsupervised Video Object Segmentation (VOS) aims at identifying the contours of primary foreground objects in videos without any prior knowledge. However, previous methods do not fully use spatial–temporal context and fail to tackle this challenging task in real-time. This motivates us to develop an efficient Long-Short Temporal Attention network (termed LSTA) for unsupervised VOS task from a holistic view. Specifically, LSTA consists of two dominant modules, i.e., Long Temporal Memory and Short Temporal Attention. The former captures the long-term global pixel relations of the past frames and the current frame, which models constantly present objects by encoding appearance pattern. Meanwhile, the latter reveals the short-term local pixel relations of one nearby frame and the current frame, which models moving objects by encoding motion pattern. To speedup the inference, the efficient projection and the locality-based sliding window are adopted to achieve nearly linear time complexity for the two light modules, respectively. Extensive empirical studies on several benchmarks have demonstrated promising performances of the proposed method with high efficiency.

Video anomaly detection based on cross-frame prediction mechanism and spatio-temporal memory-enhanced pseudo-3D encoder

Intelligent video anomaly detection (VAD) methods play a crucial role in conserving human resources, reducing the financial burden on governments, and promptly and accurately identifying abnormal behaviors. Frame prediction, which performs VAD by reasonably predicting normal video and distorting predicted anomalous video, is a popular and efficient method. Although Auto-Encoders (AE) show excellent performance in video frame prediction methods, the ability of these methods to use temporal information and poorly reconstruct anomalous videos is insufficient. To improve the shortcomings of AE in VAD, we propose a VAD method based on the cross-frame prediction mechanism and the spatio-temporal memory-enhanced pseudo-3D encoder. This method significantly enhances the recognition capability of abnormal activities in surveillance videos. Firstly, we use the prediction mechanism that uses multiple past frames with intervals to predict future frames. It can broaden the extra information and limit the memory consumption. Then we design the pseudo-3D encoder to encode the spatio-temporal information in videos, avoiding the problems that the 2D encoder cannot obtain the temporal dimensional information and the 3D encoder has complicated structure and overmuch parameters. Finally, we design the spatio-temporal memory block with three loss functions to store the spatio-temporal information of normal videos, which can expand the predicted differences between normal and abnormal examples. Experiments on UCSD Ped2, CUHK Avenue and ShanghaiTech datasets experimentally show that the proposed method achieves 99.4%, 90.5% and 74.3% of the AUC values. Our method shows excellent performance among single-stage semi-supervised anomaly detection methods.

TSDTVOS: Target-guided spatiotemporal dual-stream transformers for video object segmentation

Video object segmentation automatically separates the interested objects from the background across a video sequence and was an active research area in recent years. The crucial challenge lies in investigating an effective architecture to fully exploit spatiotemporal correlation in a given video sequence for achieving accurate segmentation results. In this paper, we propose a novel semi-supervised Transformer-based framework called Target-guided Spatiotemporal Dual-stream Transformers (TSDT) with two separate streams to enable effective spatiotemporal context propagation. Technically, the temporal stream is used to aggregate rich temporal cues from past frames, while the spatial stream is trained to encode object location and appearance information stored in the current frame. To compress and integrate temporal features, a target guidance block (TGB) is designed to retrieve target information in the past video flow under the guidance of the current frame. The experimental results on video object segmentation benchmarks demonstrate the feasibility and effectiveness of the proposed framework. Codes and trained models are available at https://github.com/zhouweii234/TSDTVOS.

Online Multi-Face Tracking With Multi-Modality Cascaded Matching

Tracking multiple faces online in unconstrained videos is a challenging problem as faces may appear drastically different over time and identities can be inferred only based on information available from past frames. Previous tracking methods focus on face information without reference to other modality information such as a person’s overall body appearance, leading to suboptimal performance. In this paper, we propose a new online multi-face tracking method, called online multi-face tracking with multi-modality cascaded matching (OMTMCM), to improve the tracking performance by using both face and body information. The proposed OMTMCM consists of two stages, namely detection alignment and detection association. In the first stage, a detection alignment module is designed to align face detection with body detection from the same person for the subsequent detection association. In the second stage, a cascaded matching module is designed to associate face detections across frames to locate trajectory of each target face by using both face and body information. Specifically, aligned face-body detections in the current frame are matched in a cascade manner with body and face features that are selected from past frames and stored in the designed feature memory. In this way, our method can track multiple faces online with both face and body information while eliminating the possibility of face detection and body detection from the same person being separately assigned with different identities. Experimental results demonstrate our method is on par with or better than other online tracking methods for multi-face tracking.

Rendezvous in time: an attention-based temporal fusion approach for surgical triplet recognition.

One of the recent advances in surgical AI is the recognition of surgical activities as triplets of [Formula: see text]instrument, verb, target[Formula: see text]. Albeit providing detailed information for computer-assisted intervention, current triplet recognition approaches rely only on single-frame features. Exploiting the temporal cues from earlier frames would improve the recognition of surgical action triplets from videos. In this paper, we propose Rendezvous in Time (RiT)-a deep learning model that extends the state-of-the-art model, Rendezvous, with temporal modeling. Focusing more on the verbs, our RiT explores the connectedness of current and past frames to learn temporal attention-based features for enhanced triplet recognition. We validate our proposal on the challenging surgical triplet dataset, CholecT45, demonstrating an improved recognition of the verb and triplet along with other interactions involving the verb such as [Formula: see text]instrument, verb[Formula: see text]. Qualitative results show that the RiT produces smoother predictions for most triplet instances than the state-of-the-arts. We present a novel attention-based approach that leverages the temporal fusion of video frames to model the evolution of surgical actions and exploit their benefits for surgical triplet recognition.

MUNet: Motion uncertainty-aware semi-supervised video object segmentation

The task of semi-supervised video object segmentation (VOS) has been greatly advanced and state-of-the-art performance has been made by dense matching-based methods. The recent methods leverage space-time memory (STM) networks and learn to retrieve relevant information from all available sources, where the past frames with object masks form an external memory and the current frame as the query is segmented using the mask information in the memory. However, when forming the memory and performing matching, these methods only exploit the appearance information while ignoring the motion information. In this paper, we advocate for the return of the motion information and propose a motion uncertainty-aware framework (MUNet) for semi-supervised VOS. First, we propose an implicit method to learn the spatial correspondences between neighboring frames, building upon a correlation cost volume. To handle the challenging cases of occlusion and textureless regions during constructing dense correspondences, we incorporate the uncertainty in dense matching and achieve motion uncertainty-aware feature representation. Second, we introduce a motion-aware spatial attention module to effectively fuse the motion features with the semantic features. Comprehensive experiments on challenging benchmarks show that using a small amount of data and combining it with powerful motion information can bring a significant performance boost. We achieve 76.5%J&F only using DAVIS17 for training22This result is initialized by the Mask-RCNN-ResNet50 weights pre-trained on COCO dataset. By initialization from ResNet50 pre-trained on ImageNet dataset, we can achieve 75.0%J&F, which is still the SOTA performance., which significantly outperforms the SOTA methods under the low-data protocol. The code and supplementary materials will be available at https://npucvr.github.io/MUNet.

Monocular-Vision-Based Moving Target Geolocation Using Unmanned Aerial Vehicle

This paper develops a framework for geolocating a ground moving target with images taken from an unmanned aerial vehicle (UAV). Unlike the usual moving target geolocation approaches that rely heavily on a laser rangefinder, multiple UAVs, prior information of the target or motion assumptions, the proposed framework performs the geolocation of a moving target with monocular vision and does not have any of the above restrictions. The proposed framework transforms the problem of moving target geolocation to the problem of stationary target geolocation by matching corresponding points. In the process of corresponding point matching, we first propose a Siamese-network-based model as the base model to match corresponding points between the current frame and the past frame. Besides the introduction of a base model, we further designed an enhanced model with two outputs, where a row-ness loss and a column-ness loss are defined for achieving a better performance. For the precision of corresponding point matching, we propose a compensation value, which is calculated from the outputs of the enhanced model and improves the accuracy of corresponding point matching. To facilitate the research on corresponding point matching, we constructed a dataset containing various aerial images with corresponding point annotations. The proposed method is shown to be valid and practical via the experiments in simulated and real environments.

Global video object segmentation with spatial constraint module

We present a lightweight and efficient semi-supervised video object segmentation network based on the space-time memory framework. To some extent, our method solves the two difficulties encountered in traditional video object segmentation: one is that the single frame calculation time is too long, and the other is that the current frame’s segmentation should use more information from past frames. The algorithm uses a global context (GC) module to achieve high-performance, real-time segmentation. The GC module can effectively integrate multi-frame image information without increased memory and can process each frame in real time. Moreover, the prediction mask of the previous frame is helpful for the segmentation of the current frame, so we input it into a spatial constraint module (SCM), which constrains the areas of segments in the current frame. The SCM effectively alleviates mismatching of similar targets yet consumes few additional resources. We added a refinement module to the decoder to improve boundary segmentation. Our model achieves state-of-the-art results on various datasets, scoring 80.1% on YouTube-VOS 2018 and a {cal J}{rm{& }}{cal F} score of 78.0% on DAVIS 2017, while taking 0.05 s per frame on the DAVIS 2016 validation dataset.

Multiple object tracking with appearance feature prediction and similarity fusion

Object tracking is a crucial research area within the field of intelligent transportation, providing a vital foundation for anomalous behavior analysis and traffic statistics. Although pedestrian detectors have shown impressive results, leading to the advancement of detection-based tracking methods, target association in complex scenarios remains a difficult and less efficient task due to the lack of feature robustness in the presence of partial occlusions. In the proposed tracking method, we extract convolutional features on each entire object and its local blocks, segmented by the superpixel algorithm. Aiming to emphasize the global and local information respectively, the global features for each entire object are extracted from the last layer of the backbone network, while local features are derived from a specific intermediate layer of the backbone network. The association between tracked targets and detected pedestrian candidates relies on fused similarity degrees. Furthermore, we use the transformer’s self-attention mechanism to predict features for the current frame based on the information within past frames, aiming to eliminate the effects of target appearance variations. Additionally, we remove redundant background pixels in the detected rectangles of pedestrian candidates by using a background modeling algorithm. Experimental results demonstrate that the tracker proposed in this paper outperforms other trackers across five publicly available datasets, indicating its effectiveness and potential for further development.

Video Frame Prediction by Joint Optimization of Direct Frame Synthesis and Optical-Flow Estimation

Video prediction is the problem of generating future frames by exploiting the spatiotemporal correlation from the past frame sequence. It is one of the crucial issues in computer vision and has many real-world applications, mainly focused on predicting future scenarios to avoid undesirable outcomes. However, modeling future image content and object is challenging due to the dynamic evolution and complexity of the scene, such as occlusions, camera movements, delay and illumination. Direct frame synthesis or optical-flow estimation are common approaches used by researchers. However, researchers mainly focused on video prediction using one of the approaches. Both methods have limitations, such as direct frame synthesis, usually face blurry prediction due to complex pixel distributions in the scene, and optical-flow estimation, usually produce artifacts due to large object displacements or obstructions in the clip. In this paper, we constructed a deep neural network Frame Prediction Network (FPNet-OF) with multiple-branch inputs (optical flow and original frame) to predict the future video frame by adaptively fusing the future object-motion with the future frame generator. The key idea is to jointly optimize direct RGB frame synthesis and dense optical flow estimation to generate a superior video prediction network. Using various real-world datasets, we experimentally verify that our proposed framework can produce high-level video frame compared to other state-of-the-art framework.

Coherence-aware context aggregator for fast video object segmentation

Semi-supervised video object segmentation (VOS) is a highly challenging problem that has attracted much research attention in recent years. Temporal context plays an important role in VOS by providing object clues from the past frames. However, most of the prevailing methods directly use the predicted temporal results to guide the segmentation of the current frame, while ignoring the coherence of temporal context, which may be misleading and degrade the performance. In this paper, we propose a novel model named Coherence-aware Context Aggregator (CCA) for VOS, which consists of three modules. First, a coherence-aware module (CAM) is proposed to evaluate the coherence of the predicted result of the current frame and then fuses the coherent features to update the temporal context. CAM can determine whether the prediction is accurate, thus guiding the update of the temporal context and avoiding the introduction of erroneous information. Second, we devise a spatio-temporal context aggregation (STCA) module to aggregate the temporal context with the spatial feature of the current frame to learn a robust and discriminative target representation in the decoder part. Third, we design a refinement module to refine the coarse feature generated from the STCA module for more precise segmentation. Additionally, CCA uses a cropping strategy and takes small-size images as input, thus making it computationally efficient and achieving a real-time running speed. Extensive experiments on four challenging benchmarks show that CCA achieves a better trade-off between efficiency and accuracy compared to state-of-the-art methods. The code will be public.

Robust voice activity detection based on weighted average of long-term quadratic Renyi and differential entropies

We propose two novel voice activity detection (VAD) algorithms by employing i) the Rényi entropy estimate, and ii) a weighted combination of Rényi and differential entropies, at each frequency bin of power spectral estimates, by considering overlapping frames in the received signal samples. The Bartlett-Welch method is employed to estimate the power spectrum from a suitably chosen number of past frames in the frequency domain. Exploiting the known fact that a long-term averaging enhances the performance of VAD, we first propose a novel frequency domain long-term quadratic Rényi entropy (FLQRE) feature, which is extracted by summing the estimated Rényi entropy values across the frequency bins over each signal frame. Later, we propose a weighted combination of the frequency domain long-term differential entropy (FLDE) and FLQRE feature for VAD, termed as a weighted quadratic and differential entropy (WQDE) feature. We evaluate the performance of the proposed FLQRE and WQDE VAD schemes, by considering twelve types of noises from NOISEX-92 at five different SNR values, artificially added to speech samples from SWITCHBOARD and TIMIT corpora. We present an extensive performance comparison study to establish the utility of our proposed VAD algorithms over state-of-the-art short- and long-term VAD techniques such as ITU-T G.729B, likelihood ratio test, long-term signal variability, and long-term spectral flatness measure-based algorithms. We show that the proposed algorithms yield the best average accuracy and noise-hit rate under the SWITCHBOARD corpus, and yield a comparable performance under the TIMIT corpus.

Autoregressive Predictive Coding: A Comprehensive Study

We review autoregressive predictive coding (APC), an approach to learn speech representation by predicting a future frame given the past frames. We present three different views of interpreting APC, and provide a historical account to the approach. To study the speech representation learned by APC, we use common speech tasks, such as automatic speech recognition and speaker verification, to demonstrate the utility of the learned representation. In addition, we design a suite of fine-grained tasks, including frame classification, segment classification, fundamental frequency tracking, and duration prediction, to probe the phonetic and prosodic content of the representation. The three views of the APC objective welcome various generalizations and algorithms to learn speech representations. Probing on the suite of fine-grained tasks suggests that APC makes a wide range of high-level speech information accessible in its learned representation.