Representation Of Video Sequences Research Articles

Black-box Attack against Self-supervised Video Object Segmentation Models with Contrastive Loss

Deep learning models have been proven to be susceptible to malicious adversarial attacks, which manipulate input images to deceive the model into making erroneous decisions. Consequently, the threat posed to these models serves as a poignant reminder of the necessity to focus on the model security of object segmentation algorithms based on deep learning. However, the current landscape of research on adversarial attacks primarily centers around static images, resulting in a dearth of studies on adversarial attacks targeting Video Object Segmentation (VOS) models. Given that a majority of self-supervised VOS models rely on affinity matrices to learn feature representations of video sequences and achieve robust pixel correspondence, our investigation has delved into the impact of adversarial attacks on self-supervised VOS models. In response, we propose an innovative black-box attack method incorporating contrastive loss. This method induces segmentation errors in the model through perturbations in the feature space and the application of a pixel-level loss function. Diverging from conventional gradient-based attack techniques, we adopt an iterative black-box attack strategy that incorporates contrastive loss across the current frame, any two consecutive frames, and multiple frames. Through extensive experimentation conducted on the DAVIS 2016 and DAVIS 2017 datasets using three self-supervised VOS models and one unsupervised VOS model, we unequivocally demonstrate the potent attack efficiency of the black-box approach. Remarkably, the J&F metric value experiences a significant decline of up to 50.08% post-attack.

Generative Adversarial Networks for Stochastic Video Prediction With Action Control

The ability of predicting future frames in video sequences, known as video prediction, is an appealing yet challenging task in computer vision. This task requires an in-depth representation of video sequences and a deep understanding of real-word causal rules. Existing approaches for tackling the video prediction problem can be classified into two categories: deterministic and stochastic methods. Deterministic methods lack the ability of generating possible future frames and often yield blurry predictions. On the other hand, although current stochastic approaches can predict possible future frames, their models lack the ability of action control in the sense that they cannot generate the desired future frames conditioned on a specific action. In this paper, we propose new generative adversarial networks (GANs) for stochastic video prediction. Our framework, called VPGAN, employs an adversarial inference model and a cycle-consistency loss function to empower the framework to obtain more accurate predictions. In addition, we incorporate a conformal mapping network structure into VPGAN to enable action control for generating desirable future frames. In this way, VPGAN is able to produce fake videos of an object moving along a specific direction. Experimental results show that the combination of VPGAN with a pre-trained image segmentation model outperforms existing stochastic video prediction methods.

Unsupervised Learning Approach for Abnormal Event Detection in Surveillance Video by Hybrid Autoencoder

Abnormal detection plays an important role in video surveillance. LSTM encoder–decoder is used to learn representation of video sequences and applied for detecting abnormal event in complex environment. The learned representation of LSTM encoder–decoder is learned from encoder, and it is crucial for decoder. However, LSTM encoder–decoder generally fails to account for the global context of the learned representation with a fixed dimension representation. In this paper, we explore a hybrid autoencoder architecture, which not only extracts better spatio-temporal context, but also improves the extrapolate capability of the corresponding decoder by the shortcut connection. The experiment shows that the hybrid model performs better than the state-of-the-art anomaly detection methods in both qualitative and quantitative ways on benchmark datasets.

3D+T dense motion trajectories as kinematics primitives to recognize gestures on depth video sequences

RGB-D sensors have allowed attacking many classical problems in computer vision such as segmentation, scene representations and human interaction, among many others. Regarding motion characterization, typical RGB-D strategies are limited to namely analyze global shape changes and capture scene flow fields to describe local motions in depth sequences. Nevertheless, such strategies only recover motion information among a couple of frames, limiting the analysis of coherent large displacements along time. This work presents a novel strategy to compute 3D+t dense and long motion trajectories as fundamental kinematic primitives to represent video sequences. Each motion trajectory models kinematic words primitives that together can describe complex gestures developed along videos. Such kinematic words were processed into a bag-of-kinematic-words framework to obtain an occurrence video descriptor. The novel video descriptor based on 3D+t motion trajectories achieved an average accuracy of 80% in a dataset of 5 gestures and 100 videos.

Efficient motion estimation using two-bit transform and modified multilevel successive elimination

Reduced bit representations of video sequences are employed in order to mitigate the computational complexity of block matching algorithms. The idea is to explore computationally cheaper matching criteria than the conventional sum of absolute difference or fuzzy theory. Such low cost matching criteria result in substantial time saving and can therefore achieve real time video compression. To reduce the computational burden of such algorithms even further, we used them in orthogonal to the algorithms which reduces the number of search points in the search window. The speedup in computation as a result of using two-bit transform is rectified by imposing a modified version of multilevel successive elimination algorithm in orthogonal form. Experimental results show that the numbers of search points can be reduced by an average of 95% for sequences with low motion content and by an average of 85% for sequences with relatively high motion content. For all the results the same peak-signal-to-noise ratio of the reconstructed frames is maintained as that of the two-bit transform.

Hierarchical and Spatio-Temporal Sparse Representation for Human Action Recognition.

In this paper, we present a novel two-layer video representation for human action recognition employing hierarchical group sparse encoding technique and spatio-temporal structure. In the first layer, a new sparse encoding method named locally consistent group sparse coding (LCGSC) is proposed to make full use of motion and appearance information of local features. LCGSC method not only encodes global layouts of features within the same video-level groups, but also captures local correlations between them, which obtains expressive sparse representations of video sequences. Meanwhile, two kinds of efficient location estimation models, namely an absolute location model and a relative location model, are developed to incorporate spatio-temporal structure into LCGSC representations. In the second layer, action-level group is established, where a hierarchical LCGSC encoding scheme is applied to describe videos at different levels of abstractions. On the one hand, the new layer captures higher order dependency between video sequences; on the other hand, it takes label information into consideration to improve discrimination of videos' representations. The superiorities of our hierarchical framework are demonstrated on several challenging datasets.

A Tensor-Driven Temporal Correlation Model for Video Sequence Classification

The task of video sequence classification plays a critical role in the development of computer vision. Considering this fact, this letter proposes a novel tensor decomposition method called tensor-driven temporal correlation in which general tensors are used as input for video sequence classification. Because distortion and redundancy may exist in the tensor representations of video sequences, we project the original tensor into subspaces spanned by spatial basis matrices in the proposed formulation. Moreover, to better preserve the temporal smoothness between consecutive slices of the tensor, the basis matrices are jointly learned by introducing an autoregressive model. An experiment on the commonly used Cambridge hand-gesture database demonstrates that our proposed method reaches convergence within a small number of iterations during the training stage and achieves promising results compared with state-of-the-art methods.

Type-2 Fuzzy Topic Models for Human Action Recognition

Following the “bag-of-words” representation for video sequences, we propose novel type-2 fuzzy topic models (T2 FTM) to recognize human actions. In traditional topic models (TM) for visual recognition, each video sequence is modeled as a “document” composed of spatial-temporal interest points called visual words. Topic models automatically assign a “topic” label to explain the action category of each word so that each video sequence becomes a mixture of action topics for recognition. Our T2 FTM differs from previous TM in that it uses type-2 fuzzy sets to encode the higher order uncertainty of each topic. We can use the primary membership function (MF) to measure the degree of uncertainty that a document or a visual word belongs to a specific action topic, and use the secondary MF to evaluate the fuzziness of the primary MF itself. In this paper, we implement two T2 FTM: 1) interval T2 FTM with all secondary grades equal one, and 2) vertical-slice T2 FTM with unequal secondary grades based on our prior knowledge. To estimate parameters in T2 FTM, we derive the efficient message-passing algorithms. Experiments on KTH, Weizmann, UCF, and Hollywood2 human action datasets demonstrate that T2 FTM performs better than other state-of-the-art topic models for human action recognition.

Optical flow-motion history image (OF-MHI) for action recognition

The motion history image (MHI) is a global spatiotemporal representation for video sequences. It is computationally very simple and efficient. It has been widely used for many real-time action recognition tasks. However, the conventional MHI assigns a fixed motion strength to each detected foreground point and then updates it with a small constant for the background point. Local body parts with different movement speeds and durations will then have the same intensity in the MHI. Similar actions may generate indistinguishable MHI patterns. In this paper, we propose a new motion history representation that incorporates both optical flow and a revised MHI. The motion strength of each pixel point is adaptively accumulated by the optical flow length at that location. It is then exponentially updated over time. It can better describe local movements of body parts in the global temporal template. The motion duration is implicitly given by the update rate for better description of various actions in the scene. For action classification, a set of training action samples are first collected and form the basis templates. An action sequence is then constructed as the linear combination of the basis templates. The coefficients of the combination give the feature vector. The Euclidean distance is finally used to evaluate the similarity between the feature vectors. Experimental results on the widely used KTH and Weizmann datasets have shown that the proposed scheme yields 100 % recognition rates on both test datasets with a fast processing rate of 47 fps on $$200\times 150$$ images.

Low-Rank Tensor Learning with Discriminant Analysis for Action Classification and Image Recovery

Tensor completion is an important topic in the area of image processing and computer vision research, which is generally built on extraction of the intrinsic structure of the tensor data. Drawing on this fact, action classification, relying heavily on the extracted features of high-dimensional tensors, may indeed benefit from tensor completion techniques. In this paper, we propose a low-rank tensor completion method for action classification, as well as image recovery. Since there may exist distortion and corruption in the tensor representations of video sequences, we project the tensors into a subspace, which contains the invariant structure of the tensors. In order to integrate useful supervisory information for classification, we adopt a discriminant analysis criterion to learn the projection matrices. The resulting multi-variate optimization problem can be effectively solved using the augmented Lagrange multiplier (ALM) algorithm. Experiments demonstrate that our method results with better accuracy compared with some other state-of-the-art low-rank tensor representation learning approaches on the MSR Hand Gesture 3D database and the MSR Action 3D database. By denoising the Multi-PIE face database, our experimental setup testifies the proposed method can also be employed to recover images.

Hierarchical representation of video sequences for annotation

Video annotation is an important issue in video content management systems. Rapid growth of the digital video data has created a need for efficient and reasonable mechanisms that can ease the annotation process. In this paper, we propose a novel hierarchical clustering based system for video annotation. The proposed system generates a top-down hierarchy of the video streams using hierarchical k-means clustering. A tree-based structure is produced by dividing the video recursively into sub-groups, each of which consists of similar content. Based on the visual features, each node of the tree is partitioned into its children using k-means clustering. Each sub-group is then represented by its key frame, which is selected as the closest frame to the centroids of the corresponding cluster, and then can be displayed at the higher level of the hierarchy. The experiments show that very good hierarchical view of the video sequences can be created for annotation in terms of efficiency.

Spatiotemporal video representation and compression

In this paper is presented a novel approach for video representation and compression based on spectral compression of spatiotemporal cube. This solution is inspired by the spatiotemporal representation of video sequences.

A Framework for Scalable Summarization of Video

Video summaries provide compact representations of video sequences, with the length of the summary playing an important role, trading off the amount of information conveyed and how fast it can be visualized. This letter proposes scalable summarization as a method to easily adapt the summary to a suitable length, according to the requirements in each case, along with a suitable framework. The analysis algorithm uses a novel iterative ranking procedure in which each summary is the result of the extension of the previous one, balancing information coverage and visual pleasantness. The result of the algorithm is a ranked list, a scalable representation of the sequence useful for summarization. The summary is then efficiently generated from the bitstream of the sequence using bitstream extraction.

Trajectory Tree as an Object-Oriented Hierarchical Representation for Video

This paper presents the trajectory tree as a hierarchical region-based representation for video sequences. Motion, as well as spatial features from multiple frames are used to generate a set of temporal regions structured within a hierarchy of scale and motion coherency. The resulting representations offer a global description of the entire video sequence and enhance semantic analysis potential. A multiscale segmentation strategy is proposed whereby region-merging criteria of progressively greater complexity are used to define partition layers of increasing aptitude for object detection. A novel data structure, called the trajectory adjacency graph, is defined for the long-term analysis of partition sequences. Furthermore, mechanisms for assessing connectivity, verifying temporal continuity, and proposing merging operations based on color, affine, and translational motion homogeneity characteristics over the entire sequence are also introduced. Finally, as demonstrated through experimental results, the trajectory tree offers a concise yet versatile support for video object segmentation, description and retrieval tasks.

Orthogonal Polyhedra in <italic>3D Time-Color Space</italic> as a Geometric Model for Representation of Video Sequences with Low or Inexistent Redundancy between Frames

Several video compression methods have in common their conciseness depends of the degree of redundancy between the frames associated to the sequences they describe. Cartoon animations are a good example of animations with an elevated redundancy and from which an elevated compression is expected. However, scientific sequences provide us with a huge set of examples where the degree of redundancy is very low or inexistent. Because of the accuracy required when these sequences are analyzed, any kind of threshold, which could elevate the redundancy degree, is prohibited. In this work, a proposed solution to the addressed problem considers the linearization of each frame. That is, a frame can be considered as a matrix but by stacking its columns on top of one another a vector is obtained. In this way each pixel can be referenced by only one coordinate in the vector. Starting from these considerations a geometric representation is built: A video sequence will be associated to a unique 3D Orthogonal Pseudo-Polyhedron (3D-OPP). Such 3D-OPP is embedded in a 3D Time-Color Space where X1-axis corresponds to the position of pixels in the linearization, the second spatial dimension (X2-axis) is associated to the color value of a pixel, and finally, X3-axis describes the displaying time of frames in the original animation. Such 3D-OPP can be compressed, manipulated, and displayed in screen by expressing it according to the Extreme Vertices Model in the n-Dimensional Space (nD-EVM). The nD-EVM shares the representation of n-Dimensional Orthogonal Pseudo-Polytopes (nD-OPPs) by considering only a subset of their vertices.

Analysis of multimodal sequences using geometric video representations

This paper presents a novel method to correlate audio and visual data generated by the same physical phenomenon, based on sparse geometric representation of video sequences. The video signal is modeled as a sum of geometric primitives evolving through time, that jointly describe the geometric and motion content of the scene. The displacement through time of relevant visual features, like the mouth of a speaker, can thus be compared with the evolution of an audio feature to assess the correspondence between acoustic and visual signals. Experiments show that the proposed approach allows to localize and track the speaker's mouth when several persons are present on the scene, in presence of distracting motion, and without prior face or mouth detection.

Digital Video Revisited

In this paper, we focus on content based access and manipulation techniques for digital video. We review different techniques developed for extraction of contents of video using visual information. We then present description of a system which can extract video object based representation of video sequences and builds up a video object based description of the content of the video collection. We also present a brief review of another technology which provides for interactive manipulation of the video content through interaction with audio-visual objects like digital actors.

Mosaic representations of video sequences based on slice image analysis

There has been a lot of activity recently surrounding the mosaic representation of video sequences. However, existing mosaic methods have in general a strong limitation. They are prone to converge to a local minimum in least-squares estimation, especially when large image motion exists between two consecutive frames or there are too many outliers, for example, when large moving objects appear in the sequence. In this paper, a novel method is proposed to obtain the initial values of the model parameters. The proposed method is based on slice image analysis, and can effectively avoid the local minimum problem and remove the accumulated distortions from the final mosaic which are common in the previous methods. And furthermore, our method is robust even if large moving objects are present in the sequence.

Mastering and Archiving Uncompressed Digital Video Test Materials

This is a report on the status of the SMPTE Ad Hoc Group (AHG) charged with creating a master set of standard-definition video sequences for subjective testing of electronic systems. The images will be available from SMPTE, for use in the evaluation of electronic systems. The first task is to create a master representation of video sequences enabling preservation in a consistent and stable storage environment. Once the sequences are restored and returned to a pristine state (when possible), they will be provided in digital file format to SMPTE. The Society will then offer the video sequences in various formats (digital data media and uncompressed digital videotape) for testing electronic systems. This report also includes representative images from sequences in a set of standard-definition materials, the first production of the AHG. Finally, feedback and suggestions on the quality and interest of this process is requested.

Resequencing of Video using Spatial Indexing

Current work on computer indexing of video is either low level, working with edits and camera motion or high level using a human entered semantic description. In this paper we propose a complementary indexing method, based on the spatial relationships of objects in the video.Spatial reasoning techniques have been applied to pictorial databases, in particular those using two-dimensional (2D) strings as an index representation have been successful. We demonstrate how one 2D string representation may be extended to represent video sequences and how it may be compressed. The presented query scheme makes efficient use of the notation by not requiring the notation to be expanded during matching. Query schemes for both single-frame queries and full subsequence queries are presented. The major improvement is that the query process is not based on the expansion to and evaluation of a sequence of 2D strings, rather the spatial relationship changes may be read directly from the edit sequence. This allows us to easily discard edits which do not affect the objects in the query, greatly reducing the processing required during querying.The use of this retrieval scheme is explored in a simple video interpretation application, that of a guide system. We present a system which sequences video clips from a database to illustrate paths between locations. The retrieval of clips is performed using the techniques for video sequence matching developed in this paper.