Video Summarization Method Research Articles

Saliency-based video summarization for face anti-spoofing

With the growing availability of databases for face presentation attack detection, researchers are increasingly focusing on video-based face anti-spoofing methods that involve hundreds to thousands of images for training the models. However, there is currently no clear consensus on the optimal number of frames in a video to improve face spoofing detection. Inspired by the visual saliency theory, we present a video summarization method for face anti-spoofing detection that aims to enhance the performance and efficiency of deep learning models by leveraging visual saliency. In particular, saliency information is extracted from the differences between the Laplacian and Wiener filter outputs of the source images, enabling the identification of the most visually salient regions within each frame. Subsequently, the source images are decomposed into base and detail images, enhancing the representation of the most important information. Weighting maps are then computed based on the saliency information, indicating the importance of each pixel in the image. By linearly combining the base and detail images using the weighting maps, the method fuses the source images to create a single representative image that summarizes the entire video. The key contribution of the proposed method lies in demonstrating how visual saliency can be used as a data-centric approach to improve the performance and efficiency for face presentation attack detection. By focusing on the most salient images or regions within the images, a more representative and diverse training set can be created, potentially leading to more effective models. To validate the method’s effectiveness, a simple CNN–RNN deep learning architecture was used, and the experimental results showcased state-of-the-art performance on four challenging face anti-spoofing datasets.

Global–local spatio-temporal graph convolutional networks for video summarization

Video summarization aims to create concise and accurate summary to enable users to quickly grasp the key content of the original video for facilitating efficient video browsing. Most existing video summarization methods mainly employ recurrent neural networks to capture long-term dependencies in videos, yielding remarkable results. Nevertheless, these methods overlook the potential spatial features inside the video when modeling the video. To tackle this issue, we introduce a global–local spatio-temporal graph convolutional networks for video summarization (GL-STGCN). Initially inspired by the concept of 3-D convolution, we segment the video into non-overlapping segments to capture localized spatial features of sequential frames. Subsequently, a spatial graph is constructed for each segment, with a fixed time interval between neighboring spatial graphs. Then we use the pooling method to randomly delete the redundant nodes in the graph. We then leverage a temporal gating convolutional network to extract the global temporal relationships within the video. Employing the spatial features, a spatial graph convolutional network is utilized to capture the spatial connections among frames. As the graph node information evolves, the node features furnish a more precise depiction of the video content. Consequently, we employ the temporal gating convolutional network once more to refine the global temporal relations within the video. Extensive experiments on two public datasets are conducted in this paper, showing that our proposed method outperforms most state-of-the-art video summarization methods in terms of performance. Experimental results demonstrate the effectiveness of integrating global temporal and local spatial relationships.

Video Summarization Generation Network Based on Dynamic Graph Contrastive Learning and Feature Fusion

Video summarization aims to analyze the structure and content of videos and extract key segments to construct summarization that can accurately summarize the main content, allowing users to quickly access the core information without browsing the full video. However, existing methods have difficulties in capturing long-term dependencies when dealing with long videos. On the other hand, there is a large amount of noise in graph structures, which may lead to the influence of redundant information and is not conducive to the effective learning of video features. To solve the above problems, we propose a video summarization generation network based on dynamic graph contrastive learning and feature fusion, which mainly consists of three modules: feature extraction, video encoder, and feature fusion. Firstly, we compute the shot features and construct a dynamic graph by using the shot features as nodes of the graph and the similarity between the shot features as the weights of the edges. In the video encoder, we extract the temporal and structural features in the video using stacked L-G Blocks, where the L-G Block consists of a bidirectional long short-term memory network and a graph convolutional network. Then, the shallow-level features are obtained after processing by L-G Blocks. In order to remove the redundant information in the graph, graph contrastive learning is used to obtain the optimized deep-level features. Finally, to fully exploit the feature information of the video, a feature fusion gate using the gating mechanism is designed to fully fuse the shallow-level features with the deep-level features. Extensive experiments are conducted on two benchmark datasets, TVSum and SumMe, and the experimental results show that our proposed method outperforms most of the current state-of-the-art video summarization methods.

Fuzzy C-mean clustering technique based visual features fusion for automatic video summarization method

Fuzzy C-mean clustering technique based visual features fusion for automatic video summarization method

Adam-Dingo optimized deep maxout network-based video surveillance system for stealing crime detection

An enormous amount of video data is generated every day as a result of the widespread deployment of security cameras due to recent technological advancements. Analyzing such videos for identifying anomalies manually is an extremely challenging issue. Stealing is an issue occurring worldwide resulting in casualties and huge financial damages yearly. Efficient techniques to identify this anomaly haven’t been much researched, and hence here a novel technique is proposed. A hybrid optimized deep learning technique is proposed for detecting stealing crime from videos obtained by surveillance cameras. Firstly, the videos are summarized using Video summarization (VSUMM) method and then video is given input to the Deep Maxout Network. A deep maxout network whose weights are updated with the help of the introduced Adam-Dingo Optimizer for identifying the stealing crime event or normal event. The devised Adam-Dingo optimized Deep Maxout network is tested for its performance with metrics, such as True Positive Rate (TPR), True Negative Rate (TNR), and accuracy and is found to have attained values of 0.940, 0.936 and 0.945, respectively. Identifying stealing crime at different weather condition can be the future work. Identifying different crime events and reporting to control room can be the future work of proposed system.

Multi-scale deep feature fusion based sparse dictionary selection for video summarization

The explosive growth of video data constitutes a series of new challenges in computer vision, and the function of video summarization (VS) is becoming more and more prominent. Recent works have shown the effectiveness of sparse dictionary selection (SDS) based VS, which selects a representative frame set to sufficiently reconstruct a given video. Existing SDS based VS methods use conventional handcrafted features or single-scale deep features, which could diminish their summarization performance due to the underutilization of frame feature representation. Deep learning techniques based on convolutional neural networks (CNNs) exhibit powerful capabilities among various vision tasks, as the CNN provides excellent feature representation. Therefore, in this paper, a multi-scale deep feature fusion based sparse dictionary selection (MSDFF-SDS) is proposed for VS. Specifically, multi-scale features include the directly extracted features from the last fully connected layer and the global average pooling (GAP) processed features from intermediate layers, then VS is formulated as a problem of minimizing the reconstruction error using the multi-scale deep feature fusion. In our formulation, the contribution of each scale of features can be adjusted by a balance parameter, and the row-sparsity consistency of the simultaneous reconstruction coefficient is used to select as few keyframes as possible. The resulting MSDFF-SDS model is solved by using an efficient greedy pursuit algorithm. Experimental results on two benchmark datasets demonstrate that the proposed MSDFF-SDS improves the F-score of keyframe based summarization more than 3% compared with the existing SDS methods, and performs better than most deep-learning methods for skimming based summarization.

Recurrent unit augmented memory network for video summarisation

AbstractVideo summarisation can relieve the pressure on video storage, transmission, archiving, and retrieval caused by the explosive growth of online videos in recent years. Most existing supervised video summarisation methods use convolutional neural network (CNN) or recurrent neural network (RNN) to model the temporal dependencies between video frames or video shots. CNN mainly focuses on local information, and RNN loses long‐term information when the input sequence is long, both of which have limited ability to obtain long‐range memory in the video. Therefore, a recurrent unit augmented memory network (RUAMN) for video summarisation is proposed, which effectively utilises the long‐term memory extraction ability of the end‐to‐end memory network (MemN2N) and solves the problem that MemN2N is insensitive to temporal sequence information. At the same time, the proposed RUAMN enhances the process of memory update between multiple computational steps (hops), and finally generates a meaningful video summarisation result. Specifically, the proposed RUAMN is mainly composed of the input module, the global‐and‐local sampling, the memory module and the output module. The input module uses a bidirectional GRU to obtain the forward and backward information of each video frame. Then the global‐and‐local sampling performs global sampling and local sampling on the output sequence of the input module respectively to obtain several shorter sequences, so that the memory modules can capture the fine‐grained relationship features between video frames more effectively. The memory module extracts the long‐term memory information in the feature sequence, and finally the frame‐level importance scores are predicted by the output module. Extensive experiments on benchmark datasets, that is, TVSum and SumMe, demonstrate the superiority of our method over several state‐of‐the‐art supervised video summarisation methods.

Generative Adversarial Networks for Video Summarization Based on Key-frame Selection

Video summarization based on generative adversarial networks (GANs) has been shown to easily produce more realistic results. However, most summary videos are composed of multiple key components. If the selection of some video frames changes during the training process, the information carried by these frames may not be reasonably reflected in the identification results. In this paper, we propose a video summarization method based on selecting keyframes over GANs. The novelty of the proposed method is the discriminator not only identifies the completeness of the video, but also takes into account the value judgment of the candidate keyframes, thus enabling the influence of keyframes on the result value. Given GANs are mainly designed to generate continuous real values, it is generally challenging to generate discrete symbol sequences during the summarization process directly. However, if the generated sample is based on discrete symbols, the slight guidance change of the discrimination network may be meaningless. To better use the advantages of GANs, the study also adopts the video summarization optimization method of GANs under a collaborative reinforcement learning strategy. Experimental results show the proposed method gets a significant summarization effect and character compared with the existing cutting-edge methods.

Unsupervised Video Summarization Based on Deep Reinforcement Learning with Interpolation.

Individuals spend time on online video-sharing platforms searching for videos. Video summarization helps search through many videos efficiently and quickly. In this paper, we propose an unsupervised video summarization method based on deep reinforcement learning with an interpolation method. To train the video summarization network efficiently, we used the graph-level features and designed a reinforcement learning-based video summarization framework with a temporal consistency reward function and other reward functions. Our temporal consistency reward function helped to select keyframes uniformly. We present a lightweight video summarization network with transformer and CNN networks to capture the global and local contexts to efficiently predict the keyframe-level importance score of the video in a short length. The output importance score of the network was interpolated to fit the video length. Using the predicted importance score, we calculated the reward based on the reward functions, which helped select interesting keyframes efficiently and uniformly. We evaluated the proposed method on two datasets, SumMe and TVSum. The experimental results illustrate that the proposed method showed a state-of-the-art performance compared to the latest unsupervised video summarization methods, which we demonstrate and analyze experimentally.

A Review on Key Features and Novel Methods for Video Summarization

In this paper, we discuss techniques, algorithms, evaluation methods used in online, offline, supervised, unsupervised, multi-video and clustering methods used for Video Summarization/Multi-view Video Summarization from various references. We have studied different techniques in the literature and described the features used for generating video summaries with evaluation methods, supervised, unsupervised, algorithms and the datasets used. We have covered the survey towards the new frontier of research in computational intelligence technique like ANN (Artificial Neural Network) and other evolutionary algorithms for VS using both supervised and unsupervised methods. We highlight on single, multi-video summarization with features like video, audio, and semantic embeddings considered for VS in the literature. A careful presentation is attempted to bring the performance comparison with Precision, Recall, F-Score, and manual methods to evaluate the VS.

An Optimized Deep Learning Method for Video Summarization Based on the User Object of Interest

An Optimized Deep Learning Method for Video Summarization Based on the User Object of Interest

Multimodal-Based and Aesthetic-Guided Narrative Video Summarization

Narrative videos usually illustrate the main content through multiple narrative information such as audios, video frames and subtitles. Existing video summarization approaches rarely consider the multiple dimensional narrative inputs, or ignore the impact of shots artistic assembly when directly applied to narrative videos. This paper introduces a multimodal-based and aesthetic-guided narrative video summarization method. Our method leverages multimodal information including visual content, subtitles and audio information through our specified key shots selection, subtitle summarization, and highlight extraction components. Furthermore, under the guidance of cinematographic aesthetic, we design a novel shots assembly module to ensure the shot content completeness and then assemble the selected shots into a desired summary. Besides, our method also provides the flexible specification for shots selection, to achieve which it automatically selects semantically related shots according to the user-designed text. By conducting a large number of quantitative experimental evaluations and user studies, we demonstrate that our method effectively preserves important narrative information of the original video, and it is capable of rapidly producing high-quality and aesthetic-guided narrative video summaries.

Video Summarization With Spatiotemporal Vision Transformer.

Video summarization aims to generate a compact summary of the original video for efficient video browsing. To provide video summaries which are consistent with the human perception and contain important content, supervised learning-based video summarization methods are proposed. These methods aim to learn important content based on continuous frame information of human-created summaries. However, simultaneously considering both of inter-frame correlations among non-adjacent frames and intra-frame attention which attracts the humans for frame importance representations are rarely discussed in recent methods. To address these issues, we propose a novel transformer-based method named spatiotemporal vision transformer (STVT) for video summarization. The STVT is composed of three dominant components including the embedded sequence module, temporal inter-frame attention (TIA) encoder, and spatial intra-frame attention (SIA) encoder. The embedded sequence module generates the embedded sequence by fusing the frame embedding, index embedding and segment class embedding to represent the frames. The temporal inter-frame correlations among non-adjacent frames are learned by the TIA encoder with the multi-head self-attention scheme. Then, the spatial intra-frame attention of each frame is learned by the SIA encoder. Finally, a multi-frame loss is computed to drive the learning of the network in an end-to-end trainable manner. By simultaneously using both inter-frame and intra-frame information, our method outperforms state-of-the-art methods in both of the SumMe and TVSum datasets. The source code of the spatiotemporal vision transformer will be available at https://github.com/nchucvml/STVT.

Adaptive Multiview Graph Difference Analysis for Video Summarization

Adapting detection to different shot types is a significant challenge for video summarization methods based on shot boundary detection. In our recent work, a new graph model was introduced in the feature modelling of frames and analysed for changes in graph structure to improve the detection of shot boundaries. In this paper, we further explore the potential of graph models and propose a more general framework for online, real-time automatic video summarization. The framework develops a novel adaptive multiview graph difference analysis method to improve the algorithm’s robustness in detecting different shot transitions. Previous fusion methods typically used a priori knowledge to assign weights to the various feature differences from videos. In contrast, our framework can weigh and fuse the resulting differences by learning the importance of various video features from the structural changes of the corresponding multiview graphs. Additionally, we propose a new threshold-based adaptive decision method which can dynamically select the most accurate shot boundary decision threshold by analysing a small number of historical frames and learning the tolerance factor in the current shot. The experimental results show that the proposed method outperforms state-of-the-art methods in terms of precision and F-score on the VSUMM and YouTube datasets.

Unsupervised video summarization using deep Non-Local video summarization networks

Video summarization is to extract effective information from videos to quickly obtain the most informative summary. Most of the existing video summarization methods use recurrent neural networks and their variants such as long and short-term memory (LSTM), to simulates the variable range time dependence between video frames. However, those methods can only process serial inputs of the video frames along with the hidden layer information from the previous time step, which affects the performance and the quality of video summarization. To tackle this issue, we present a deep non-local video summarization network (DN-VSN) for original video abstracts in this paper. Our unsupervised model treats video summarization as a sequence of decision problems. Given an input video, the probability that a video frame is selected as a part of the summary is obtained through a non-local convolutional network, and a strategy gradient algorithm of reinforcement learning is adopted for optimization in the training phase. The proposed method has been tested on four widely used datasets. The experimental results show the superiority of the proposed unsupervised model over the state-of-the-art methods.

A Video Summarization Model Based on Deep Reinforcement Learning with Long-Term Dependency.

Deep summarization models have succeeded in the video summarization field based on the development of gated recursive unit (GRU) and long and short-term memory (LSTM) technology. However, for some long videos, GRU and LSTM cannot effectively capture long-term dependencies. This paper proposes a deep summarization network with auxiliary summarization losses to address this problem. We introduce an unsupervised auxiliary summarization loss module with LSTM and a swish activation function to capture the long-term dependencies for video summarization, which can be easily integrated with various networks. The proposed model is an unsupervised framework for deep reinforcement learning that does not depend on any labels or user interactions. Additionally, we implement a reward function () that jointly considers the consistency, diversity, and representativeness of generated summaries. Furthermore, the proposed model is lightweight and can be successfully deployed on mobile devices and enhance the experience of mobile users and reduce pressure on server operations. We conducted experiments on two benchmark datasets and the results demonstrate that our proposed unsupervised approach can obtain better summaries than existing video summarization methods. Furthermore, the proposed algorithm can generate higher F scores with a nearly 6.3% increase on the SumMe dataset and a 2.2% increase on the TVSum dataset compared to the DR-DSN model.

Cricket Video Highlight Generation Methods: A Review

The key events extraction from a video for the bestrepresentation of its contents is known as video summarization.In this study, the game of cricket is specifically consideredfor extracting important events such as boundaries, sixes andwickets. The cricket video highlight generation frameworksrequire extensive key event identification. These key events canbe identified by extracting the audio, visual and textual featuresfrom any cricket video.The prediction accuracy of the cricketvideo summarization mainly depends on the game rules, player’sform, their skill, and different natural conditions. This paperprovides a complete survey of latest research in cricket videosummarization methods. It includes the quantitative evaluationof the outcomes of the existing frameworks. This extensive reviewhighly recommended developing deep learning-assisted videosummarization approaches for cricket video due to their morerepresentative feature extraction and classification capabilitythan the conventional edge, texture features, and classifiers. Thescope of this analysis also includes future visions and researchopportunities in cricket highlight generation.

Lecture Video Automatic Summarization System Based on DBNet and Kalman Filtering

Video summarization for educational scenarios aims to extract and locate the most meaningful frames from the original video based on the main contents of the lecture video. Aiming at the defect of existing computer vision-based lecture video summarization methods that tend to target specific scenes, a summarization method based on content detection and tracking is proposed. Firstly, DBNet is introduced to detect the contents such as text and mathematical formulas in the static frames of these videos, which is combined with the convolutional block attention module (CBAM) to improve the detection precision. Then, frame-by-frame data association of content instances is performed using Kalman filtering, the Hungarian algorithm, and appearance feature vectors to build a tracker. Finally, video segmentation and key frame location extraction are performed according to the content instance lifelines and content deletion events constructed by the tracker, and the extracted key frame groups are used as the final video summary result. Experimenting on a variety of scenarios of lecture video, the average precision of content detection is 89.1%; the average recall of summary results is 92.1%.

Video Summarization using Deep Convolutional Neural Networks and Mutual Probability-based K-Nearest Neighbour

ABSTRACT The video summarisation is an advanced mechanism for enabling users to handle and browse large videos in an effective manner. Various video summarisation methods are developed in recent days, in which handling of synchronisation and timing issues remain as the important challenge. The proposed video summarisation technique produces a short summary from the huge video stream. Initially, from an input database, the cricket videos containing number of frames are fed to keyframe extraction unit. Here, the keyframe extraction is done by the Euclidean distance and discrete cosine transform, and the best keyframes are selected based on the Euclidean distance. The residual frame is obtained by passing the input frames through deep convolutional neural network. Then, the similarity is calculated by Bhattacharyya distance. For video summarisation process, the optimal frameset is evaluated by matching residual keyframe with obtained keyframes. Here, input queries consisting of face object are subjected to object matching process, which is performed using the proposed mutual probability-based k-nearest neighbour (MP-KNN) to obtain relevant frames based on texture features. The performance of the proposed MP-KNN is superior based on precision, recall, and F-measure with values 0.963, 0.960, and 0.909, respectively.

Video Summarization Using Deep Learning for Cricket Highlights Generation

Recently, video surveillance technology has grown pervasive in many aspects of our lives. Automatic video monitoring produces massive amounts of data that need human examination at some point. The primary emphasis is on reducing storage usage by compressing or eliminating superfluous frames without sacrificing real information. The current effort seeks to close the growing gap between the amounts of real data and the volume. Searching through key events in large video collections is time-consuming and tedious. In this paper, smart surveillance for various applications by using video summarization has been presented. A method for generating highlights has presented which pre-processes extracted Video Frames. Convolutional Neural Networks are then used to evaluate these highlighted frames. The proposed technique extracts and calculates characteristics utilized to generate summary movies. For training deep neural networks, cricket datasets have been used. Experimental results show that the proposed solution attains improved results than other advanced summarization methodologies. Experimental results show that the proposed video summarization method consistently generates high-quality reviews for all types of videos. The proposed video summarization method is easy to use, and it can also help extract highlights of cricket games with high accuracy.