Video Summarization Algorithm Research Articles

Video Summarization using Python

Video summarization, the process of condensing lengthy videos into shorter, more concise representations while preserving essential content, has garnered significant attention due to the exponential growth of video data across various domains. Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content.[10] This paper presents a comprehensive review and implementation of video summarization techniques leveraging Python programming language. In addition to the theoretical exploration, this paper provides practical insights into implementing video summarization algorithms using Python libraries such as OpenCV, TensorFlow, and PyTorch. A step-by-step guide to preprocessing video data, extracting features, and generating summaries is presented, accompanied by code snippets for clarity and reproducibility. Through this review and implementation, this paper aims to provide researchers and practitioners with a comprehensive understanding of video summarization techniques and practical guidance for developing efficient and effective summarization systems using Python

A Moving Target Tracking Framework Based on a Set and Its Topological Space

Moving target tracking is a technology that matches frames and images based on target characteristics. This technology is widely utilized in intelligent transportation, logistics transportation, public security, sports event broadcasting, and other fields. Existing research focuses primarily on improving target detection and tracking algorithms to improve target retrieval and tracking efficiency. However, the majority of studies focus on global and full-range retrieval. More importantly, in large video scenes with multiple camera collaborations, these methods rarely consider the efficiency of target retrieval and tracking. Based on relevant theories and methods of video GIS, set theory, and topology, in this paper, a set and its topology space covering road networks, cameras, videos, and key frames were constructed. Additionally, the positioning, tracking, and track representation of a moving target based on the set and its topology space were solved. Compared to the feature vector algorithm, video summarization and Meanshift algorithm, the experimental findings reveal that the target retrieval performance, algorithm stability, and robustness are improved.

Traffic Anomaly Detection and Video Summarization Using Spatio-Temporal Rough Fuzzy Granulation With Z-Numbers

Existing traffic video summarization algorithms are capable of detecting one-class (i.e., collision) anomaly and cannot handle uncertainty issues arising between two-class anomalies, such as collision and near-miss. To address the issues, a new video summarization algorithm, namely Z-number s-based spatio-temporal rough fuzzy granulation (Z-STRFG) is developed. In Z-STRFG, various spatio-temporal features are computed over the video frames and used for obtaining the approximate anomaly-prone regions in terms of granules. In these regions, uncertainty (i.e., fuzziness) may arise among three scenarios, namely collision, near-miss, and normal traffic. Therefore, two types of rough fuzzy granules (RFGs) along with their roughness scores are computed to distinguish the aforesaid three scenarios. For each RFG, Z-number is computed based on the membership value of its roughness score to ensure a higher degree of reliability in the detection of anomaly class. Aforesaid characteristics of Z-STRFG improve its speed and accuracy for traffic anomaly detection. The efficacy of Z-STRFG has been demonstrated over 130 real-time traffic videos containing collisions, near-misses, and normal traffics. The superiority of Z-STRFG over some state-of-the-art is also proved through extensive experiments.

Multi View Video Summarization Using RNN and SURF Based High Level Moving Object Feature Frames

Multi-View Video summarization is a process to ease the storage consumption that facilitates organized storage, and perform other mainline videos analytical task. This in-turn helps quick search or browse and retrieve the video data with minimum time and without losing crucial data. In static video summarization, there is less challenge in time and sequence issues to rearrange the video-synopsis. The low-level features are easy to compute and retrieve. But for high-level features like event detection, emotion detection, object recognition, face detection, gesture detection, and others requires the comprehension of the video content. This research is to propose an approach to over- come the difficulties in handling the high-level features. The distinguishable contents from the videos are identified by object detection and feature-based area strategy. The major aspect of the proposed solution is to retrieve the attributes of a motion source from a video frame. By dividing the details of the object that are available in the video frame wavelet decomposition are achieved. The motion frequency scoring method records the time of motions in the video. The frequency motion feature of video usage is a challenge given the continuous change of objects shape. Therefore, the object position and corner points are spotted using Speeded Up Robust Features (SURF) feature points. Support vector machine clustering extracts keyframes. The memory-based re- current neural network (RNN) recognizes the object in the video frame and remembers a long sequence. RNN is an artificial neural network where nodes form a temporal relationship. The attention layer in the proposed RNN network extracts the details about the objects in motion. The motion objects identified using the three video clippings is finally summarized using video summarization algorithm. To perform the simulation, MATLAB R 2014b software was used.

Robust video summarization algorithm using supervised machine learning

The proposed approach uses ResNet-18 for feature extraction and with the help of temporal interest proposals generated for the video sequences, generates a video summary. The ResNet-18 is a convolutional neural network with eighteen layers. The existing methods don't address the problem of the summary being temporally consistent. The proposed work aims to create a temporally consistent summary. The classification and regression module are implemented to get fixed length inputs of the combined features. After this, the non-maximum suppression algorithm is applied to reduce the redundancy and remove the video segments having poor quality and low confidence-scores. Video summaries are generated using the kernel temporal segmentation (KTS) algorithm which converts a given video segment into video shots. The two standard datasets TVSum and SumMe are used to evaluate the proposed model. It is seen that the F-score obtained on TVSum and SumMe dataset is 56.13 and 45.06 respectively.

Extractive Text and Video Summarization using TF-IDF Algorithm

Abstract: Text summarization is a technique for extracting concise summaries from a large text without sacrificing any important information. It's a good way to extract crucial information from documents. The rapid rise of the internet has resulted in a substantial surge in data all across the world. It has become difficult for humans to manually summarise big documents. Automatic Text Summarization is an NLP technique that lowers the time and efforts required by a human to create a summary. Text summarising techniques are divided into two categories: extractive and abstractive. In the extractive approach, text summarising techniques choose sentences from documents based on a set of criteria. In the abstractive approach, text summarising techniques strive to improve sentence coherence by reducing redundancies and explaining the context of sentences. The extractive summarization approach is the subject of this paper. There are several methods for summarising data, including TF-IDF, Text Rank, PageRank, and Latent Dirichlet Allocation (LDA). This work examines Text Summarization using the TFIDF Algorithm, a numerical measure that ranks the value of a word in a document based on how frequently it appears in that document and a set of documents. The application of the TF-IDF Algorithm for text, document, article, and video summarization is described in this study. There are no repetitions in the results, and for some searches, they are nearly identical to the summary results provided by humans. This algorithm offers a sentence extraction technique that selects the most diverse top-ranked sentences. Keywords: Extractive Summarization, Term Frequency-Inverse Document Frequency (TF-IDF), Natural Language Processing (NLP), Text Summarization, Video Summarization

Video Summarization Through Reinforcement Learning With a 3D Spatio-Temporal U-Net.

Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatio-temporal U-Net is used to efficiently encode spatio-temporal information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatio-temporal latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatio-temporal CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information.

Retraction Note to: Risk assessment of extreme rainfall climate change and sports stadium sports based on video summarization algorithm

Based on the research of existing video summarization algorithms, this paper proposes a video summarization algorithm based on block sparse representation. This algorithm is mainly used to record multi-screen videos with similar content and to create video advertisements for multiple videos. First, the video is divided into multiple blocks according to the similarity, and the size of the block is controlled, and then, multiple video blocks are embedded in the space, and representative video blocks are selected through sparse display. This study is based on the annual heavy rainfall in a specific area of the Loess Plateau in 2020 and takes the factors that affect the risk of college sports as the research object. The goal of this article is to evaluate and determine the degree of sports risk in colleges and universities through the establishment of a fuzzy comprehensive evaluation model, determine the weight of risk indicators, and determine the risk prevention system. On this basis, this article puts forward some suggestions: improve students’ self-awareness, improve safety awareness, strengthen the safety guidance of physical education teachers, increase the importance of school leaders’ safety, improve the school’s risk prevention mechanism, and strengthen the emergency response capabilities of ordinary medical staff, strengthen the management of medical staff, strengthen the management of school sports facilities, build and improve student sports injury insurance, make contributions to the management of sports risk prevention in colleges and universities, and lay the foundation for the construction of a healthy campus. Therefore, this paper uses the existing abstract video technology to describe and study the extreme rainfall climate in a certain place and conduct a narrative research on the sports risk assessment in university stadiums.

Video Summarization Using Deep Neural Networks: A Survey

Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content. Several approaches have been developed over the last couple of decades, and the current state of the art is represented by methods that rely on modern deep neural network architectures. This work focuses on the recent advances in the area and provides a comprehensive survey of the existing deep-learning-based methods for generic video summarization. After presenting the motivation behind the development of technologies for video summarization, we formulate the video summarization task and discuss the main characteristics of a typical deep-learning-based analysis pipeline. Then, we suggest a taxonomy of the existing algorithms and provide a systematic review of the relevant literature that shows the evolution of the deep-learning-based video summarization technologies and leads to suggestions for future developments. We then report on protocols for the objective evaluation of video summarization algorithms, and we compare the performance of several deep-learning-based approaches. Based on the outcomes of these comparisons, as well as some documented considerations about the amount of annotated data and the suitability of evaluation protocols, we indicate potential future research directions.

RETRACTED ARTICLE: Risk assessment of extreme rainfall climate change and sports stadium sports based on video summarization algorithm

RETRACTED ARTICLE: Risk assessment of extreme rainfall climate change and sports stadium sports based on video summarization algorithm

Video Summarization Algorithm Based on Improved Fully Convolutional Network

Video Summarization Algorithm Based on Improved Fully Convolutional Network

Similarity Based Block Sparse Subset Selection for Video Summarization

Video summarization (VS) is generally formulated as a subset selection problem where a set of representative keyframes or key segments is selected from an entire video frame set. Though many sparse subset selection based VS algorithms have been proposed in the past decade, most of them adopt linear sparse formulation in the explicit feature vector space of video frames, and don’t consider the local or global relationships among frames. In this paper, we first extend the conventional sparse subset selection for VS into kernel block sparse subset selection (KBS3) to utilize the advantage of kernel sparse coding and introduce a local inter-frame relationship through packing of frame blocks. Going a step further, we propose a similarity based block sparse subset selection (SB2S3) model by applying a specially designed transformation matrix on the KBS3 model in order to introduce a kind of global inter-frame relationship through the similarity. Finally, a greedy pursuit based algorithm is devised for the proposed NP-hard model optimization. The proposed SB2S3 has the following advantages: 1) through the similarity between each frame and any other frame, the global relationship among all frames can be considered; 2) through block sparse coding, the local relationship of adjacent frames is further considered; and 3) it has a wider application, since features can derive similarity, but not vice versa. It is believed that the effect of modeling such global and local relationships among frames in this paper, is similar to that of modeling the long-range and short-range dependencies among frames in deep learning based methods. Experimental results on three benchmark datasets have demonstrated that the proposed approach is superior to not only other sparse subset selection based VS methods but also most unsupervised deep-learning based VS methods.

Video Summarization Based on Mutual Information and Entropy Sliding Window Method.

This paper proposes a video summarization algorithm called the Mutual Information and Entropy based adaptive Sliding Window (MIESW) method, which is specifically for the static summary of gesture videos. Considering that gesture videos usually have uncertain transition postures and unclear movement boundaries or inexplicable frames, we propose a three-step method where the first step involves browsing a video, the second step applies the MIESW method to select candidate key frames, and the third step removes most redundant key frames. In detail, the first step is to convert the video into a sequence of frames and adjust the size of the frames. In the second step, a key frame extraction algorithm named MIESW is executed. The inter-frame mutual information value is used as a metric to adaptively adjust the size of the sliding window to group similar content of the video. Then, based on the entropy value of the frame and the average mutual information value of the frame group, the threshold method is applied to optimize the grouping, and the key frames are extracted. In the third step, speeded up robust features (SURF) analysis is performed to eliminate redundant frames in these candidate key frames. The calculation of Precision, Recall, and F are optimized from the perspective of practicality and feasibility. Experiments demonstrate that key frames extracted using our method provide high-quality video summaries and basically cover the main content of the gesture video.

Cost-Effective Video Summarization Using Deep CNN With Hierarchical Weighted Fusion for IoT Surveillance Networks

Video summarization (VS) has attracted intense attention recently due to its enormous applications in various computer vision domains, such as video retrieval, indexing, and browsing. Traditional VS researches mostly target at the effectiveness of the VS algorithms by introducing the high quality of features and clusters for selecting representative visual elements. Due to the increased density of vision sensors network, there is a tradeoff between the processing time of the VS methods with reasonable and representative quality of the generated summaries. It is a challenging task to generate a video summary of significant importance while fulfilling the needs of Internet of Things (IoT) surveillance networks with constrained resources. This article addresses this problem by proposing a new computationally effective solution through designing a deep CNN framework with hierarchical weighted fusion for the summarization of surveillance videos captured in IoT settings. The first stage of our framework designs discriminative rich features extracted from deep CNNs for shot segmentation. Then, we employ image memorability predicted from a fine-tuned CNN model in the framework, along with aesthetic and entropy features to maintain the interestingness and diversity of the summary. Third, a hierarchical weighted fusion mechanism is proposed to produce an aggregated score for the effective computation of the extracted features. Finally, an attention curve is constituted using the aggregated score for deciding outstanding keyframes for the final video summary. Experiments are conducted using benchmark data sets for validating the importance and effectiveness of our framework, which outperforms the other state-of-the-art schemes.

Domain independent redundancy elimination based on flow vectors for static video summarization

Video summarization aims to find a compact representation of input videos. The method finds out interesting parts of the video by discarding the remaining parts of the video. The abstracts thus generated enhances browsing and retrieval of video data. The quality of summaries generated by video summarization algorithms can be improved if the redundant frames in the input video are taken care of before summarization. This paper presents a novel domain-independent method for redundancy elimination from input videos before summarization maintaining keyframes in the original video. The frames of input video are first presampled by selecting two frames in one second. The flow vectors between consecutive frames are computed using SIFT Flow algorithm. The magnitude of flow vectors at each pixel position of the frame are summed up to find the displacement magnitude between the consecutive frames. The redundant frames are filtered out based on local averaging of the displacement values. The evaluation of the method is done using two standard datasets namely VSUMM and OVP. The results demonstrate that an average reduction rate of 97.64% is achieved consistently on videos of all categories. The method also gives superior results compared to other state-of-the-art redundancy elimination methods for video summarization

Latency-Aware Adaptive Video Summarization for Mobile Edge Clouds

With the technological advances in wireless multimedia domain, these videos made by mobile edge devices dominate network traffics. The video summarization technology enables users to understand the storyline of a video before a client requests the complete video content. Summarizing a video on edge devices and transmitting the summary between them requires a user-oriented and adaptive solution due to the limited capability and the dynamic wireless links of edge devices. Therefore, it is beneficial to improve the user's viewing experience and the bandwidth utilization ratio if we generate and transmit a video summary based on network connections and the user's tolerant latency. Unfortunately, previous summarization approaches are incapable of adjusting the summary size adapted to the varying network bandwidth and the user's attitude towards latency. To timely and flexibly deal with mobile videos, we first formulate the video summarization optimization problem with the elastic number of selected representative segments and the outlier detection within a bounded time budget. Furthermore, we develop an online greedy algorithm called the Elastic Video Summarization Algorithm (EVS) to solve the NP hard problem. We analyze the properties associated with EVS and further design an improved EVS-II to reduce computation complexity. Finally, the experimental results demonstrate that our proposed algorithms outperform other existing researches in fitting network bandwidth and detecting outliers.

Video Summarization via Nonlinear Sparse Dictionary Selection

Video summarization (VS) is to identify important content from a given video, which can help users quickly comprehend video content. Recently, sparse dictionary selection (SDS) has demonstrated to be an effective solution for VS problems, which generally assumes a linear relationship between keyframes and non-keyframes. However, this assumption is not always true for video frames which possess intrinsic nonlinear structures and properties. In this paper, by exploiting the nonlinearity between video frames, a nonlinear SDS model is formulated for VS, in which the nonlinearity is transformed to linearity by projecting a video to a high-dimensional feature space induced by a kernel function. We also propose two greedy optimization algorithms to solve the resulting model, namely the standard kernel SDS (KSDS) greedy algorithm and the robust KSDS greedy algorithm with a backtracking strategy. In order to achieve an intuitive and flexible configuration of the VS process, an adaptive criterion, namely energy ratio, is devised to produce video summaries with different lengths for different video contents. Experimental results on two different benchmark video datasets demonstrate that the proposed algorithm outperforms several state-of-the-art VS algorithms.

Automatic 3D Reconstruction From Unstructured Videos Combining Video Summarization and Structure From Motion

Social media and collection of large volumes of multimedia data such as images, videos and the accompanying text is of prime importance in today’s society. This is stimulated by the power of the humans to communicate one with the others. A useful paradigm of exploitation of such a huge amount of multimedia volumes is the 3D reconstruction and modelling of sites, historical cultural cities/regions or objects of interest from the short videos captured by simple users mainly for personal or touristic purposes. The main challenge in this research is the unstructured nature of the videos and the fact that they contain many information which is not related with the object the 3D model we ask for but for personal usage such as humans in front of the objects, weather conditions, etc. In this article, we propose an automatic scheme for 3D modelling/reconstruction of objects of interest by collecting pools of short duration videos that have been captured mainly for touristic purposes. Initially a video summarization algorithm is introduced using a discriminant Principal Component Analysis (d-PCA). The goal of this innovative scheme is to extract the frames so that bunches within each video cluster that contains videos of content referring to the same object present the maximum coherency of image data while content across bunches the minimum one. Experimental results on cultural objects indicate the efficiency pf the proposed method to 3D reconstruct assets of interest using an unstructured image content information. □

Spatio-temporal summarization of dance choreographies

An important issue in performing dance analysis is the automatic extraction of its choreographic patterns, since these elements provide an abstract representation of the semantics of the dance and encode the overall dance storytelling. However, application of conventional video summarization algorithms on dance sequences cannot appropriately retrieve their choreographic patterns, since a dance is composed of an ordered set of sequential elements which are often repeated in time. Additionally, 3D geometry is lost using color information. For this reason, in this paper we propose a new dance summarization scheme of 3D motion captured data (in the form of skeleton joints coordinates) recorded using the Vicon motion capture system. The proposed key frame extraction method implements a hierarchical scheme that exploits spatio-temporal variations of dance features. Initially, global holistic descriptors are extracted to localize the key choreographic steps of a dance (coarse representation). Then, each segment is further decomposed into finer sub-segments to improve dance representativity (fine representation). The abstraction scheme exploits the concepts of a Sparse Modeling Representative Selection (SMRS) appropriately modified to enable spatio-temporal modelling of the dance sequences through a hierarchical decomposition algorithm. Our approach is evaluated on thirty folkloric dance sequences recorded at the Aristotle University of Thessaloniki under the framework of Terpsichore project representing five different choreographies and on publicly available datasets from Carnegie–Mellon University, which depict performances on theatrical kinesiology. Comparisons with other traditional video summarization methods indicate a clear superiority of the proposed hierarchical spatio-temporal decomposition scheme.

Design and Evaluation of Video Summarization Algorithm based on EEG Information

Design and Evaluation of Video Summarization Algorithm based on EEG Information