Video Summarization Research Articles

ViSTORY: Effective Video Storyboard Generation with Visual Keyframes using Discrete Cosine Transform

Nowadays, multimedia content utility is increasing rapidly. Multimedia search engines like Google, Yahoo, Bing, etc., are available just a click away to all users. There are around 500-600 hours of video uploads per unit of time to the Internet. So, among other types of multimedia content, such as text and images, video is the most complicated content for indexing, browsing, and retrieval. Videos give more scope for implementation because of their complex and unstructured nature. This paper proposes a new method of video storyboard generation with keyframe extraction in spatial and frequency domains using Discrete Cosine Transform (DCT) for video summarization. It discusses the empirical appraisal of video visual keyframes with t-test analysis in comparison with spatial and frequency domains, resulting in a quick response to customer demands by providing static storyboards. This study proposes a new performance measure as matching frames by analyzing input videos and the standard benchmarks video dataset, i.e., Open Video Project (OVP) and SumMe. Among all the keyframe extraction techniques, DCT gives higher accuracy and a better matching rate.

NLP based Video Summarisation using Machine Learning

More people are capturing their daily lives with video data, because to the wide availability of recording equipment. However, the sheer volume of video content makes it more challenging to manage, especially for lengthy movies like security or CCTV footage. For a richer and more succinct condensing of the film will result from automatically detecting the key sections and frames of larger videos and providing them with captions. Users still need to spend time searching or navigating through a summarized video. To extract a shortened version of the footage's information into text form, automatic video summarizing has been proposed. With simply a text summary, the suggested system provides a quick semantic understanding of a lengthy film using LSTM model and the summary can be taken in 3 major different languages (English, Hindi, & Marathi).

Topic-aware video summarization using multimodal transformer

Video summarization aims to generate a short and compact summary to represent the original video. Existing methods mainly focus on how to extract a general objective synopsis that precisely summaries the video content. However, in real scenarios, a video usually contains rich content with multiple topics and people may cast diverse interests on the visual contents even for the same video. In this paper, we propose a novel topic-aware video summarization task that generates multiple video summaries with different topics. To support the study of this new task, we first build a video benchmark dataset by collecting videos from various types of movies and annotate them with topic labels and frame-level importance scores. Then we propose a multimodal Transformer model for the topic-aware video summarization, which simultaneously predicts topic labels and generates topic-related summaries by adaptively fusing multimodal features extracted from the video. Experimental results show the effectiveness of our method.

Generative Adversarial Networks for Video Summarization Based on Key-frame Selection

Video summarization based on generative adversarial networks (GANs) has been shown to easily produce more realistic results. However, most summary videos are composed of multiple key components. If the selection of some video frames changes during the training process, the information carried by these frames may not be reasonably reflected in the identification results. In this paper, we propose a video summarization method based on selecting keyframes over GANs. The novelty of the proposed method is the discriminator not only identifies the completeness of the video, but also takes into account the value judgment of the candidate keyframes, thus enabling the influence of keyframes on the result value. Given GANs are mainly designed to generate continuous real values, it is generally challenging to generate discrete symbol sequences during the summarization process directly. However, if the generated sample is based on discrete symbols, the slight guidance change of the discrimination network may be meaningless. To better use the advantages of GANs, the study also adopts the video summarization optimization method of GANs under a collaborative reinforcement learning strategy. Experimental results show the proposed method gets a significant summarization effect and character compared with the existing cutting-edge methods.

Unsupervised Video Summarization Based on Deep Reinforcement Learning with Interpolation.

Individuals spend time on online video-sharing platforms searching for videos. Video summarization helps search through many videos efficiently and quickly. In this paper, we propose an unsupervised video summarization method based on deep reinforcement learning with an interpolation method. To train the video summarization network efficiently, we used the graph-level features and designed a reinforcement learning-based video summarization framework with a temporal consistency reward function and other reward functions. Our temporal consistency reward function helped to select keyframes uniformly. We present a lightweight video summarization network with transformer and CNN networks to capture the global and local contexts to efficiently predict the keyframe-level importance score of the video in a short length. The output importance score of the network was interpolated to fit the video length. Using the predicted importance score, we calculated the reward based on the reward functions, which helped select interesting keyframes efficiently and uniformly. We evaluated the proposed method on two datasets, SumMe and TVSum. The experimental results illustrate that the proposed method showed a state-of-the-art performance compared to the latest unsupervised video summarization methods, which we demonstrate and analyze experimentally.

Query-based video summarization with multi-label classification network

Query-based video summarization with multi-label classification network

Two stream multi-layer convolutional network for keyframe-based video summarization

Two stream multi-layer convolutional network for keyframe-based video summarization

Video summarization using deep learning techniques: a detailed analysis and investigation.

One of the critical multimedia analysis problems in today's digital world is video summarization (VS). Many VS methods have been suggested based on deep learning methods. Nevertheless, These are inefficient in processing, extracting, and deriving information in the minimum amount of time from long-duration videos. Detailed analysis and investigation of numerous deep learning approach accomplished to determine root of problems connected with different deep learning methods in identifying and summarizing the essential activities in such videos. Various deep learning techniques have been investigated and examined to detect the event and summarization capability for detecting and summarizing multiple activities. Keyframe selection Event detection, categorization, and the activity feature summarization correspond to each activity. The limitations related to each category are also discussed in depth. Concerns about detecting low activity using the deep network on various types of public datasets are also discussed. Viable strategies are suggested to evaluate and improve the generated video summaries on such datasets. Moreover, Potential recommended applications based on literature are listed out. Various deep learning tools for experimental analysis have also been discussed in the paper. Future directions are presented for further exploration of research in VS using deep learning strategies.

Biosignal based emotion-oriented video summarization

Biosignal based emotion-oriented video summarization

A static video summarization approach via block-based self-motivated visual attention scoring mechanism

A static video summarization approach via block-based self-motivated visual attention scoring mechanism

A comprehensive study of automatic video summarization techniques

A comprehensive study of automatic video summarization techniques

Behavior-Based Video Summarization System for Dog Health and Welfare Monitoring

The popularity of dogs has been increasing owing to factors such as the physical and mental health benefits associated with raising them. While owners care about their dogs' health and welfare, it is difficult for them to assess these, and frequent veterinary checkups represent a growing financial burden. In this study, we propose a behavior-based video summarization and visualization system for monitoring a dog's behavioral patterns to help assess its health and welfare. The system proceeds in four modules: (1) a video data collection and preprocessing module; (2) an object detection-based module for retrieving image sequences where the dog is alone and cropping them to reduce background noise; (3) a dog behavior recognition module using two-stream EfficientNetV2 to extract appearance and motion features from the cropped images and their respective optical flow, followed by a long short-term memory (LSTM) model to recognize the dog's behaviors; and (4) a summarization and visualization module to provide effective visual summaries of the dog's location and behavior information to help assess and understand its health and welfare. The experimental results show that the system achieved an average F1 score of 0.955 for behavior recognition, with an execution time allowing real-time processing, while the summarization and visualization results demonstrate how the system can help owners assess and understand their dog's health and welfare.

SOR-TC: Self-attentive octave ResNet with temporal consistency for compressed video action recognition

Modeling and recognizing video activities from videos are key parts of many promising applications such as visual surveillance, human–computer interaction, and video summarization. However, current approaches mainly suffer from two issues: (a) Short-term local and global spatial features are not well represented. The spatial redundancy and dependency have not been well considered on CNN-based action recognition, which may result in a further increase in both memory and computation cost. (b) Long-term temporal consistency is not well captured. The action consistency across multiple clips has been ignored in video-level action recognition approaches. To address these two issues, we propose a Self-Attentive Octave ResNet with Temporal Consistency (SOR-TC) for compressed video action recognition to better capture the short-term and long-term features in video and improve the efficiency and effectiveness of action recognition. In addition, this paper introduces a consistency hypothesis that adjacent clips should predict similar actions. So the consistency loss function is designed to learn the correlation of clips. Finally, extensive experimental results on two benchmark datasets HMDB-51 and UCF-101 verify the effectiveness of our proposed method.

Data-driven enabled approaches for criteria-based video summarization: a comprehensive survey, taxonomy, and future directions

Data-driven enabled approaches for criteria-based video summarization: a comprehensive survey, taxonomy, and future directions

A Review on Key Features and Novel Methods for Video Summarization

In this paper, we discuss techniques, algorithms, evaluation methods used in online, offline, supervised, unsupervised, multi-video and clustering methods used for Video Summarization/Multi-view Video Summarization from various references. We have studied different techniques in the literature and described the features used for generating video summaries with evaluation methods, supervised, unsupervised, algorithms and the datasets used. We have covered the survey towards the new frontier of research in computational intelligence technique like ANN (Artificial Neural Network) and other evolutionary algorithms for VS using both supervised and unsupervised methods. We highlight on single, multi-video summarization with features like video, audio, and semantic embeddings considered for VS in the literature. A careful presentation is attempted to bring the performance comparison with Precision, Recall, F-Score, and manual methods to evaluate the VS.

Perceptive SARS-CoV-2 End-To-End Ultrasound Video Classification through X3D and Key-Frames Selection

The SARS-CoV-2 pandemic challenged health systems worldwide, thus advocating for practical, quick and highly trustworthy diagnostic instruments to help medical personnel. It features a long incubation period and a high contagion rate, causing bilateral multi-focal interstitial pneumonia, generally growing into acute respiratory distress syndrome (ARDS), causing hundreds of thousands of casualties worldwide. Guidelines for first-line diagnosis of pneumonia suggest Chest X-rays (CXR) for patients exhibiting symptoms. Potential alternatives include Computed Tomography (CT) scans and Lung UltraSound (LUS). Deep learning (DL) has been helpful in diagnosis using CT scans, LUS, and CXR, whereby the former commonly yields more precise results. CXR and CT scans present several drawbacks, including high costs. Radiation-free LUS imaging requires high expertise, and physicians thus underutilise it. LUS demonstrated a strong correlation with CT scans and reliability in pneumonia detection, even in the early stages. Here, we present an LUS video-classification approach based on contemporary DL strategies in close collaboration with Fondazione IRCCS Policlinico San Matteo's Emergency Department (ED) of Pavia. This research addressed SARS-CoV-2 patterns detection, ranked according to three severity scales by operating a trustworthy dataset comprising ultrasounds from linear and convex probes in 5400 clips from 450 hospitalised subjects. The main contributions of this study are related to the adoption of a standardised severity ranking scale to evaluate pneumonia. This evaluation relies on video summarisation through key-frame selection algorithms. Then, we designed and developed a video-classification architecture which emerged as the most promising. In contrast, the literature primarily concentrates on frame-pattern recognition. By using advanced techniques such as transfer learning and data augmentation, we were able to achieve an F1-Score of over 89% across all classes.

Privacy Preserved Video Summarization of Road Traffic Events for IoT Smart Cities

The purpose of smart surveillance systems for automatic detection of road traffic accidents is to quickly respond to minimize human and financial losses in smart cities. However, along with the self-evident benefits of surveillance applications, privacy protection remains crucial under any circumstances. Hence, to ensure the privacy of sensitive data, European General Data Protection Regulation (EU-GDPR) has come into force. EU-GDPR suggests data minimisation and data protection by design for data collection and storage. Therefore, for a privacy-aware surveillance system, this paper targets the identification of two areas of concern: (1) detection of road traffic events (accidents), and (2) privacy preserved video summarization for the detected events in the surveillance videos. The focus of this research is to categorise the traffic events for summarization of the video content, therefore, a state-of-the-art object detection algorithm, i.e., You Only Look Once (YOLOv5), has been employed. YOLOv5 is trained using a customised synthetic dataset of 600 annotated accident and non-accident video frames. Privacy preservation is achieved in two steps, firstly, a synthetic dataset is used for training and validation purposes, while, testing is performed on real-time data with an accuracy from 55% to 85%. Secondly, the real-time summarized videos (reduced video duration to 42.97% on average) are extracted and stored in an encrypted format to avoid un-trusted access to sensitive event-based data. Fernet, a symmetric encryption algorithm is applied to the summarized videos along with Diffie–Hellman (DH) key exchange algorithm and SHA256 hash algorithm. The encryption key is deleted immediately after the encryption process, and the decryption key is generated at the system of authorised stakeholders, which prevents the key from a man-in-the-middle (MITM) attack.

Video summarization for event-centric videos

Video summarization has long been used to ease video browsing and plays a more crucial role with the explosion of online videos. In the context of event-centric videos, we aim to extract the corresponding clips of more important events in the video. To tackle the dilemma between the detection precision and the clip completeness faced by previous methods, we present an efficient Boundary-Aware framework for Summary clip Extraction (BASE) to extract summary clips with more precise boundaries while maintaining their completeness. Specifically, we propose a new distance-based importance signal to reflect the progress information in each video. The signal can not only help us to detect boundaries with higher precision, but also make it possible to preserve the clip completeness. For the feature presentation part, we also explore new information types to facilitate video summarization. Our approach outperforms current state-of-the-art video summarization models in terms of more precise clip boundaries and more complete summary clips. Note that we even yield comparable results to manual annotations.

Scene Classification for Sports Video Using Transfer Learning

Abstract: The quality of scene classification is particularly crucial because it is a key step in both video summarization and video analysis. This article proposes a strategy to accomplish high-quality scene classification and focuses on several practical implementation gaps over the previous methodologies. We identify five scene types, including batting, bowling, boundary, crowd, and close-up, using cricket as a case study. For scene categorization, we use the ResNet152 model which is pre-trained on imagenet. The proposed method employs new, fully connected layers in an encoder fashion. We also prepared a dataset of cricket video and manually labelled them into five classes which was later used to train the model. The model achieved mean accuracy of 99.15 upon training on the dataset.

Stochastic Study on CNN Approach for Classifying Images

Recent years have seen a surge in interest in artificial intelligence and computer vision applications, which are essential for many real-time tasks like video summarization, picture retrieval, and image categorization. Convolutional neural networks, which are employed in numerous image processing and computer vision applications, are one of the most popular deep learning techniques. In this study, a CNN model for classifying colour images is proposed. The proposed model was developed using deep learning tools in MATLAB. Additionally, three separate datasets of varying sizes were used to test the suggested model. With the largest dataset, the suggested model produced the highest accuracy, precision, and sensitivity results, which are as follows: accuracy is 0.9924, precision is 0.9947, and sensitivity is 0.9931 when compared to other models