Video Summarization Method Research Articles

Towards key-frame extraction methods for 3D video: a review

The increasing rate of creation and use of 3D video content leads to a pressing need for methods capable of lowering the cost of 3D video searching, browsing and indexing operations, with improved content selection performance. Video summarisation methods specifically tailored for 3D video content fulfil these requirements. This paper presents a review of the state-of-the-art of a crucial component of 3D video summarisation algorithms: the key-frame extraction methods. The methods reviewed cover 3D video key-frame extraction as well as shot boundary detection methods specific for use in 3D video. The performance metrics used to evaluate the key-frame extraction methods and the summaries derived from those key-frames are presented and discussed. The applications of these methods are also presented and discussed, followed by an exposition about current research challenges on 3D video summarisation methods.

Visual saliency models for summarization of diagnostic hysteroscopy videos in healthcare systems.

In clinical practice, diagnostic hysteroscopy (DH) videos are recorded in full which are stored in long-term video libraries for later inspection of previous diagnosis, research and training, and as an evidence for patients’ complaints. However, a limited number of frames are required for actual diagnosis, which can be extracted using video summarization (VS). Unfortunately, the general-purpose VS methods are not much effective for DH videos due to their significant level of similarity in terms of color and texture, unedited contents, and lack of shot boundaries. Therefore, in this paper, we investigate visual saliency models for effective abstraction of DH videos by extracting the diagnostically important frames. The objective of this study is to analyze the performance of various visual saliency models with consideration of domain knowledge and nominate the best saliency model for DH video summarization in healthcare systems. Our experimental results indicate that a hybrid saliency model, comprising of motion, contrast, texture, and curvature saliency, is the more suitable saliency model for summarization of DH videos in terms of extracted keyframes and accuracy.

An Interactive SpiralTape Video Summarization

A majority of video summarization systems use linear representations, such as rectangular storyboards and timelines at linear scales. In this paper, we propose a novel nonlinear dynamic representation called SpiralTape that summarizes a video in a smooth spiral pattern. SpiralTape provides an unusual and fresh activity suitable for stimulating environments such as science and technology museums, in which children or young individuals can have enjoyable experiences that create meaningful learning outcomes. In addition, SpiralTape provides an uninterrupted overall structure of video content and takes design principles including compactness, continuity, efficient overview, and interactivity into consideration. A working SpiralTape system was developed and deployed in pilot applications and exhibitions. Elaborate user studies with evaluation benchmarks on multiple metrics were conducted to compare SpiralTape with two representative linear video summarization methods and a state-of-the-art radial video visualization. The evaluation results demonstrate the effectiveness and natural interaction performance of SpiralTape.

A static video summarization method based on the sparse coding of features and representativeness of frames

This paper presents a video summarization method that is specifically for the static summary of consumer videos. Considering that the consumer videos usually have unclear shot boundaries and many low-quality or meaningless frames, we propose a two-step approach where the first step skims a video and the second step performs content-aware clustering with keyframe selection. Specifically, the first step removes most of redundant frames that contain only little new information by employing the spectral clustering method with color histogram features. As a result, we obtain a condensed video that is shorter and has clearer temporal boundaries than the original. In the second step, we perform rough temporal segmentation and then apply refined clustering for each of the temporal segments, where each frame is represented by the sparse coding of SIFT features. The keyframe selection from each cluster is based on the measure of representativeness and visual quality of frames, where the representativeness is defined from the sparse coding and the visual quality is the combination of contrast, blur, and image skew measures. The problem of keyframe selection is to find the frames that have both representativeness and high quality, which is formulated as an optimization problem. Experiments on videos with various lengths show that the resulting summaries closely follow the important contents of videos.

A novel clustering method for static video summarization

Static video summarization is recognized as an effective way for users to quickly browse and comprehend large numbers of videos. In this paper, we formulate static video summarization as a clustering problem. Inspired by the idea from high density peaks search clustering algorithm, we propose an effective clustering algorithm by integrating important properties of video to gather similar frames into clusters. Finally, all clusters' center will be collected as static video summarization. Compared with existing clustering-based video summarization approaches, our work can detect frames which are highly relevant and generate representative clusters automatically. We evaluate our proposed work by comparing it with several state-of-the-art clustering-based video summarization methods and some classical clustering algorithms. The experimental results evidence that our proposed method has better performance and efficiency.

Scalable video summarization via sparse dictionary learning and selection simultaneously

Every day, a huge amount of video data is generated worldwide and processing this kind of data requires powerful resources in terms of time, manpower, and hardware. Therefore, to help quickly understand the content of video data, video summarization methods have been proposed. Recently, sparse formulation-based methods have been found to be able to summarize a large amount of video compared to other methods. In this paper, we propose a new method in which video summarization is performed as training and selection sparse dictionary problem simultaneously. It is shown that the proposed method is able to improve the summarization of a large amount of video data compared to other methods. Finally, the performance of the proposed method is compared to state-of-the-art methods using standard data sets, in which the key frames are manually tagged. The obtained results demonstrate that the proposed method improves video summarization compared to other methods.

A Survey on Video Summarization Techniques

Video summarization methods attempt to abstract the main occurrences, scenes, or objects in a clip in order to provide an easily interpreted synopsis. The main aim of Video summarization is to provide clear analysis of video by removing duplications and extracting key frames from the video. Video Summarization will divide the frames of the video into blocks and calculating the mean, variance, skew, kurtosis histogram of every block and comparing the same with the corresponding blocks of the next frame. There are many different methods used for Key frame extraction in video Summarization. Some important methods are compared. The frame with highest mean is selected as the key frame. The best method is selected based on the color distribution.

Structuring Lecture Videos by Automatic Projection Screen Localization and Analysis.

We present a fully automatic system for extracting the semantic structure of a typical academic presentation video, which captures the whole presentation stage with abundant camera motions such as panning, tilting, and zooming. Our system automatically detects and tracks both the projection screen and the presenter whenever they are visible in the video. By analyzing the image content of the tracked screen region, our system is able to detect slide progressions and extract a high-quality, non-occluded, geometrically-compensated image for each slide, resulting in a list of representative images that reconstruct the main presentation structure. Afterwards, our system recognizes text content and extracts keywords from the slides, which can be used for keyword-based video retrieval and browsing. Experimental results show that our system is able to generate more stable and accurate screen localization results than commonly-used object tracking methods. Our system also extracts more accurate presentation structures than general video summarization methods, for this specific type of video.

Static and Multiresolution Feature Extraction for Video Summarization

The objective of this paper is to develop a video summarization system to extract significant frames of interest from a given video. To meet the objective, it is proposed to consider both the static features and the wavelet features. Visual attention integrated from the static and wavelet feature set are combined using a prioritized fusion method. Experimental results shows that the static features dominate in certain videos and wavelet features dominate in certain videos. Hence, the proposed fusion approach is suitable for slow motion videos and fast moving videos. Further, the performance of proposed work outperforms the state-of-art methods for video summarization.

Video summarization via minimum sparse reconstruction

The rapid growth of video data demands both effective and efficient video summarization methods so that users are empowered to quickly browse and comprehend a large amount of video content. In this paper, we formulate the video summarization task with a novel minimum sparse reconstruction (MSR) problem. That is, the original video sequence can be best reconstructed with as few selected keyframes as possible. Different from the recently proposed convex relaxation based sparse dictionary selection method, our proposed method utilizes the true sparse constraint L0 norm, instead of the relaxed constraint L2,1 norm, such that keyframes are directly selected as a sparse dictionary that can well reconstruct all the video frames. An on-line version is further developed owing to the real-time efficiency of the proposed MSR principle. In addition, a percentage of reconstruction (POR) criterion is proposed to intuitively guide users in obtaining a summary with an appropriate length. Experimental results on two benchmark datasets with various types of videos demonstrate that the proposed methods outperform the state of the art.

Comparisons of Video Summarization Methods

Video summarization is a process of removing the redundant frames and generating the most informative key frames of the videos. In this paper we have explained two efficient methods for video summarization and given the comparison between these two methods. The first method is Video summarization using CLD feature extraction and adaptive threshold technique for shot boundary detection. Second method is video summarization using aggregation function where three different features such as Histogram difference, correlation difference and moment of inertia difference are combined and that difference value is compared with the predefined threshold value.

Speeding up a Video Summarization Approach Using GPUs and Multicore CPUs

The recent progress of digital media has stimulated the creation, storage and distribution of data, such as digital videos, generating a large volume of data and requiring efficient technolo- gies to increase the usability of these data. Video summarization methods generate concise summaries of video contents and enable faster browsing, indexing and accessing of large video collections, however, these methods often perform slow with large duration and high quality video data. One way to reduce this long time of execution is to develop a parallel algorithm, using the advantages of the recent computer architectures that allow high parallelism. This paper introduces parallelizations of a summarization method called VSUMM, targetting either Graphic Processor Units (GPUs) or multicore Central Processor Units (CPUs), and ultimately a sensible distribution of computation steps onto both hardware to maximise performance, called “hybrid”. We performed experiments using 180 videos varying frame resolution (320 × 240, 640 × 360, and 1920 × 1080) and video length (1, 3, 5, 10, 20, and 30minutes). From the results, we observed that the hybrid version reached the best results in terms of execution time, achieving 7× speed up in average.

Automatic summarization and annotation of videos with lack of metadata information

The advances in computer and network infrastructure together with the fast evolution of multimedia data has resulted in the growth of attention to the digital video’s development. The scientific community has increased the amount of research into new technologies, with a view to improving the digital video utilization: its archiving, indexing, accessibility, acquisition, store and even its process and usability. All these parts of the video utilization entail the necessity of the extraction of all important information of a video, especially in cases of lack of metadata information. The main goal of this paper is the construction of a system that automatically generates and provides all the essential information, both in visual and textual form, of a video. By using the visual or the textual information, a user is facilitated on the one hand to locate a specific video and on the other hand is able to comprehend rapidly the basic points and generally, the main concept of a video without the need to watch the whole of it. The visual information of the system emanates from a video summarization method, while the textual one derives from a key-word-based video annotation approach. The video annotation technique is based on the key-frames, that constitute the video abstract and therefore, the first part of the system consists of the new video summarization method.According to the proposed video abstraction technique, initially, each frame of the video is described by the Compact Composite Descriptors (CCDs) and a visual word histogram. Afterwards, the proposed approach utilizes the Self-Growing and Self-Organized Neural Gas (SGONG) network, with a view to classifying the frames into clusters. The extraction of a representative key frame from every cluster leads to the generation of the video abstract. The most significant advantage of the video summarization approach is its ability to calculate dynamically the appropriate number of final clusters. In the sequel, a new video annotation method is applied to the generated video summary leading to the automatic generation of key-words capable of describing the semantic content of the given video. This approach is based on the recently proposed N-closest Photos Model (NCP). Experimental results on several videos are presented not only to evaluate the proposed system but also to indicate its effectiveness.

Video summarisation based on collaborative temporal tags

PurposeVideo summarisation is one of the most active fields in content‐based video retrieval research. A new video summarisation scheme is proposed by this paper based on socially generated temporal tags.Design/methodology/approachTo capture users' collaborative tagging activities the proposed scheme maintains video bookmarks, which contain some temporal or positional information about videos, such as relative time codes or byte offsets. For each video all the video bookmarks collected from users are then statistically analysed in order to extract some meaningful key frames (the video equivalent of keywords), which collectively constitute the summary of the video.FindingsCompared with traditional video summarisation methods that use low‐level audio‐visual features, the proposed method is based on users' high‐level collaborative activities, and thus can produce semantically more important summaries than existing methods.Research limitations/implicationsIt is assumed that the video frames around the bookmarks inserted by users are informative and representative, and therefore can be used as good sources for summarising videos.Originality/valueFolksonomy, commonly called collaborative tagging, is a Web 2.0 method for users to freely annotate shared information resources with keywords. It has mostly been used for collaboratively tagging photos (Flickr), web site bookmarks (Del.icio.us), or blog posts (Technorati), but has never been applied to the field of automatic video summarisation. It is believed that this is the first attempt to utilise users' high‐level collaborative tagging activities, instead of low‐level audio‐visual features, for video summarisation.

Automatic categorization and summarization of documentaries

In this paper, we propose automatic categorization and summarization of documentaries using subtitles of videos. We propose two methods for video categorization. The first makes unsupervised categorization by applying natural language processing techniques on video subtitles and uses the WordNet lexical database and WordNet domains. The second has the same extraction steps but uses a learning module to categorize. Experiments with documentary videos give promising results in discovering the correct categories of videos. We also propose a video summarization method using the subtitles of videos and text summarization techniques. Significant sentences in the subtitles of a video are identified using these techniques and a video summary is then composed by finding the video parts corresponding to these summary sentences.

Activity-driven content adaptation for effective video summarization

In this paper, we present a novel method for content adaptation and video summarization fully implemented in compressed-domain. Firstly, summarization of generic videos is modeled as the process of extracted human objects under various activities/events. Accordingly, frames are classified into five categories via fuzzy decision including shot changes ( cut and gradual transitions), motion activities ( camera motion and object motion) and others by using two inter-frame measurements. Secondly, human objects are detected using Haar-like features. With the detected human objects and attained frame categories, activity levels for each frame are determined to adapt with video contents. Continuous frames belonging to same category are grouped to form one activity entry as content of interest (COI) which will convert the original video into a series of activities. An overall adjustable quota is used to control the size of generated summarization for efficient streaming purpose. Upon this quota, the frames selected for summarization are determined by evenly sampling the accumulated activity levels for content adaptation. Quantitative evaluations have proved the effectiveness and efficiency of our proposed approach, which provides a more flexible and general solution for this topic as domain-specific tasks such as accurate recognition of objects can be avoided.