Key Frame Selection Research Articles

Dynamic selection and effective compression of key frames for video abstraction

This paper reports on a new key frame based video abstraction method. With our method, a video sequence is first segmented into a number of video shots. Several key frames are selected in each shot using a dynamic selection technique. For these key frames, a motion-based clustering algorithm is applied so that key frames in the same cluster are alike in sense of motion compensation error, while those from different clusters are quit dissimilar. Then a novel cluster-based coding scheme is developed for efficient representation of the key frames. Simulations show that the proposed method can select key frames according to the dynamics of a video sequence and abstract the video with different levels of scalability.

Iterative key frame selection in the rate-constraint environment

Video representation through key frames has been addressed frequently as an efficient way of preserving the whole temporal information of sequence with a considerably smaller amount of data. Such compact video representation is suitable for the purpose of video browsing in limited storage or transmission bandwidth environments. In this case, the controllability of the total key frame number (i.e. key frame rate) depending on the storage or bandwidth capacity is an important requirement for the key frame selection method. In this paper, we present a sequential key frame selection method when the number of key frames is given as a constraint. It first selects the pre-determined number of initial key frames and time-intervals. Then, it adjusts the positions of key frames and time-intervals by iteration, which reduces the distortion step by step. Experimental results demonstrate the improved performance of our algorithm over the existing approaches.

Video segmentation using a histogram-based fuzzy c-means clustering algorithm

The purpose of video segmentation is to segment video sequence into shots where each shot represents a sequence of frames having the same contents, and then select key frames from each shot for indexing. Existing video segmentation methods can be classified into two groups: the shot change detection (SCD) approach for which thresholds have to be pre-assigned, and the clustering approach for which a prior knowledge of the number of clusters is required. In this paper, we propose a video segmentation method using a histogram-based fuzzy c-means (HBFCM) clustering algorithm. This algorithm is a hybrid of the two approaches aforementioned, and is designed to overcome the drawbacks of both approaches. The HBFCM clustering algorithm is composed of three phases: the feature extraction phase, the clustering phase, and the key-frame selection phase. In the first phase, differences between color histogram are extracted as features. In the second phase, the fuzzy c-means (FCM) is used to group features into three clusters: the shot change (SC) cluster, the suspected shot change (SSC) cluster, and the no shot change (NSC) cluster. In the last phase, shot change frames are identified from the SC and the SSC, and then used to segment video sequences into shots. Finally, key frames are selected from each shot. Simulation results indicate that the HBFCM clustering algorithm is robust and applicable to various types of video sequences.

An interactive video delivery and caching system using video summarization

With the advance of high-speed network technologies, the availability and popularity of streaming media content over the Internet has grown rapidly in recent years. The delivery and caching of streaming media must be handled in a different fashion than that of traditional non-streaming objects such as HTML or image files, because of its distinct characteristics and user viewing patterns. We propose a novel scheme that provides users with the video summary (a number of key-frame images) before they download the file, and options for them to select the starting playback position. We introduce the content analysis service to achieve these functionalities. The video content analysis performs shot boundary detection, key-frame selection, and face detection and tracking. The results of the processing are a segmented video sequence and an XML-based meta-data describing the video content. We also design a caching system that utilizes our video abstraction and summarization technique. Our integrated video delivery and caching system combines content-aware segmentation, prefix caching, prefetching, and cooperative caching. We describe how our scheme can be applied in three proposed caching architectures.

Non-sequential video content representation using temporal variation of feature vectors

An efficient and low complexity algorithm for non-sequential video content representation is proposed. Our method is based on extracting a set of limited but meaningful frames (key-frames), able to represent the video content. The temporal variation of feature vectors for all frames within a shot, which form a trajectory in a high dimensional space, is used for key-frame selection. In particular, key-frames are extracted by estimating appropriate curve points, able to characterize the feature trajectory. The magnitude of the second derivative of the frame feature vectors with respect to time is used as a curvature measure in our approach. Due to low complexity of the algorithm, the proposed method can be easily implemented in hardware devices of even low processing capabilities thus can be embedded in many consumer electronics systems. For feature vector formulation, the video is first analyzed and several descriptors are extracted using a multiscale implementation of the recursive shortest spanning tree (RSST) algorithm, which significantly reduces the segmentation complexity. In addition, the whole procedure exploits information that exists in MPEG video databases so as to achieve a faster implementation. Finally, the extracted descriptors are classified using a fuzzy formulation scheme. Experimental results to real-life video sequences are presented to indicate the good performance of the proposed algorithm.

Content analysis of video using principal components

We use principal component analysis (PCA) to reduce the dimensionality of features of video frames for the purpose of content description. This low-dimensional description makes practical the direct use of all the frames of a video sequence in later analysis. The PCA representation circumvents or eliminates several of the stumbling blocks in current analysis methods and makes new analyses feasible. We demonstrate this with two applications. The first accomplishes high-level scene description without shot detection and key-frame selection. The second uses the time sequences of motion data from every frame to classify sports sequences.

Content-based access to video objects: Temporal Segmentation, visual summarization, and feature extraction

The classical approach to content-based video access has been ‘frame-based’, consisting of shot boundary detection, followed by selection of key frames that characterize the visual content of each shot, and then clustering of the camera shots to form story units. However, in an object-based multimedia environment, content-based random access to individual video objects becomes a desirable feature. To this effect, this paper introduces an ‘object-based’ approach to temporal video partitioning and content-based indexing, where the basic indexing unit is ‘lifespan of a video object’, rather than a ‘camera shot’ or a ‘story unit’. We propose to represent each video object by an adaptive 2D triangular mesh. A mesh-based object tracking scheme is then employed to compute the motion trajectories of all mesh node points until the object exits the field of view. A new similarity measure that is based on motion discontinuities and shape changes of the tracked object is defined to detect content changes, resulting in temporal lifespan segments. A set of ‘key snapshots’ which constitute a visual summary of the lifespan of the object is automatically selected. These key snapshots are then used to animate objects of interest using tracked motion trajectories for a moving visual representation. The proposed scheme provides such functionalities as object-based search/browsing for interactive video retrieval, surveillance video analysis, and object-based content manipulation/editing for studio postprocessing and desktop multimedia authoring. The approach is applicable to any video data where the initial appearance of object(s) can be specified, and the object motion can be modeled by a piecewise affine transformation. The system is demonstrated using different types of video: virtual studio productions (composited video), surveillance video, and TV broadcast video.

Indexing and retrieval of the MPEG compressed video

To keep pace with the increased popularity of digital video as an archival medium, the development of techniques for fast and efficient analysis of video streams is essential. In particular, solutions to the problems of storing, indexing, browsing, and retrieving video data from large multimedia databases are necessary to allow access to these collections. Given that video is often stored efficiently in a compressed format, the costly overhead of decompression can be reduced by analyzing the compressed representation directly. In earlier work, we presented compressed domain parsing techniques which identified shots, subshots, and scenes. In this article, we present efficient key frame selection, feature extraction, indexing, and retrieval techniques that are directly applicable to MPEG compressed video. We develop a frame type independent representation which normalizes spatial and temporal features including frame type, frame size, macroblock encoding, and motion compensation vectors. Features for indexing are derived directly from this representation and mapped to a low-dimensional space where they can be accessed using standard database techniques. Spatial information is used as primary index into the database and temporal information is used to rank retrieved clips and enhance the robustness of the system. The techniques presented enable efficient indexing, querying, and retrieval of compressed video as demonstrated by our system which typically takes a fraction of a second to retrieve similar video scenes from a database, with over 95% recall.