Key Frame Selection Research Articles

Predicting Important Objects for Egocentric Video Summarization

We present a video summarization approach for egocentric or "wearable" camera data. Given hours of video, the proposed method produces a compact storyboard summary of the camera wearer's day. In contrast to traditional keyframe selection techniques, the resulting summary focuses on the most important objects and people with which the camera wearer interacts. To accomplish this, we develop region cues indicative of high-level saliency in egocentric video---such as the nearness to hands, gaze, and frequency of occurrence---and learn a regressor to predict the relative importance of any new region based on these cues. Using these predictions and a simple form of temporal event detection, our method selects frames for the storyboard that reflect the key object-driven happenings. We adjust the compactness of the final summary given either an importance selection criterion or a length budget; for the latter, we design an efficient dynamic programming solution that accounts for importance, visual uniqueness, and temporal displacement. Critically, the approach is neither camera-wearer-specific nor object-specific; that means the learned importance metric need not be trained for a given user or context, and it can predict the importance of objects and people that have never been seen previously. Our results on two egocentric video datasets show the method's promise relative to existing techniques for saliency and summarization.

Comparing key frame selection for one-two hand gesture recognition using different methods

Hand gesture recognition system has gained potential importance in application areas of human-computer interaction, machine vision, etc. Vision-based hand gesture recognition involves visual analysis of hand shape, position and/or motion. Finite State Machine (FSM) and Dynamic Time Warping (DTW) are most prominent methods used for gesture recognition. In this paper, we use object-based key frame selection from video sequence for segmenting Video Object Plane (VOP). Each VOP is a meaningful hand position, where Hausdorff and Euclidean distance are used for shape similarity. The key VOPs are selected on the basis of hand shape changes significantly and subsequently. The core approach of proposed work is that the frames of gesture video sequence are used with FSM and non-linear time alignment methods with key frame selection facility and trajectory features. Experimental results show accuracy and effectiveness of proposed system, for one-handed American and two-handed British sign language gesture recognition.

Terrain Classification From Body-Mounted Cameras During Human Locomotion.

This paper presents a novel algorithm for terrain type classification based on monocular video captured from the viewpoint of human locomotion. A texture-based algorithm is developed to classify the path ahead into multiple groups that can be used to support terrain classification. Gait is taken into account in two ways. Firstly, for key frame selection, when regions with homogeneous texture characteristics are updated, the frequency variations of the textured surface are analyzed and used to adaptively define filter coefficients. Secondly, it is incorporated in the parameter estimation process where probabilities of path consistency are employed to improve terrain-type estimation. When tested with multiple classes that directly affect mobility-a hard surface, a soft surface, and an unwalkable area-our proposed method outperforms existing methods by up to 16%, and also provides improved robustness.

Temporal segmentation and keyframe selection methods for user-generated video search-based annotation

In this paper we propose a temporal segmentation and a keyframe selection method for User-Generated Video (UGV). Since UGV is rarely structured in shots and usually user’s interest are revealed through camera movements, a UGV temporal segmentation system has been proposed that generates a video partition based on a camera motion classification. Motion-related mid-level features have been suggested to feed a Hierarchical Hidden Markov Model (HHMM) that produces a user-meaningful UGV temporal segmentation. Moreover, a keyframe selection method has been proposed that picks a keyframe for fixed-content camera motion patterns such as zoom, still, or shake and a set of keyframes for varying-content translation patterns.The proposed video segmentation approach has been compared to a state-of-the-art algorithm, achieving 8% performance improvement in a segmentation-based evaluation. Furthermore, a complete search-based UGV annotation system has been developed to assess the influence of the proposed algorithms on an end-user task. To that purpose, two UGV datasets have been developed and made available online. Specifically, the relevance of the considered camera motion types has been analyzed for these two datasets, and some guidelines are given to achieve the desired performance-complexity tradeoff. The keyframe selection algorithm for varying-content translation patterns has also been assessed, revealing a notable contribution to the performance of the global UGV annotation system. Finally, it has been shown that the UGV segmentation algorithm also produces improved annotation results with respect to a fixed-rate keyframe selection baseline or a traditional method relying on frame-level visual features.

Video super-resolution based on automatic key-frame selection and feature-guided variational optical flow

This paper proposes a new video super-resolution method based on feature-guided variational optical flow. The key-frames are automatically selected and super-resolved using a method based on sparse regression. To overcome the blocking artifacts and deal with the case of small structures with large displacement, an efficient method based on feature-guided variational optical flow is used to super-resolve the non-key-frames. Experimental results show that the proposed method outperforms the existing benchmark in terms of both subjective visual quality and objective peak signal-to-noise ratio (PSNR). The average PSNR improvement from the bi-cubic interpolation is 7.15dB for four datasets. Thanks to the flexibility of designed automatic key-frame selection and the validness of feature-guided variational optical flow, the proposed method is applicable to various practical video sequences.

CONTENT BASED VIDEO RETRIEVAL SYSTEM

Video retrieval is a young field which has its genealogy rooted in artificial intelligence, digital signal processing, statistics, natural language understanding, databases, psychology, computer vision, and pattern recognition. However, none of these parental fields alone has been able to directly solve the retrieval problem. In this paper we present a System that supports video mining from multimedia warehouse using multimodal feature has two stages, the first one is building the multimedia warehouse and the second one is retrieving the video from that multimedia warehouse. The Video Retrieval system includes various steps: Video Segmentation, Key frames Selection, Feature Extraction. For retrieving the video from warehouse, the retrieval subsystem processes the presented query, performs similarity matching operations and this can be done using Euclidian Distance Algorithm, and finally displays the result to end user.

µ-Mosaicing and Thermo-Mosaicing using Image In-painting

Microscopy has enormous applications in both medical and non-medical fields. This has gained interest of many researchers in the field of digital image processing to develop algorithms and applications helping in improving their observation and analysis. Here, novel method called ‘μMosaicing’ (read as Micro-Mosaicing) is introduced to create mosaics of microscopic images from a video sequence to obtain large view, high resolution images of specimen. A thermal camera forms images by sensing the infrared radiations from an object. High resolution thermal images can be used to determine temperatures in detailed thermal analysis. However, the cost of thermal camera increases with their resolution. While, thermo-graphy plays an important role in the diagnosis of various diseases, high resolution large field of view images will be certainly helpful in diagnosis and treatment. So a new approach to create mosaics of thermal images from video sequence, called ‘Thermo-Mosaicing’, is proposed which can find wide applications in medical field. Both, μ-Mosaicing and Thermo-Mosaicing employ Key Frame Selection Algorithm, Image Matching using Normalized Cross Correlation (NCC), Image Stitching using 3-point Mask Blending Algorithm and Image In-painting.

Surveillance Video Retrieval Based on Moving Objects Detection

In this article, we proposed a surveillance video retrieval method based on detection of moving objects and researched on key problems in surveillance video, such as moving objects detection, video segmentation, selection and description of key frames, and measurement of feature similarities. We detected moving objects via movement detection algorithm based on codebook, and realized automatic segmentation of surveillance video via calculating present Frame Movement Amount based on the foreground and background information in movement detection. After the above process, we proposed a video retrieval method based on the combination of SIFT and color feature.

Real-Time Visual SLAM for Autonomous Underwater Hull Inspection Using Visual Saliency

This paper reports a real-time monocular visual simultaneous localization and mapping (SLAM) algorithm and results for its application in the area of autonomous underwater ship hull inspection. The proposed algorithm overcomes some of the specific challenges associated with underwater visual SLAM, namely, limited field of view imagery and feature-poor regions. It does so by exploiting our SLAM navigation prior within the image registration pipeline and by being selective about which imagery is considered informative in terms of our visual SLAM map. A novel online bag-of-words measure for intra and interimage saliency are introduced and are shown to be useful for image key-frame selection, information-gain-based link hypothesis, and novelty detection. Results from three real-world hull inspection experiments evaluate the overall approach, including one survey comprising a 3.4-h/2.7-km-long trajectory.

Keypoint-Based Keyframe Selection

Keyframe selection has been crucial for effective and efficient video content analysis. While most of the existing approaches represent individual frames with global features, we, for the first time, propose a keypoint-based framework to address the keyframe selection problem so that local features can be employed in selecting keyframes. In general, the selected keyframes should both be representative of video content and containing minimum redundancy. Therefore, we introduce two criteria, coverage and redundancy, based on keypoint matching in the selection process. Comprehensive experiments demonstrate that our approach outperforms the state of the art.

Tsallis entropy-based information measures for shot boundary detection and keyframe selection

Automatic shot boundary detection and keyframe selection constitute major goals in video processing. We propose two different information-theoretic approaches to detect the abrupt shot boundaries of a video sequence. These approaches are, respectively, based on two information measures, Tsallis mutual information and Jensen–Tsallis divergence, that are used to quantify the similarity between two frames. Both measures are also used to find out the most representative keyframe of each shot. The representativeness of a frame is basically given by its average similarity with respect to the other frames of the shot. Several experiments analyze the behavior of the proposed measures for different color spaces (RGB, HSV, and Lab), regular binnings, and entropic indices. In particular, the Tsallis mutual information for the HSV and Lab color spaces with only 8 regular bins for each color component and an entropic index between 1.5 and 1.8 substantially improve the performance of previously proposed methods based on mutual information and Jensen–Shannon divergence.

Optimized selection of key frames for monocular videogrammetric surveying of civil infrastructure

Videogrammetry is an inexpensive and easy-to-use technology for spatial 3D scene recovery. When applied to large scale civil infrastructure scenes, only a small percentage of the collected video frames are required to achieve robust results. However, choosing the right frames requires careful consideration. Videotaping a built infrastructure scene results in large video files filled with blurry, noisy, or redundant frames. This is due to frame rate to camera speed ratios that are often higher than necessary; camera and lens imperfections and limitations that result in imaging noise; and occasional jerky motions of the camera that result in motion blur; all of which can significantly affect the performance of the videogrammetric pipeline. To tackle these issues, this paper proposes a novel method for automating the selection of an optimized number of informative, high quality frames. According to this method, as the first step, blurred frames are removed using the thresholds determined based on a minimum level of frame quality required to obtain robust results. Then, an optimum number of key frames are selected from the remaining frames using the selection criteria devised by the authors. Experimental results show that the proposed method outperforms existing methods in terms of improved 3D reconstruction results, while maintaining the optimum number of extracted frames needed to generate high quality 3D point clouds.

Video fingerprinting based on graph model

A robust video fingerprinting based on graph model is proposed in this paper, where two graph models are constructed for key frames selection and foreground extraction, respectively. First, the video is represented as a complete undirected graph and a binary tree is formed using normalized cut algorithm to select key frames. Then, the pixels of each key frame are modeled as a Markov Random Field and another graph model is formed to extract foreground by graph cut. Finally, the fourth-order cumulant of foreground is computed to generate video fingerprints. Experimental results show that the proposed algorithm has good robustness and discrimination.

An Improved Keyframe Extraction Method Based on HSV Colour Space

Video segmentation and keyframe extraction are the basis of Content-based Video Retrieval (CBVR), in which keyframe selection is at the very core of CBVR. At shot level, key-frame extraction provides sufficient indexing and browsing of large video databases. In this paper, we proposed two improved approaches of key-frame extraction for video summarization. In our first synthesis method based on histogram-based method and pixel-based method, videos were firstly segmented into shots according to video content, by our improved histogram-based method, with the use of histogram intersection and nonuniform partitioning and weighting. Then, the obtained results are secondly detected to optimize the results. On the other hand, we realized an improved clustering algorithm for video shot segmentation, in consideration of video characteristics. Within each shot, key-frames were determined with the calculation of image entropy of every frame in HSV colour space. Our simulation results in section 4 prove that extracted key frames with our method are compact and faithful to the original video. Moreover, according to the types of test videos, different methods for shot segmentation are highly recommended.

Dominant Color and Texture Approached for Content Based Video Images Retrieval

Content-based retrieval allows finding information by searching its content rather than its attributes. Content-based search and retrieval of video data becomes a challenging and important problem. Every year video content is growing in volume and there are different techniques available to capture, compress, display, store and transmit video while editing and manipulating video based on their content is still a non-trivial activity. Recent advances in multimedia technologies allow the capture and storage of video data with relatively inexpensive computers. However, without appropriate search techniques all these data are hardly usable. Today research is focused on video retrieval.Moreover, content-based video retrieval system requires first of all segment the video stream into separate shots. Video Shot Afterwards features are extracted for video shots representation. And finally, choose a similarity/distance metric and an algorithm that is efficient enough to retrieve query - related videos results. There are two main issues in this process; the first is how to determine the best way for video segmentation and key frame selection. The second is the features used for video representation. Various features can be extracted for this sake including either low or high level features. A key issue is how to bridge the gap between low and high level features. In this paper we presented approach for content based video retrieval based on Dominant color and texture of a video image. We also talk about video Representation, feature extraction from like texture, dominant color and color histogram from video frame. Keywords- Video retrieval, dominant color, Gray level co occurrence matrix. Feature extraction, Key frame extraction, Video representation, and Video segmentation. Image Retrieval, color Histogram

Sports video summarization based on motion analysis

Non-annotated video is more common than ever and this fact leads to an emerging field called video summarization. Key frame selection using motion analysis can greatly increase the understanding of the video content by presenting a series of frames summarizing the intended video. In this paper, we present an automatic video summarization technique based on motion analysis. The proposed technique defines motion metrics estimated from two optical flow algorithms, each using two different key frame selection criteria. We conducted a subjective user study to evaluate the performance of the motion metrics. The summarization process is threshold free and experimental results have verified the effectiveness of the method.

Boosted key-frame selection and correlated pyramidal motion-feature representation for human action recognition

In this paper we propose a novel method for human action recognition based on boosted key-frame selection and correlated pyramidal motion feature representations. Instead of using an unsupervised method to detect interest points, a Pyramidal Motion Feature (PMF), which combines optical flow with a biologically inspired feature, is extracted from each frame of a video sequence. The AdaBoost learning algorithm is then applied to select the most discriminative frames from a large feature pool. In this way, we obtain the top-ranked boosted frames of each video sequence as the key frames which carry the most representative motion information. Furthermore, we utilise the correlogram which focuses not only on probabilistic distributions within one frame but also on the temporal relationships of the action sequence. In the classification phase, a Support-Vector Machine (SVM) is adopted as the final classifier for human action recognition. To demonstrate generalizability, our method has been systematically tested on a variety of datasets and shown to be more effective and accurate for action recognition compared to the previous work. We obtain overall accuracies of: 95.5%, 93.7%, and 36.5% with our proposed method on the KTH, the multiview IXMAS and the challenging HMDB51 datasets, respectively.

A Novel Algorithm for Key Frames Selection

The main objective of this study framework is to help the user to management the interesting video images from the huge storage device. The study can be divided into three main steps is listing as follows: shot boundary detection, key frame selection and evaluation method are designed by analyzing the video content. In this paper we proposed a novel algorithm for shot boundary detection. First, transform the RGB color model into LUV color model according our experimental results. The useful features of each frame are extracted from sequence images by the automatically computation of their color moment-preserving and matching 64-predefined bit planes in order to detect the shot boundary. The excellent shot boundary detection will result in the good key frames selection. Second, the key frame selection is proceed by applying the geometric invariant moments which successful be used in target tracing, pattern recognition, and facial recognition. In our experiments we used five test sequences that belong to five different program categories: movies, advertising, sport game, news, and commercial documentary. The experimental results showed that our method is feasible, effective, and is better than the former approaches.

Adaptive Image Enhancement and Accelerated Key Frame Selection for Echocardiogram Images

Adaptive Image Enhancement and Accelerated Key Frame Selection for Echocardiogram Images

Optimal key-frame selection for video-based structure-from-motion

Proposed is a novel key-frame selection method for the video-based structure-from-motion technique. This method consists of two main steps. First, unreliable features are eliminated by analysing the lifetimes of the tracked features. Next, an optimal set of key-frames is selected considering sufficient baselines between key-frames, data redundancy, and the avoidance of degeneracy. The proposed method is experimentally verified in terms of accuracy and stability for various video sequences.