Keyframe Selection Techniques Research Articles

FastPicker: Adaptive independent two-stage video-to-video summarization for efficient action recognition

Video datasets suffer from huge inter-frame redundancy, which prevents deep networks from learning effectively and increases computational costs. Therefore, several methods adopt random/uniform frame sampling or key-frame selection techniques. Unfortunately, most of the learnable frame selection methods are customized for specific models and lack generality, independence, and scalability. In this paper, we propose a novel two-stage video-to-video summarization method termed FastPicker, which can efficiently select the most discriminative and representative frames for better action recognition. Independently, the discriminative frames are selected in the first stage based on the inter-frame motion computation, whereas the representative frames are selected in the second stage using a novel Transformer-based model. Learnable frame embeddings are proposed to estimate each frame contribution to the final video classification certainty. Consequently, the frames with the largest contributions are the most representative. The proposed method is carefully evaluated by summarizing several action recognition datasets and using them to train various deep models with several backbones. The experimental results demonstrate a remarkable performance boost on Kinetics400, Something-Something-v2, ActivityNet-1.3, UCF-101, and HMDB51 datasets, e.g., FastPicker downsizes Kinetics400 by 78.7% of its size while improving the human activity recognition.

Comparative Analysis of Keyframe Extraction Techniques for Video Summarization

Background: With the technological advancement, the quality of life of a human were improved. Also with the technological advancement large amount of data were produced by human. The data is in the forms of text, images and videos. Hence there is a need for significant efforts and means of devising methodologies for analyzing and summarizing them to manage with the space constraints. Video summaries can be generated either by keyframes or by skim/shot. The keyframe extraction is done based on deep learning based object detection techniques. Various object detection algorithms have been reviewed for generating and selecting the best possible frames as keyframes. A set of frames were extracted out of the original video sequence and based on the technique used, one or more frames of the set are decided as a keyframe, which then becomes the part of the summarized video. The following paper discusses the selection of various keyframe extraction techniques in detail. Methods : The research paper is focused at summary generation for office surveillance videos. The major focus for the summary generation is based on various keyframe extraction techniques. For the same various training models like Mobilenet, SSD, and YOLO were used. A comparative analysis of the efficiency for the same showed YOLO giving better performance as compared to the others. Keyframe selection techniques like sufficient content change, maximum frame coverage, minimum correlation, curve simplification, and clustering based on human presence in the frame have been implemented. Results: Variable and fixed length video summaries were generated and analyzed for each keyframe selection techniques for office surveillance videos. The analysis shows that he output video obtained after using the Clustering and the Curve Simplification approaches is compressed to half the size of the actual video but requires considerably less storage space. The technique depending on the change of frame content between consecutive frames for keyframe selection produces the best output for office room scenarios. The technique depending on frame content between consecutive frames for keyframe selection produces the best output for office surveillance videos. Conclusion: In this paper, we discussed the process of generating a synopsis of a video to highlight the important portions and discard the trivial and redundant parts. First, we have described various object detection algorithms like YOLO and SSD, used in conjunction with neural networks like MobileNet to obtain the probabilistic score of an object that is present in the video. These algorithms generate the probability of a person being a part of the image, for every frame in the input video. The results of object detection are passed to keyframe extraction algorithms to obtain the summarized video. From our comparative analysis for keyframe selection techniques for office videos will help in determining which keyframe selection technique is preferable.

Video summarization approaches in wireless capsule endoscopy: A review

Wireless capsule endoscopy (WCE) is medical examination process for gastrointestinal tract (GIT). This noninvasive multi advantageous procedure can be made more popular by overcoming the problem of prolonged analysis time. Video summarization is a concise and meaningful representation of a video. Along with automated detection and segmentation methods, summarized video will serve as an additional source for confirming the analysis of WCE video before the final diagnosis without any dropouts. Nowadays, Internet of Things (IoT) environments are predominant in healthcare sectors. Considering limited resources of smart phones and long duration of WCE process, it is impractical to send all the WCE data to health-care centers or gastroenterologists. This paper reviews video summarization types and the techniques used for WCE video summarization by various researchers. Feature set selection, clustering methods and key frame selection techniques play important role in performance of summarization technique.

Selection of keyframes for video colourization using steerable filtering

The appropriate selection of distinctive keyframes to represent the salient contents of a video is a critical task in video processing applications that rely on content analysis or information retrieval. Although many of the existing keyframe selection techniques perform satisfactorily in capturing salient visual contents, they often fail to adequately highlight the changes in visual information brought about by motion of objects between frames. In this paper, we propose a technique for keyframe selection by formulating the dissimilarity between the frames of a video shot in terms of the change in orientations that the corresponding objects of the two frames have undergone due to motion. This is accomplished by steerable filtering of the frames in order to extract the information about the local orientation of pixels within each frame. The frame to frame dissimilarity is adaptively thresholded over a group of frames in order to select the keyframes. In essence, keyframes are selected at the temporal instances where the change in orientation attains local maxima. Our keyframe selection methodology is specifically relevant to video colourization due to the fact that the keyframes that are to be employed for colourization must be chosen such that they capture all orientational changes effectively, while ensuring adequate content coverage.

Dynamic key‐frame selection technique for enhanced odometry estimation based on laser scan similarity comparison

A dynamic key-frame selection technique is proposed for enhanced odometry estimation. The odometry estimation problem has a drift problem in which errors are accumulated over time. The proposed technique finds the key-frame which reduces the odometry drift for every time step based on scan similarity comparisons. The enhancement of the proposed method is verified through the performance comparison with the frame-to-frame and the conventional frame to key-frame methods.

Spatial and temporal scoring for egocentric video summarization

We present a summarization approach for egocentric video. Given hours of video, the proposed method produces a compact storyboard summary of the camera wearer's day. In contrast to traditional keyframe selection techniques, the resulting summary focuses on the most important video shots which reflect high stable salience, discrimination and representativeness. To accomplish this, we utilize egocentric salience cues, motion cues and a selection model to capture stable salience weight, discriminative weight and representative weight of a video shot respectively. We further combine these weights in a unified framework to predict the importance score of a shot, based on which, important shots are selected for the storyboard. Critically, the approach is neither camera-wearer-specific nor object-specific; that means the learned importance metric need not be trained for a given user or context, and it can predict the importance of shots that have never been seen previously. Experimental results on three video datasets across various genres demonstrate that our proposed approach clearly outperforms several state-of-the-art methods.

Video Partitioning and Secured Keyframe Extraction of MPEG Video

Video partitioning and keyframe extraction (KFE) are the key foundations of video analysis and Content based video retrieval. The use of keyframes reduces the amount of data that is necessary in video indexing and provides the outline for dealing with the video content. In the last few years, many algorithms of keyframe extraction concentrated on the original compressed video stream. This can increase computational complexity when decompression is required before video processing. Keyframe is the frame, which can be a prototype of the salient content and information of the shot. The keyframes extracted must summarize the significant features of the video in time sequence. Therefore, there is a commensurate need of an efficient and secured keyframe selection technique in an efficient CBVR system. We propose an algorithm for keyframe extraction of compressed video shots using adaptive threshold method. Extensive computation on 200 plus video clips is performed and results are accurate and satisfactory.

Investigating the impact of frame rate towards robust human action recognition

Human action recognition from videos is very important for visual analytics. Due to increasing abundance of diverse video content in the era of big data, research on human action recognition has recently shifted towards more challenging and realistic settings. Frame rate is one of key issues in diverse and realistic video settings. While there have been several evaluation studies investigating different aspects of action recognition such as different visual descriptors, the frame rate issue has been seldom addressed in the literature. Therefore, in this paper, we investigate the impact of frame rate on human action recognition with several state-of-the-art approaches and three benchmark datasets. Our experimental results indicate that those state-of-the-art approaches are not robust to the variations of frame rate. As a result, more robust visual features and advanced learning algorithms are required to further improve human action recognition performance towards its more practical deployments. In addition, we investigate key-frame selection techniques for choosing a set of suitable frames from an action sequence for action recognition. Promising results indicate that well designed key-frame selection methods can produce a set of representative frames and eventually reduce the impact of frame rate on the performance of human action recognition.

Efficient Key Frame Selection Approach for Object Detection in Wide Area Surveillance Applications

This paper presents an efficient preprocessing algorithm for object detection in wide area surveillance video analysis. The proposed key-frame selection method utilizes the pixel intensity differences among subsequent frames to automatically select only the frames that contain the desired contextual information and discard the rest of the insignificant frames. For improving effectiveness and efficiency, a batch updating based on a modular representation strategy is also incorporated. Experimental results show that the proposed key frame selection technique has a significant positive performance impact on wide area surveillance applications such as automatic object detection and recognition in aerial imagery.

Predicting Important Objects for Egocentric Video Summarization

We present a video summarization approach for egocentric or "wearable" camera data. Given hours of video, the proposed method produces a compact storyboard summary of the camera wearer's day. In contrast to traditional keyframe selection techniques, the resulting summary focuses on the most important objects and people with which the camera wearer interacts. To accomplish this, we develop region cues indicative of high-level saliency in egocentric video---such as the nearness to hands, gaze, and frequency of occurrence---and learn a regressor to predict the relative importance of any new region based on these cues. Using these predictions and a simple form of temporal event detection, our method selects frames for the storyboard that reflect the key object-driven happenings. We adjust the compactness of the final summary given either an importance selection criterion or a length budget; for the latter, we design an efficient dynamic programming solution that accounts for importance, visual uniqueness, and temporal displacement. Critically, the approach is neither camera-wearer-specific nor object-specific; that means the learned importance metric need not be trained for a given user or context, and it can predict the importance of objects and people that have never been seen previously. Our results on two egocentric video datasets show the method's promise relative to existing techniques for saliency and summarization.