Key Frame Selection Research Articles

Fast key-frame image retrieval of intelligent city security video based on deep feature coding in high concurrent network environment

Aiming at the problems faced by the connection management module in single packet processing in intelligent city security video retrieval, we firstly propose the traffic locality quantization index based on the traffic characteristics of the backbone link to quantitatively analyze the traffic locality characteristics in the backbone link. And then, a key-frame abstraction and retrieval of videos based on deep learning is proposed to improve the efficiency and accuracy of video retrieval, where an adaptive key-frame selection algorithm is designed and the existing convolutional neural network framework is used to extract the features of key-frames, and unsupervised, semi-supervised and supervised retraining models are designed to improve the effectiveness of the feature extraction of the convolutional neural network and the accuracy of the video retrieval. Experimental results based on the public video datasets show that our proposed key-frame image retrieval model realizes a good precision for key-frame representation, and achieves high accuracy and efficiency for video retrieval.

Unsupervised key frame selection using information theory and colour histogram difference

Key frame selection is one of the important research issues in video content analysis, as it helps effective video browsing and retrieval as well as efficient storage. Key frames would typically be...

Video summarization approaches in wireless capsule endoscopy: A review

Wireless capsule endoscopy (WCE) is medical examination process for gastrointestinal tract (GIT). This noninvasive multi advantageous procedure can be made more popular by overcoming the problem of prolonged analysis time. Video summarization is a concise and meaningful representation of a video. Along with automated detection and segmentation methods, summarized video will serve as an additional source for confirming the analysis of WCE video before the final diagnosis without any dropouts. Nowadays, Internet of Things (IoT) environments are predominant in healthcare sectors. Considering limited resources of smart phones and long duration of WCE process, it is impractical to send all the WCE data to health-care centers or gastroenterologists. This paper reviews video summarization types and the techniques used for WCE video summarization by various researchers. Feature set selection, clustering methods and key frame selection techniques play important role in performance of summarization technique.

Subject Review: Video Compression Algorithms

Video compression is the process of decreasing the number of bits required to represent a certain video. Video compression can be done by a specific algorithm for deciding the most ideal approach to reduce the amount of data storage. The video file is coded in such a way that consuming less space than the original file and is easier to transmit over the Internet. The basic idea of video compression based on removing the redundant data that exists in the video. There are four types of redundancy in digital video: colorize redundancy, temporal redundancy, statistical redundancy, and spatial redundancy. Video compression algorithms must reduce these redundancies in such a way that keep the quality of the compressed video when the decompression process is done. Most video compression techniques consist of the following steps: Motion Estimation,Motion Compensation, Discrete Cosine Transform, Run Length Encoding, Huffman Coding. Frame Difference. This paper discusses the characteristic of video coding from the scratch of key frame selection to the evolutions of various standards

Vulnerability Attacks of SVD-Based Video Watermarking Scheme in an IoT Environment

This study reviews the superiority and limitations of the Singular Value Decomposition (SVD)-based video watermarking scheme. First applies key frame selection to the video sequence to obtain an appropriate video frame for the purpose of watermark embedding in this paper. The Fibonacci-Lucas image transformation scrambles the grayscale watermark image before the embedding process. The former scheme satisfies the imperceptibility and robustness aspects. However, it can suffer from the False-Positive-Problem (FPP) in which a counterfeit watermark image can be easily reconstructed by a malicious attacker. This study presents a new technique for overcoming the FPP of the aforementioned scheme. The proposed method embeds the principal components of the watermark image rather than the Singular Value Matrix (SVM) into the host video. As results, the proposed methods yield promising results in terms of imperceptibility as well as robustness. At the same time, it solves the FPP. A theoretical analysis of embedded watermark information is also provided in this work. The theoretical analysis meets the experimental-based results in the SVD-based video watermarking scheme.

DeepReS: A Deep Learning-Based Video Summarization Strategy for Resource-Constrained Industrial Surveillance Scenarios

The exponential growth in the production of video contents in different industries causes an urgent need for effective video summarization (VS) techniques, in order to get an optimal storage and preservation of key information in the video. Compared to other domains, industrial videos are more challenging to process, as they usually contain diverse and complex events, which make their online processing a difficult task. In this article, we introduce an online system for intelligent video capturing, coarse and fine redundancy removal, and summary generation. First, we capture video data through resource-constrained devices in an industrial Internet of Things network, equipped with vision sensors and apply coarse redundancy removal through the comparison of low-level features. Second, we transmit the resulting frames to the cloud for detailed analysis, where sequential features are extracted for the selection of candidate keyframes. Finally, we refine the candidate keyframes in order to discriminate those with maximum information as part of the summary. The key contributions of this article include the coarse and fine refining of video data implemented over resource-restricted devices and the presentation of important data in the form of a summary. Experiments 1 1 [Online]. Available: https://github.com/tanveer-hussain/DeepRes-Video-Summarization . over publicly available datasets evince a 0.3-unit increase in the F1 score when compared to state-of-the-art and with reduced time complexity. Furthermore, we provide convincing results on our newly created dataset in an industrial environment, which is made publicly available for the research community along with its labeled ground truth.

RETRACTED ARTICLE: Fall detection based on posture classification for smart home environment

Automatic human fall detection is one of the major components in the elderly monitoring system. Detection of human fall in a smart home environment can be utilized as a means to improve the quality of care offered to elderly people thus reducing the risk factor when they are alone. Recently various fall detection approaches have been proposed, among which computer vision based approaches offer promising and effective solutions. In this paper, an analysis of fall detection is carrier out based on the automatic, feature learning in a hybrid approach. Initially, a model is generated using the training dataset that contains samples of both fall and normal active events. Then key frames are extracted from the video sequence that is subjected to two stream classification. The classification results are approved if both the streams project the same results, failing so, additional information are used to classify the fall from the normal activity. The selection of key frames depends on the displacement in the centroid of the detected object have threshold greater than the predefined value. Experiments show that the proposed approach achieves reliable results compared with other methods, and a better result is achieved in our method even when training with fewer training samples.

Deep Neural Network–Based Loop Detection for Visual Simultaneous Localization and Mapping Featuring Both Points and Lines

Herein, a visual simultaneous localization and mapping (SLAM) is proposed in which both points and lines are extracted as features and a deep neural network is adopted for loop detection. Its working principles, including the representation, extraction, description, and matching of lines, initialization, keyframe selection, optimization of tracking and mapping, and loop detection using a deep neural network, are set forth in detail. The overall trajectory estimation and loop detection performance is investigated using the TUM RGB‐D (indoor) benchmark and KITTI (outdoor) datasets. Compared with the conventional SLAMs, the experimental results of this study indicate that the proposed SLAM is able to improve the accuracy and robustness of trajectory estimation, especially for the scenes with insufficient points. As for loop detection, the deep neural network turns out to be superior to the traditional bag‐of‐words model, because it decreases the accumulated errors of both the estimated trajectory and reconstructed scenes.

Scattering Key-Frame Extraction for Comprehensive VideoSAR Summarization: A Spatiotemporal Background Subtraction Perspective

Video synthetic aperture radar (VideoSAR) presents significant potential to enhance the performance of automatic information retrieval and interpretation in the dynamic region of interest (DROI). Key-frame extraction represents effective contents in processing massive video data. In this article, a novel computer vision-based background subtraction perspective is proposed for automatic VideoSAR scattering key-frame selection. A universal parameterization scattering key-frame model is firstly investigated, which serves to reveal the VideoSAR scattering key frames. Then, a spatiotemporal scattering extractor, called the subaperture energy gradient (SEG), together with a modified statistical and knowledge-based object tracker (MSAKBOT) is proposed to robustly discriminate the scattering key-frame state of the alternation between transient persistence and disappearance. The proposed SEG-MSAKBOT method presents a more comprehensive measurement of scattering key frames that is adaptive to the multitemporal video sequences effectively. Finally, experimental results and performance assessment conducted on two real airborne VideoSAR data with the coherent integration angles of 21° and 27° demonstrate that the proposed perspective is robust and reliable to capture scattering key frames and video content summarization with highly accurate descriptions in various DROIs.

Learning an intrinsic garment space for interactive authoring of garment animation

Authoring dynamic garment shapes for character animation on body motion is one of the fundamental steps in the CG industry. Established workflows are either time and labor consuming (i.e., manual editing on dense frames with controllers), or lack keyframe-level control (i.e., physically-based simulation). Not surprisingly, garment authoring remains a bottleneck in many production pipelines. Instead, we present a deep-learning-based approach for semi-automatic authoring of garment animation, wherein the user provides the desired garment shape in a selection of keyframes, while our system infers a latent representation for its motion-independent intrinsic parameters (e.g., gravity, cloth materials, etc.). Given new character motions, the latent representation allows to automatically generate a plausible garment animation at interactive rates. Having factored out character motion, the learned intrinsic garment space enables smooth transition between keyframes on a new motion sequence. Technically, we learn an intrinsic garment space with an motion-driven autoencoder network, where the encoder maps the garment shapes to the intrinsic space under the condition of body motions, while the decoder acts as a differentiable simulator to generate garment shapes according to changes in character body motion and intrinsic parameters. We evaluate our approach qualitatively and quantitatively on common garment types. Experiments demonstrate our system can significantly improve current garment authoring workflows via an interactive user interface. Compared with the standard CG pipeline, our system significantly reduces the ratio of required keyframes from 20% to 1 -- 2%.

Real-Time Visual–Inertial SLAM Based on Adaptive Keyframe Selection for Mobile AR Applications

Simultaneous localization and mapping (SLAM) technology is used in many applications, such as augmented reality (AR)/virtual reality, robots, drones, and self-driving vehicles. In AR applications, rapid camera motion estimation, actual size, and scale are important issues. In this research, we introduce a real-time visual–inertial SLAM based on an adaptive keyframe selection for mobile AR applications. Specifically, the SLAM system is designed based on the adaptive keyframe selection visual–inertial odometry method that includes the adaptive keyframe selection method and the lightweight visual–inertial odometry method. The inertial measurement unit data are used to predict the motion state of the current frame and it is judged whether or not the current frame is a keyframe by an adaptive selection method based on learning and automatic setting. Relatively unimportant frames (not a keyframe) are processed using a lightweight visual–inertial odometry method for efficiency and real-time performance. We simulate it in a PC environment and compare it with state-of-the-art methods. The experimental results demonstrate that the mean translation root-mean-square error of the keyframe trajectory is 0.067 m without the ground-truth scale matching, and the scale error is 0.58% with the EuRoC dataset. Moreover, the experimental results of the mobile device show that the performance is improved by 34.5%–53.8% using the proposed method.

Video summarization via block sparse dictionary selection

The explosive growth of video data has raised new challenges for many video processing tasks such as video browsing and retrieval, hence, effective and efficient video summarization (VS) is urgently demanded to automatically summarize a video into a succinct version. Recent years have witnessed the advancements of sparse representation based approaches for VS. However, video frames are analyzed individually for keyframe selection in existing methods, which could lead to redundancy among selected keyframes and poor robustness to outlier frames. Due to that adjacent frames are visually similar, candidate keyframes often occur in temporal blocks, in addition to sparse presence. Therefore, in this paper, the block-sparsity of candidate keyframes is taken into consideration, by which the VS problem is formulated as a block sparse dictionary selection model. Moreover, a simultaneous block version of Orthogonal Matching Pursuit (SBOMP) algorithm is designed for model optimization. Two keyframe selection strategies are also explored for each block. Experimental results on two benchmark datasets, namely VSumm and TVSum datasets, demonstrate that the proposed SBOMP based VS method clearly outperforms several state-of-the-art sparse representation based methods in terms of F-score, redundancy among keyframes and robustness to outlier frames.

Quality-guided key frames selection from video stream based on object detection

Object detection technique is widely applied in modern intelligent systems, such as pedestrian tracking, video surveillance. Key frames selection aims to select more informative frames and reduce amount of redundant information frames. Traditional methods leveraged SIFT feature, which have high key frame selection error rate. In this paper, we propose a novel key frames selection method based on object detection and image quality. Specifically, we first leverage object detector to detect object, such as pedestrian, vehicles. Then, each training frame will be assigned with a quality score, where frames contain objects have high quality score. Afterwards, we leverage CNN based AlexNet architecture for deep feature representation extraction. Our algorithm combines mutual information entropy and SURF image local features to extract key frames. Comprehensive experiments verify the feasibility of practicing the key frame extractor based on convolutional neural network by training the model, and conduct a quality assessment model study.

Static and Dynamic Video Summarization

Video represented by a large number of frames synchronized with audio making video saving requires more storage, it's delivery slower, and computation cost expensive. Video summarization provides entire video information in minimum amount of time. This paper proposes static and dynamic video summarizationmethods. The proposed static video summarization method includes several steps which are extracting frames from video, keyframes selection, feature extraction and description, and matching feature descriptor with bag of visual words, and finally save frames when features matched. The proposed dynamic video summarizationmethod includes in general extracting audio from video, calculating audio features using the average of samples in windows and find the highest average which reflects portion of video with loudest sound. The experimental results for the proposed static video summarization show that there is no redundancy between selected representative keyframes and the subjective evaluation results ensure the importance of the selected keyframes. While the experimental results for the proposed static video summarization show that all the segments of goals have been extracted to provide video summary. Static and dynamic video summarization methods done to football or soccer video type.

Improved Quality Keyframe Selection Method for HD Video

With the widespread use of the Internet, services for providing large-capacity multimedia data such as video-on-demand (VOD) services and video uploading sites have greatly increased. VOD service providers want to be able to provide users with high-quality keyframes of high quality videos within a few minutes after the broadcast ends. However, existing keyframe extraction tends to select keyframes whose quality as a keyframe is insufficiently considered, and it takes a long computation time because it does not consider an HD class image. In this paper, we propose a keyframe selection method that flexibly applies multiple keyframe quality metrics and improves the computation time. The main procedure is as follows. After shot boundary detection is performed, the first frames are extracted as initial keyframes. The user sets evaluation metrics and priorities by considering the genre and attributes of the video. According to the evaluation metrics and the priority, the low-quality keyframe is selected as a replacement target. The replacement target keyframe is replaced with a high-quality frame in the shot. The proposed method was subjectively evaluated by 23 votes. Approximately 45% of the replaced keyframes were improved and about 18% of the replaced keyframes were adversely affected. Also, it took about 10 minutes to complete the summary of one hour video, which resulted in a reduction of more than 44.5% of the execution time.

PL-SLAM: A Stereo SLAM System Through the Combination of Points and Line Segments

Traditional approaches to stereo visual simultaneous localization and mapping (SLAM) rely on point features to estimate the camera trajectory and build a map of the environment. In low-textured environments, though, it is often difficult to find a sufficient number of reliable point features and, as a consequence, the performance of such algorithms degrades. This paper proposes PL-SLAM, a stereo visual SLAM system that combines both points and line segments to work robustly in a wider variety of scenarios, particularly in those where point features are scarce or not well-distributed in the image. PL-SLAM leverages both points and line segments at all the instances of the process: visual odometry, keyframe selection, bundle adjustment, etc. We contribute also with a loop-closure procedure through a novel bag-of-words approach that exploits the combined descriptive power of the two kinds of features. Additionally, the resulting map is richer and more diverse in three-dimensional elements, which can be exploited to infer valuable, high-level scene structures, such as planes, empty spaces, ground plane, etc. (not addressed in this paper). Our proposal has been tested with several popular datasets (such as EuRoC or KITTI), and is compared with state-of-the-art methods such as ORB-SLAM2, revealing a more robust performance in most of the experiments while still running in real time. An open-source version of the PL-SLAM C++ code has been released for the benefit of the community.

Dense-ArthroSLAM: Dense Intra-Articular 3-D Reconstruction With Robust Localization Prior for Arthroscopy

Arthroscopy is a minimally invasive surgery that imposes great physical and mental challenges to surgeons. Extensive experience is required to safely navigate camera and instruments in narrow spaces of human joints. Robust camera localization as well as a detailed reconstruction of the anatomy can benefit surgeons and would be essential for future robotic assistants. Our existing simultaneous localization and mapping (SLAM) system provides a robust, at-scale camera localization and a sparse map. However, a denser map is required to be of clinical relevance. In this latter, we propose a new system that combines the robust localizer with a keyframe selection strategy and a batch multiview stereo (MVS) for three-dimensional reconstruction. Tissues are reconstructed at scale, accurately and densely even under challenging arthroscopic conditions. The consistency of our system is verified in tests with synthetic noise and several keyframing strategies. Nine experiments were performed in phantom and three cadavers including various imaging conditions, camera settings, and scope motions. Our system reconstructed surfaces of more than $12\,\text{cm}^2$ with a root mean square error of no more than 0.5 $\text{mm}$ . In comparison, monocular state-of-the-art SLAM feature-based (ORBSLAM) and direct (LSDSLAM) methods commonly failed to track more than 20% of any camera motion and, in the few successful cases, yielded much larger estimation errors.

Hybrid feature-based analysis of video’s affective content using protagonist detection

Extracting an effective representation to analyze affection in videos is an inherent challenge. To address this problem, we present a novel emotion recognition system that can intelligently analyze and automatically recognize video emotions. We observe that most of the emotions in a video are closely related to the roles, especially those of the protagonist, which motivates us to explore how to utilize the protagonist’s information to help recognize video emotions. By analyzing the traits of the protagonists, we suggest a new solution to detect the protagonists to adapt not only to the whole video but also to a video clip. Moreover, a new keyframe selection strategy based on the protagonist is designed to select a set of representative frames from video clips. Furthermore, the scale invariant feature transform (SIFT) features matrix, built from each keyframe, is fed into a convolutional neural network (CNN) to learn the discriminative representations, which makes the CNN and local features complement each other. Considering that emotions are usually continuous, we introduce temporal information into the CNN by using optical flow images. Additionally, we extract some handcrafted visual and audio features as a supplement. Finally, all the features, including the features learned from the CNN and the handcrafted features, are fused and input into a support vector machine (SVM) and a support vector regression (SVR) for video emotion recognition. The proposed system is validated on a public dataset (LIRIS-ACCEDE) and a new dataset (PMSZU). The experimental results demonstrate the promising performance of the proposed system, which achieves better performance than the compared methods. Our designed system for recognizing video emotions could also facilitate the development of similar video expert systems, for applications such as video recommendations, video classification and video retrieval.

Understanding the Dynamics of Social Interactions

In this article, we deal with the problem of understanding human-to-human interactions as a fundamental component of social events analysis. Inspired by the recent success of multi-modal visual data in many recognition tasks, we propose a novel approach to model dyadic interaction by means of features extracted from synchronized 3D skeleton coordinates, depth, and Red Green Blue (RGB) sequences. From skeleton data, we extract new view-invariant proxemic features, named Unified Proxemic Descriptor (UProD), which is able to incorporate intrinsic and extrinsic distances between two interacting subjects. A novel key frame selection method is introduced to identify salient instants of the interaction sequence based on the joints’ energy. From Red Green Blue Depth (RGBD) videos, more holistic CNN features are extracted by applying an adaptive pre-trained Convolutional Neural Networks (CNNs) on optical flow frames. For better understanding the dynamics of interactions, we expand the boundaries of dyadic interactions analysis by proposing a fundamentally new modeling for non-treated problem aiming to discern the active from the passive interactor. Extensive experiments have been carried out on four multi-modal and multi-view interactions datasets. The experimental results demonstrate the superiority of our proposed techniques against the state-of-the-art approaches.

A Novel Key-Frames Selection Framework for Comprehensive Video Summarization

Video summarization (VSUMM) has become a popular method in processing massive video data. The key point of VSUMM is to select the key frames to represent the effective contents of a video sequence. The existing methods can only extract the static images of videos as the content summarization, but they ignore the representation of motion information. To cope with these issues, a novel framework for an efficient video content summarization as well as video motion summarization is proposed. Initially, Capsules Net is trained as a spatiotemporal information extractor, and an inter-frames motion curve is generated based on those spatiotemporal features. Subsequently, a transition effects detection method is proposed to automatically segment the video streams into shots. Finally, a self-attention model is introduced to select key-frames sequences inside the shots; thus, key static images are selected as video content summarization, and optical flows can be calculated as video motion summarization. The ultimate experimental results demonstrate that our method is competitive on VSUMM, TvSum, SumMe, and RAI datasets about shot segmentation and video content summarization, and can also represent a good motion summarization result.