Key Frame Extraction Algorithm Research Articles

Video summary generation by visual shielding compressed sensing coding and double-layer affinity propagation

Video summary technology based on keyframe extraction is an effective means to rapidly access video content. Traditional video summary generation technology requires high video resolution, which poses a problem as most existing studies have no targeted solutions for videos that are subject to privacy protection. We propose a novel keyframe extraction algorithm for video data in the visual shielding domain, named visual shielding compressed sensing coding and double-layer affinity propagation (VSCS-DAP). VSCS-DAP involves three main steps. First, the video is compressed by compressed sensing technology to provide a visual shielding effect (protecting the privacy of monitored figures), while the data volume is significantly reduced. Then, pyramid histogram of oriented gradients (PHOG) features are extracted from the compressed video to be clustered by the first step affinity propagation (AP) to gain the summaries of the first stage. Finally, the PHOG and Hist fusion features are extracted from the keyframes of the first stage, and they cluster the fused PHOG-Hist features by the second step AP algorithm to obtain the final output summaries. Experimental results obtained on two common video datasets show that our method exhibits advantages including low redundancy and few missing frames, low computational complexity, strong real-time performance, and robustness to vision-shielded video.

Key frame extraction for Human Action Videos in dynamic spatio-temporal slice clustering

Detecting representative frames in videos based on human actions is quite challenging because of the combined factors of human pose in action and the background. This paper proposed a key frame extraction algorithm based on dynamic spatio-temporal slice clustering. This algorithm firstly uses the dynamic spatio-temporal slice position selection method based on the human mask heat map to calculate the position of slice to realize the dynamic selection of slice positions, then complete the extraction of spatio-temporal slice images. After clustering the spatio-temporal slice images, this method extracts key frames according to the clustering results. The experimental results prove the validity of spatio-temporal slice location selection method, the proposed algorithm can effectively solve the problems of information redundancy and key information missing in existing methods. We conduct experiments on a challenging human action dataset UCF101 and show that our method can detect key frames with high accuracy.

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network.

Sign language is the most important way of communication for hearing-impaired people. Research on sign language recognition can help normal people understand sign language. We reviewed the classic methods of sign language recognition, and the recognition accuracy is not high enough because of redundant information, human finger occlusion, motion blurring, the diversified signing styles of different people, and so on. To overcome these shortcomings, we propose a multi-scale and dual sign language recognition Network (SLR-Net) based on a graph convolutional network (GCN). The original input data was RGB videos. We first extracted the skeleton data from them and then used the skeleton data for sign language recognition. SLR-Net is mainly composed of three sub-modules: multi-scale attention network (MSA), multi-scale spatiotemporal attention network (MSSTA) and attention enhanced temporal convolution network (ATCN). MSA allows the GCN to learn the dependencies between long-distance vertices; MSSTA can directly learn the spatiotemporal features; ATCN allows the GCN network to better learn the long temporal dependencies. The three different attention mechanisms, multi-scale attention mechanism, spatiotemporal attention mechanism, and temporal attention mechanism, are proposed to further improve the robustness and accuracy. Besides, a keyframe extraction algorithm is proposed, which can greatly improve efficiency by sacrificing a little accuracy. Experimental results showed that our method can reach 98.08% accuracy rate in the CSL-500 dataset with a 500-word vocabulary. Even on the challenging dataset DEVISIGN-L with a 2000-word vocabulary, it also reached a 64.57% accuracy rate, outperforming other state-of-the-art sign language recognition methods.

Fusion of gradient and feature similarity for Keyframe extraction

Several computer vision applications such as e-learning, video editing, video compression, video-on-demand and surveillance etc. are popular in recent days. Most of the applications need videos to be retrieved and processed regularly. First and foremost step towards video retrieval and management is keyframe extraction. The perfect identification of shot transition boundaries is trivial in extracting keyframes. In present article, a framework for shot transition detection and keyframe extraction have been proposed. The proposed method is efficient, simple and does not require supervision which makes it attractive. The proposed method establishes the shot transition boundaries by estimating feature similarity (FSIM) between gradient magnitudes of consecutive frames. Then the frame with the highest mean and standard deviation is chosen as keyframe to that shot. In any situation if one feature fails to establish shot transition boundary another feature may succeed in establishment of shot transition boundary at proper frame locations of video. The proposed algorithm is tested on four different datasets, among them one is developed by us, two are well known standard datasets to evaluate keyframe extraction algorithm and the other one is standard surveillance video dataset. All the datasets are publicly available. Performance evaluation of the method is done in terms of Figure of merit, Detection percentage, Accuracy and Missing factor. The experimental results prove that the proposed method outperforms other state-of-art methods.

Inaccessible Island vs Urban Education in Greece. Knowledge-Based Summarization of Educational Videos by Analyzing Students’ Watching Patterns

During the last decade, the problem of inequality in education has been studied by several researchers, especially regarding urban and rural (inaccessible) areas. Most studies conclude that funding and adoption of new technologies are necessary to reduce inequalities. However they do not propose any specific means to overcome these inequalities. Towards this direction, in this paper a specific method is proposed that contributes to the reduction of inequalities, without demanding any funding. The method is designed especially for inaccessible areas (including isolated islands), which face network connectivity issues or Internet speed becomes very slow. In particular knowledge-based video summarization is introduced, to overcome boredom, frustration, anger etc. when students try to watch online educational videos, using limited throughput channels. The main question is: which information is less important so that it can be excluded or roughly described ? In this paper students play the role of human sensors, by letting recording and analyzing their watching behaviors. Then average watching patterns are extracted and used for video summarization, which is accomplished by a keyframes extraction algorithm that detects uncorrelated content. Finally, results are presented, to delimit the potential of such applications as well as to set the bases for future work.

User-generated video emotion recognition based on key frames

Video is an important medium in communication and entertainment, and thus, an intelligent understanding of videos has attracted widespread interest in academic community. Video content diversity and sparse emotional expression are challenging for video emotion recognition, especially for user-generated video. In this paper, we propose a key frames extraction algorithm based on affective saliency estimation. By estimating the affective saliency value of video frames, key frames are extracted to avoid the influence of emotion-independent frames on the recognition result. Efficient deep visual features are extracted using pretrained models and traditional models Support Vector Machine (SVM), Random Forests (RF) and deep model Convolutional Neural Networks (CNN) are used to perform emotion recognition. Moreover, we propose a hybrid fusion mechanism that combines score fusion and Top-K decision fusion to further improve recognition accuracy. Extensive experiments are conducted on user-generated video datasets Ekman-6 and VideoEmotion-8, and the average recognition accuracy are 59.51% and 52.85% respectively. The experimental results show that the proposed method can improve the recognition performance and is superior to the current user-generated video emotion recognition methods.

The key frame extraction algorithm based on the indigenous disturbance variation difference video

Abstract In view of the traditional support vector machine (SVM) learning algorithm for widespread learning parameters is not easy determined in the process of video key frame extraction, the problem of low accuracy, an independent perturbation variable difference SVM algorithm used for video key frame extraction. First of all, the study of the biological mechanism of differential evolution algorithm, put forward an improved way of autonomic disturbance variation. Secondly, combined with improved forms of independent disturbance differential evolution algorithm for SVM parameters optimization, designed the key frame extraction algorithm of the video based on the improved differential of SVM algorithm. Through the standard test functions and video test database experiments show that the improvement of autonomic disturbance variation difference video key frame extraction algorithm can more effectively optimize parameters of support vector machine, so as to contribute to the improvement of the recall of video retrieval rate of (quasi) two algorithms performance evaluation standard.

Intelligent Sports Video Classification Based on Deep Neural Network (DNN) Algorithm and Transfer Learning.

Traditional text annotation-based video retrieval is done by manually labeling videos with text, which is inefficient and highly subjective and generally cannot accurately describe the meaning of videos. Traditional content-based video retrieval uses convolutional neural networks to extract the underlying feature information of images to build indexes and achieves similarity retrieval of video feature vectors according to certain similarity measure algorithms. In this paper, by studying the characteristics of sports videos, we propose the histogram difference method based on using transfer learning and the four-step method based on block matching for mutation detection and fading detection of video shots, respectively. By adaptive thresholding, regions with large frame difference changes are marked as candidate regions for shots, and then the shot boundaries are determined by mutation detection algorithm. Combined with the characteristics of sports video, this paper proposes a key frame extraction method based on clustering and optical flow analysis, and experimental comparison with the traditional clustering method. In addition, this paper proposes a key frame extraction algorithm based on clustering and optical flow analysis for key frame extraction of sports video. The algorithm effectively removes the redundant frames, and the extracted key frames are more representative. Through extensive experiments, the keyword fuzzy finding algorithm based on improved deep neural network and ontology semantic expansion proposed in this paper shows a more desirable retrieval performance, and it is feasible to use this method for video underlying feature extraction, annotation, and keyword finding, and one of the outstanding features of the algorithm is that it can quickly and effectively retrieve the desired video in a large number of Internet video resources, reducing the false detection rate and leakage rate while improving the fidelity, which basically meets people's daily needs.

KeyFrame extraction for human motion capture data via multiple binomial fitting

AbstractIn this paper, we make two contributions. The first is to propose a new keyframe extraction algorithm, which reduces the keyframe redundancy and reduces the motion sequence reconstruction error. Secondly, a new motion sequence reconstruction method is proposed, which further reduces the error of motion sequence reconstruction. Specifically, we treated the input motion sequence as curves, then the binomial fitting was extended to obtain the points where the slope changes dramatically in the vicinity. Then we took these points as inputs to obtain keyframes by density clustering. Finally, the motion curves were segmented by keyframes and the segmented curves were fitted by binomial formula again to obtain the binomial parameters for motion reconstruction. Experiments show that our methods outperform existing techniques, in terms of reconstruction error.

Video Summarization Based on Mutual Information and Entropy Sliding Window Method.

This paper proposes a video summarization algorithm called the Mutual Information and Entropy based adaptive Sliding Window (MIESW) method, which is specifically for the static summary of gesture videos. Considering that gesture videos usually have uncertain transition postures and unclear movement boundaries or inexplicable frames, we propose a three-step method where the first step involves browsing a video, the second step applies the MIESW method to select candidate key frames, and the third step removes most redundant key frames. In detail, the first step is to convert the video into a sequence of frames and adjust the size of the frames. In the second step, a key frame extraction algorithm named MIESW is executed. The inter-frame mutual information value is used as a metric to adaptively adjust the size of the sliding window to group similar content of the video. Then, based on the entropy value of the frame and the average mutual information value of the frame group, the threshold method is applied to optimize the grouping, and the key frames are extracted. In the third step, speeded up robust features (SURF) analysis is performed to eliminate redundant frames in these candidate key frames. The calculation of Precision, Recall, and F are optimized from the perspective of practicality and feasibility. Experiments demonstrate that key frames extracted using our method provide high-quality video summaries and basically cover the main content of the gesture video.

Key Frames Extraction Based on Optical-flow and Mutual Information Entropy

Key frames provide a suitable video summary and framework for video indexing, browsing and retrieval, and how to extract key frames is the core in the field of video retrieval. In order to solve the problem of redundancy and missing selection in key frame extraction, this paper proposed a key frame extraction algorithm based on optical flow and mutual information entropy. This algorithm integrates mutual information entropy and optical flow characteristics to extract key frames. First, the optical flow method calculated the total optical flow of each frame of the image, and selected the video frame with extreme optical flow difference in the neighbourhood as the key frame, and the other video frames were put into the candidate key frame set; by calculating the mutual information entropy of the key frame set, the minimum mutual information entropy is taken as the threshold; Then, The mutual information entropy of the candidate key frame set was calculated, and the image frames larger than the threshold value were put into the key frame set; Finally, redundant key frames are deleted through the measurement of inter-frame similarity, and the remaining key frames are the key frames to be extracted. The experimental results show that compared with other methods, the accuracy of extracted key frames by this approach is significantly improved on precision, recall and F-score, and the extracted key frames can reflect video content more effectively.

Large-Scale Video Retrieval via Deep Local Convolutional Features

In this paper, we study the challenge of image-to-video retrieval, which uses the query image to search relevant frames from a large collection of videos. A novel framework based on convolutional neural networks (CNNs) is proposed to perform large-scale video retrieval with low storage cost and high search efficiency. Our framework consists of the key-frame extraction algorithm and the feature aggregation strategy. Specifically, the key-frame extraction algorithm takes advantage of the clustering idea so that redundant information is removed in video data and storage cost is greatly reduced. The feature aggregation strategy adopts average pooling to encode deep local convolutional features followed by coarse-to-fine retrieval, which allows rapid retrieval in the large-scale video database. The results from extensive experiments on two publicly available datasets demonstrate that the proposed method achieves superior efficiency as well as accuracy over other state-of-the-art visual search methods.

A novel key-frame selection-based sign language recognition framework for the video data

ABSTRACT Sign language is a medium of communication for people with hearing disabilities. Static and dynamic gestures are identified in a video-based sign language recognition and translated them into humanly understandable phrases to achieve the communication objective. However, videos contain redundant Key-frames which require additional processing. Number of such Key-frames can be reduced. The selection of particular Key-frames without losing the required information is a challenging task. The Key-frame extraction algorithm is used which helps to speed-up the sign language recognition process by extracting essential key-frames. The proposed framework eliminates the computation overhead by picking up the distinct Key-frames for the recognition process. Discrete Wavelet Transform (DWT), Discrete Cosine Transform (DCT), and Histograms of Oriented Gradient (HOG) are used for unique features extraction. We used the bagged tree, boosted tree ensemble method, Fine KNN, and SVM for classification. We tested methodology on video-based datasets of Pakistani Sign Language. It achieved an overall 97.5% accuracy on 37 Urdu alphabets and 95.6% accuracy on 100 common words.

An Action Recognition Method Based on Deformable Convolution Network

In order to solve the problem of behavior recognition in short video, this paper first proposes a key frame extraction algorithm based on mutual information entropy, which uses sliding window to preserve the timing information between frames. Based on the key frame extraction, a based on Deform-GoogLeNet, a two-stream CNN method for deformable convolutional networks, uses the two-stream network to extract the RGB features and optical Flow characteristics of the image separately, and uses the weighted average method to obtain the results of behavior recognition. On the public dataset Charades dataset, mAP is 22.9, which is higher than the similar fusion algorithm, which proves that the proposed algorithm has a good effect in short video behavior recognition.

Key Frame Extraction Algorithm of Motion Video Based on Priori

Key frame extraction technology is one of the core technologies of content-based video retrieval. For video types with complex content, various scenes, and rich actions, the performance of existing key frame extraction methods is not ideal. Based on the Visual Geometry Group (VGG), this article proposes an image saliency extraction model assisted by deep prior information, and uses a large-scale data set for training on the server to obtain a trained model, and then integrates multiple features. The saliency extraction algorithm is combined with the image saliency extraction model assisted by deep prior information, and a saliency extraction algorithm based on multi-feature fusion and deep prior information is proposed. A new method for extracting key frames of motion video is introduced in detail. Taking into account that sports videos in real applications are susceptible to interference from various factors, resulting in poor picture quality, this article constructs a new visual attention model for moving targets in sports videos, which integrates images. The combination of multiple features of the bottom-level features and the skin color confidence map of the moving target overcomes the problem that a single feature cannot fully express the moving target. Since the processing object in this article is for the moving target in the video of the sports room, the extracted moving target can provide samples for video post-processing. The experimental results show that the proposed key frame extraction algorithm can quickly grasp the pedestrian information in the motion video and provide effective processing samples for the motion target for video post-processing.

Multi-Sensor Integration for Key-Frame Extraction From First-Person Videos

Key-frame extraction for first-person vision (FPV) videos is a core technology for selecting important scenes and memorizing impressive life experiences in our daily activities. The difficulty of selecting key frames is the scene instability caused by head-mounted cameras used for capturing FPV videos. Because head-mounted cameras tend to frequently shake, the frames in an FPV video are noisier than those in a third-person vision (TPV) video. However, most existing algorithms for key-frame extraction mainly focus on handling the stable scenes in TPV videos. The technical development of key-frame extraction techniques for noisy FPV videos is currently immature. Moreover, most key-frame extraction algorithms mainly use visual information from FPV videos, even though our visual experience in daily activities is associated with human motions. To incorporate the features of dynamically changing scenes in FPV videos into our methods, integrating motions with visual scenes is essential. In this paper, we propose a novel key-frame extraction method for FPV videos that uses multi-modal sensor signals to reduce noise and detect salient activities via projecting multi-modal sensor signals onto a common space by canonical correlation analysis (CCA). We show that the two proposed multi-sensor integration models for key-frame extraction (a sparse-based model and a graph-based model) work well on the common space. The experimental results obtained using various datasets suggest that the proposed key-frame extraction techniques improve the precision of extraction and the coverage of entire video sequences.

Key Frame Extraction of Multi-Resolution Remote Sensing Images Under Quality Constraint

Abstract In key frame extraction of multi-resolution remote sensing image using traditional key frame image feature extraction method, only the feature information of remote sensing images, rather than cluster operation of the remote sensing images is considered, which leads to low efficiency and poor quality of extraction results. To this end, the key frame extraction algorithm of multi-resolution remote sensing image under quality constraint was proposed. Through similarity between image features and the selected image frame, rough key frame can be extracted. On this basis, the key frame extraction of multi resolution remote sensing image based on quality constraints was used to perform clustering operation for multi-resolution remote sensing image corresponding to rough key frame, which shortened the time length for retrieval of key frame image. According to the clustering results, multi-resolution remote sensing images were divided into several clusters. The key frame of each cluster can be obtained by calculating the distance between remote sensing image and cluster center. For key frames that had been determined, their quality was evaluated to meet standard, so as to realize effective extraction of key frame of multi-resolution remote sensing images. The experimental results show that the proposed method can significantly improve the quality of key frame extraction of multi-resolution remote sensing images.

Keyframe extraction using Pearson correlation coefficient and color moments

Keyframe extraction plays a significant role in wide variety of real-time video processing applications such as video summarization, video management and retrieval, etc. A keyframe captures the whole content of its shot and does not contain any redundant information. The keyframe extraction algorithms are facing challenges due to different visual characteristics in videos of different categories. Therefore, a single feature is not enough to capture visual characteristics of a variety of videos. In order to tackle this problem, we propose an approach of keyframe extraction that uses hybridization of features. In the present article, we propose a novel shot detection-based keyframe extraction algorithm based on combination of two features: one is Pearson correlation coefficient (PCC) and other is color moments (CM). The linear transformation invariance property of PCC facilitates the proposed algorithm to work well under varying lighting conditions. On the other hand, the scale and rotation invariance properties of color moments are beneficial for representation of complex objects that may be present in different poses and orientations. These sustained reasons support the combination of these two features, which brings significant benefits for keyframe extraction in the proposed method. The proposed method detects shot boundaries by employing combo feature set (PCC and CM). From each shot, the frame with highest mean and standard deviation is selected as keyframe. Furthermore, another important contribution is that we developed a new dataset by collecting the videos of different categories such as movies, news, serials, animations and personal interviews and made it available online. The proposed method is experimented on three datasets: two publicly available datasets and one dataset developed by us. The performance of the proposed method on these datasets has been evaluated on the basis of different evaluation parameters: figure of merit, detection percentage, accuracy, and missing factor. Principal advantage of proposed work lies in the fact that it is capable to detect both the abrupt and gradual shot transitions. In real-time videos, it is common to have abrupt and small transitions. The experimental results show the superior performance of the proposed method over the other state-of-the-art methods.

Key Frame Extraction Based on Binocular Cameras

Key frames provide the effective summary information for video retrieval and browsing, which mostly represents the main content for video sequences. The statistical results prove that color, as a feature that represents image content, proved widely used in the key frame extraction. Stereo correspondence has been an indispensable research direction in the field of computational stereo vision. Stereo vision is introduced for the purpose of improving previous key frame extraction algorithms, which is the key contribution of our work presented in this paper. A novel image similarity measurement is constructed by combining the color moment with SAD. Experimental results demonstrate that the average evaluation coefficient obtained by the improved algorithm in this paper reach up to 95% in the situation of camera stillness and motion. The novel algorithm solves the issue of redundancy which stems from the large changes of content among consecutive frames in the process of extracting key frames using monocular cameras, and performs more robust compared with traditional algorithms.

AN IMPROVED ACTION KEY FRAMES EXTRACTION ALGORITHM FOR COMPLEX COLOUR VIDEO SHOT SUMMARIZATION

Key frame extraction is one of the critical techniques in computer vision fields such as video search, video identification and video forgeries detection. The extracted key frames should be sufficient key frames that preserve main actions in a video with compact representation. The objective of this work is to improve our previous action key frames extraction algorithm (AKF) by adapting a threshold which selects action key frames as final key frames. The threshold adaptation was achieved by using the mean value, the standard deviation, and the L1-norm instead of the comparison of user summaries evaluation method to obtain a fully automatic video summarisation algorithm, and by eliminating the conditions in selecting the final key frames to reduce the complexity of the algorithm. We have validated our proposed Improved AKF on complex colour video shots instead of the simple grey level video shots. The Improved AKF algorithm was able to extract a compact number of action key frames by preventing redundant key frames, reduce processing complexity, and preserve sufficient information about the main actions in a video shot. We then evaluated the Improved AKF algorithm with the-state-of-the-art algorithms in terms of compression ratio using Paul videos and Shih-Tang dataset. The evaluation results showed that the Improved AKF algorithm achieved better compression ratio and retained sufficient information in the extracted action key frames under different testing video shots. Therefore, the improved AKF algorithm is a suitable technique for applications in computer vision fields such as passive object-based video authentication systems