Video Key Frame Research Articles

Self-Supervised Learning to Detect Key Frames in Videos

Detecting key frames in videos is a common problem in many applications such as video classification, action recognition and video summarization. These tasks can be performed more efficiently using only a handful of key frames rather than the full video. Existing key frame detection approaches are mostly designed for supervised learning and require manual labelling of key frames in a large corpus of training data to train the models. Labelling requires human annotators from different backgrounds to annotate key frames in videos which is not only expensive and time consuming but also prone to subjective errors and inconsistencies between the labelers. To overcome these problems, we propose an automatic self-supervised method for detecting key frames in a video. Our method comprises a two-stream ConvNet and a novel automatic annotation architecture able to reliably annotate key frames in a video for self-supervised learning of the ConvNet. The proposed ConvNet learns deep appearance and motion features to detect frames that are unique. The trained network is then able to detect key frames in test videos. Extensive experiments on UCF101 human action and video summarization VSUMM datasets demonstrates the effectiveness of our proposed method.

Static video summarization using multi-CNN with sparse autoencoder and random forest classifier

A summarization system detects the parts of the input video that contain an essential message. Such a system aims to generate a very compact and meaningful representation of the original video. A novel method to detect key-frames for static summarization is presented in this paper. The method detects key-frames based on feature vectors extracted from multiple pre-trained Convolutional Neural Network models (Multi-CNN). The features are extracted using four pre-trained models of CNN. These vectors are fed to Sparse Autoencoder, which outputs a combined representation of the input feature vectors. The key-frames of input video are extracted based on combined feature vectors using Random Forest Classifier. The evaluation of the method is done using two datasets: VSUMM and OVP, based on user summaries present in the ground-truth. The method was able to achieve an average F-score of 0.83 on VSUMM dataset and 0.82 on OVP dataset, respectively. The method attained promising results compared to other state-of-the-art methods in the literature. Multi-CNN model was also able to generate high-quality summaries consistently from videos of all categories. Further experiments prove that Multi-CNN model in combination with Random Forest classifier performs better than other classifiers considered in the study.

A Low-Latency Object Detection Algorithm for the Edge Devices of IoV Systems

The emergence of edge computing (EC) and intelligent vision-based driver assistance system is of great significance for the prospective development of Internet of Vehicle (IoV). The additional computation capability and extensive network coverage provides energy-limited smart devices with more opportunities to enable IoV system for time-sensitive applications. However, when implemented in a vision-based driver assistance system, the transmission of a large amount of redundant data not only causes delay but also severely compromises the accuracy of object detection. In this paper, an improved object detection algorithm based on video key-frame for latency reduction on edge IoV system is proposed. It can significantly improve latency reduction performance at the expense of small detection accuracy. In our proposal, we adopt an important coefficient and frame similarity comparison algorithm to filter redundant frames and achieve key frames for object detection. Then an improved Haar-like feature based classification algorithm is used for object detection under the edge computation model. Finally, a scalable cluster object detection system is implemented as a practical EC case to verify our proposal, and extensive simulations confirm the superiority of the proposed scheme over regular schemes. It can speed up about 84 times with 40% of the similar frames filtered in comparison.

Emerging intelligent big data analytics for cloud and edge computing

Emerging intelligent big data analytics for cloud and edge computing

Proposed Copyright Protection Systems for 3D Video Based on Key Frames

Recently, three Dimension (3D) multimedia technology has grown over a wide range, where the 3D used in many applications such as medical, television, cinema, game, education, etc. The developments also include the computer network, these evaluations make the store and distribute the media very easy and at the same time very critical, where any person can access these data and manipulate it or redistribute it with illegal copyrights. Therefore, it is necessary to find a watermarking technique to keep the security of these media. 3D multimedia, especially 3D Video contains a huge amount of data, trying achievement copyright protection for all these data consumes the computation, and slow down securing process, thus 3D video key frame extraction process is necessary. In this paper, the histogram-based key frame extraction technique is applied to extract essential and important data of the 3D video. A modified invisible, and blind watermark system is suggested in this research for 3D Video copyrights and authentication protection using Dual-Tree Complex Wavelet Transform (DTCWT). The experiment results show that 3D video key extraction system has given high compression ratio ( from 4:1 to 5:1) and the protection system could put an invisible watermark while keeping the quality of the 3D video, as well as the proposed watermark system, is robust where the watermark extracted successfully after applying a different types of attack.

基于概率学习像素分类法的农业植物视频图像阴影处理技术研究

In order to solve the problem of difficult pre-processing of crop video image shadows, a probable learning pixel classification method is proposed to study its processing technology. The algorithm effectively detects the shadow area by performing intelligent video collaborative detection on the shaded parts of the crop video sequence. Firstly, the cloud collaborative detection algorithm that can be widely used in agriculture was proposed. The video key frame was obtained and the background modeling algorithm with strong adaptability to crop illumination was applied to realize real-time detection of the target, so as to construct the crop pixel model. Finally, the proposed algorithm and the constructed model are applied to the processing of shadows of agricultural plant video images for experimental verification. The results show that in video frames 47, 194 and 258, the probable learning pixel classification method can be used to determine the shaded part of each frame, which can greatly improve the detection accuracy of crop shadows. The research in this paper shows that the probability learning pixel classification method can better enhance the shadow robustness and accuracy of crop video images.

Human Interaction Recognition Based on Whole-Individual Detection.

Human interaction recognition technology is a hot topic in the field of computer vision, and its application prospects are very extensive. At present, there are many difficulties in human interaction recognition such as the spatial complexity of human interaction, the differences in action characteristics at different time periods, and the complexity of interactive action features. The existence of these problems restricts the improvement of recognition accuracy. To investigate the differences in the action characteristics at different time periods, we propose an improved fusion time-phase feature of the Gaussian model to obtain video keyframes and remove the influence of a large amount of redundant information. Regarding the complexity of interactive action features, we propose a multi-feature fusion network algorithm based on parallel Inception and ResNet. This multi-feature fusion network not only reduces the network parameter quantity, but also improves the network performance; it alleviates the network degradation caused by the increase in network depth and obtains higher classification accuracy. For the spatial complexity of human interaction, we combined the whole video features with the individual video features, making full use of the feature information of the interactive video. A human interaction recognition algorithm based on whole–individual detection is proposed, where the whole video contains the global features of both sides of action, and the individual video contains the individual detail features of a single person. Making full use of the feature information of the whole video and individual videos is the main contribution of this paper to the field of human interaction recognition and the experimental results in the UT dataset (UT–interaction dataset) showed that the accuracy of this method was 91.7%.

Network video summarization based on key frame extraction via superpixel segmentation

AbstractThe spread of insecure online video has been a serious social problem. The video summarization becomes one of key step for automatic filtering the expected video from the Internet. At present, the most existing video summarization methods are based on calculating the image similarity between video frames, so that the key frame can be selected properly. In this article, we introduce a superpixel segmentation based image similarity calculation, and then the metric is applied into video summarization. To identify the video key frames, we introduce superpixel segmentation to cluster the pixels locally by estimating the optical flow displacement field between successive frames, which can extract key frames and reduce video redundancy. On the VSUMM dataset and YouTube dataset, the experimental results demonstrate that the proposed method has clear advantages on both subjectively qualitative analysis and objectively quantitative evaluation comparing with the state of the art methods.

Automatic reduction of wireless capsule endoscopy reviewing time based on factorization analysis

Wireless capsule endoscopy (WCE) is a painless and easy process to screening of the gastrointestinal (GI) tract. During WCE procedure, a huge amount of the endoscopy video frames is generated, however, a limited amount of data is actually useful for diagnosis. Manually reviewing all endoscopy frames is tedious, time-consuming and prone to physician error. In this paper, we propose a novel capsule video summarization framework to reduce WCE reviewing time using the factorization analysis based on sliding window singular value decomposition (SVD). Through the proposed approach, in a quality assessment stage, poor quality frames are removed from the endoscopy video. Adaptive sliding window SVD is then employed to extract the salient video frames. The average recall and precision were estimated by 0.92 and 0.94 for our local database. The quantitative and qualitative results demonstrate that the proposed approach outperforms the existing WCE keyframe extraction methods and provides video keyframes to the gastroenterologists in the clinical applications without discarding significant diagnosis information.

An End to End Framework With Adaptive Spatio-Temporal Attention Module for Human Action Recognition

Human action recognition is a challenging task in computer vision. Modeling spatial-temporal information in videos effectively is crucial for the performance improvement of action recognition. In this paper, we introduce an end to end framework, Spatio-Temporal Attention ConvNet (STACNet), which combines two novel attention modules and convolutional neural networks for action recognition. Two novel attention modules, Spatial Attention Module (SAM) and Temporal Attention Module (TAM), are proposed respectively. Spatial Attention Module is established by fusing the value feature and the gradient feature of the feature map, making the representation of ConvNet for action recognition focus on the informative motion regions of actions. Temporal Attention Module is built by combining global average pooling and global max pooling to explore key frames in videos. With the two attention modules, STACNet can adaptively distinguish key frames in a sequence and selectively pay different levels of attention to different spatial motion regions of human actions, at virtually negligible increase in computation cost. We demonstrate the effectiveness of SAM and TAM for action recognition, respectively. The experimental results show that STACNet can obtain superior performance on the datasets of HMDB51 and UCF101.

Video Key Frame Monitoring Algorithm and Virtual Reality Display Based on Motion Vector

In this article, a motion vector-based video key frame detection algorithm is proposed to solve the problem of miss election and missing selection caused by the difficulty in detecting the moving target characteristics of the video key frame. Firstly, the entropy of adjacent frame difference and the two-dimensional entropy of image are introduced, and the combination of the two is taken as the measurement of the difference between video frames. Secondly, outliers are detected by statistical tools to obtain the lens boundary, thus realizing the adaptive lens detection of video content. Then, ViBe algorithm is used to detect the foreground object in the video sequence and extract the scale-invariant feature transformation features of the foreground moving object. Finally, the motion vector is introduced, and the sum of the block matching results is motion vector by partitioning the two adjacent frames and performing block matching. The magnitude of motion vector reflects the intensity of motion in the video, so active and inactive motion regions are obtained, and the similarity of video frames is calculated according to the defined formula, and key frames are extracted in these regions respectively. The experimental results show that the detection algorithm proposed in this article improves the video results with rich motion information obviously, and the objective indexes and subjective scores are improved to some extent, which improves the universality of the algorithm. In addition, this article also studies the display mode of key frame extraction results in virtual reality environment. The key frame display mode of video in virtual reality is optimized mainly by changing the display mode of information and changing the scene and testing the result of user task execution.

The adaptive key frames extraction method based on representativeness and independence features

For the existing traditional method to extract key frames from the video sequence with gradual change of content, there are problems of having too much redundancy or too much loss in the extraction results and other issues, a new key frames extracting method for adaptively extracting the video key frames has proposed in the paper. The experimental results have shown that the proposed method can effectively extract video key frames, especially for the video with gradual change of content, which can express more succinctly and comprehensively. The experiments on a large number of videos containing various scenes have shown that the key frames extracted by the proposed algorithm can more fully express the main content of video. When dealing with content gradient video, the effect is better. The content adaptively determines the number of key frames based on video.

The adaptive key frames extraction method based on representativeness and independence features

For the existing traditional method to extract key frames from the video sequence with gradual change of content, there are problems of having too much redundancy or too much loss in the extraction results and other issues, a new key frames extracting method for adaptively extracting the video key frames has proposed in the paper. The experimental results have shown that the proposed method can effectively extract video key frames, especially for the video with gradual change of content, which can express more succinctly and comprehensively. The experiments on a large number of videos containing various scenes have shown that the key frames extracted by the proposed algorithm can more fully express the main content of video. When dealing with content gradient video, the effect is better. The content adaptively determines the number of key frames based on video.

Applying Pose Estimation to Predict Amateur Golf Swing Performance Using Edge Processing

We present an analysis and approach for utilizing vision-based pose estimation to find key video frames in a full golf swing to assist in providing feedback for improvement. Using both still photos and videos, the proposed system discovers key moments in the golf swing to be evaluated and can identify metrics of the golfer such as posture, swing tempo, and swing consistency. These key frames can also predict the swing outcome by creating a path projection. The images and videos processed analyze the golf swing from a down the line perspective. For the computations, we utilize a low cost tensor processing unit (TPU) to run inference and data processing which set the performance baseline for the video capturing system. Hardware and pose estimation limitations and inaccuracies are identified and compensated for by using a Savitzky-Golay filter. This will allow for a markerless swing tracking analysis system in a low cost, small form factor.

Social media video summarization using multi-Visual features and Kohnen's Self Organizing Map

Social networking tools such as Facebook, YouTube, Twitter, and Instagram, are becoming major platforms for communication. YouTube as one of the primary video sharing platform serves over 100 million distinct videos, 300 hours of videos are uploaded on YouTube every minute along with textual data. This massive amount of multimedia data needs to be managed with high efficiency, the irrelevant and redundant data needs to be removed. Video summarization ideals with the problem of redundant data in a video. A summarized video contains the most distinct frames which are termed as key frames. Most of the research work on key frames extraction considers only a single visual feature which is not sufficient for capturing the full pictorial details and hence affecting the quality of video summary generated. So there is a need to explore multiple visual features for key frames extraction. In this research work a key frame extraction technique based upon fusion of four visual features namely: correlation of RGB color channels, color histogram, mutual information and moments of inertia is proposed. Kohonen Self Organizing map as a clustering approach is used to find the most representative frames from the list of frames coming after fusion. Useless frames are discarded and frames having maximum Euclidean distance within a cluster are selected as final key frames. The results of the proposed technique are compared with the existing video summarization techniques: User generated summary, Video SUMMarization (VSUMM), and Video Key Frame Extraction through Dynamic Delaunay Clustering (VKEDDCSC) which shows a considerable improvement in terms of fidelity and Shot Reconstruction Degree (SRD) score.

Deep Image-to-Video Adaptation and Fusion Networks for Action Recognition.

Existing deep learning methods for action recognition in videos require a large number of labeled videos for training, which is labor-intensive and time-consuming. For the same action, the knowledge learned from different media types, e.g., videos and images, may be related and complementary. However, due to the domain shifts and heterogeneous feature representations between videos and images, the performance of classifiers trained on images may be dramatically degraded when directly deployed to videos. In this paper, we propose a novel method, named Deep Image-to-Video Adaptation and Fusion Networks (DIVAFN), to enhance action recognition in videos by transferring knowledge from images using video keyframes as a bridge. The DIVAFN is a unified deep learning model, which integrates domain-invariant representations learning and cross-modal feature fusion into a unified optimization framework. Specifically, we design an efficient cross-modal similarities metric to reduce the modality shift among images, keyframes and videos. Then, we adopt an autoencoder architecture, whose hidden layer is constrained to be the semantic representations of the action class names. In this way, when the autoencoder is adopted to project the learned features from different domains to the same space, more compact, informative and discriminative representations can be obtained. Finally, the concatenation of the learned semantic feature representations from these three autoencoders are used to train the classifier for action recognition in videos. Comprehensive experiments on four real-world datasets show that our method outperforms some state-of-the-art domain adaptation and action recognition methods.

An Adaptive Correlation Based Video Data Mining using Machine Learning

With the immense growth in the multimedia contents for education and other purposes, the availability of the video contents has also increased Nevertheless, the retrieval of the content is always a challenge. The identification of two video contents based on internal content similarity highly depends on extraction of key frames and that makes the process highly time complex. In the recent time, many of research attempts have tried to approach this problem with the intention to reduce the time complexity using various methods such as video to text conversion and further analysing both extracted text similarity analysis. Regardless to mention, this strategy is again language dependent and criticised for various reasons like local language dependencies and language paraphrase dependencies. Henceforth, this work approaches the problem with a different dimension with reduction possibilities of the video key frames using adaptive similarity. The proposed method analyses the key frames extracted from the library content and from the search video data based on various parameters and reduces the key frames using adaptive similarity. Also, this work uses machine learning and parallel programming algorithms to reduce the time complexity to a greater extend. The final outcome of this work is a reduced time complex algorithm for video data-based search to video content retrieval. The work demonstrates a nearly 50% reduction in the key frame without losing information with nearly 70% reduction in time complexity and 100% accuracy on search results.

A Novel Hybrid Adaptive Filter to Improve Video Keyframe Clustering to Support Event Resolution in Cricket Videos

Automated summary generation of sports videos poses many challenges of detecting exciting events of a game. In our research, we focus on the table of content-based video summarization for cricket videos to facilitate efficient indexing of cricket events. Initially, we have identified event boundaries accurately to detect the event start and end points. To distinguish the different types of events, we need to analyze the sequence of the camera focus area. By observing the characteristics, we have categorized the camera focus area into several categories. To overcome the challenge of low accuracy we have introduced a novel algorithm for adaptive filtering for the comparison of hue histogram. The results prove that this algorithm is sufficient for accurate image clustering and this may be used in other sports event clustering as well.

Video Abstraction in H.264/AVC Compressed Domain

Video abstraction allows searching, browsing and evaluating videos only by accessing the useful contents. Most of the studies are using pixel domain, which requires the decoding process and needs more time and process consuming than compressed domain video abstraction. In this paper, we present a new video abstraction method in H.264/AVC compressed domain, AVAIF. The method is based on the normalized histogram of extracted I-frame prediction modes in H.264 standard. The frames’ similarity is calculated by intersecting their I-frame prediction modes’ histogram. Moreover, fuzzy c-means clustering is employed to categorize similar frames and extract key frames. The results show that the proposed method achieves on average 85% accuracy and 22% error rate in compressed domain video abstraction, which is higher than the other tested methods in the pixel domain. Moreover, on average, it generates video key frames that are closer to human summaries and it shows robustness to coding parameters.

Domain independent redundancy elimination based on flow vectors for static video summarization

Video summarization aims to find a compact representation of input videos. The method finds out interesting parts of the video by discarding the remaining parts of the video. The abstracts thus generated enhances browsing and retrieval of video data. The quality of summaries generated by video summarization algorithms can be improved if the redundant frames in the input video are taken care of before summarization. This paper presents a novel domain-independent method for redundancy elimination from input videos before summarization maintaining keyframes in the original video. The frames of input video are first presampled by selecting two frames in one second. The flow vectors between consecutive frames are computed using SIFT Flow algorithm. The magnitude of flow vectors at each pixel position of the frame are summed up to find the displacement magnitude between the consecutive frames. The redundant frames are filtered out based on local averaging of the displacement values. The evaluation of the method is done using two standard datasets namely VSUMM and OVP. The results demonstrate that an average reduction rate of 97.64% is achieved consistently on videos of all categories. The method also gives superior results compared to other state-of-the-art redundancy elimination methods for video summarization