Key Frame Extraction Method Research Articles

Multi-features combinatorial optimization for keyframe extraction

<abstract><p>Recent advancements in network and multimedia technologies have facilitated the distribution and sharing of digital videos over the Internet. These long videos contain very complex contents. Additionally, it is very challenging to use as few frames as possible to cover the video contents without missing too much information. There are at least two ways to describe these complex videos contents with minimal frames: the keyframes extracted from the video or the video summary. The former lays stress on covering the whole video contents as much as possible. The latter emphasizes covering the video contents of interest. As a consequence, keyframes are widely used in many areas such as video segmentation and object tracking. In this paper, we propose a keyframe extraction method based on multiple features via a novel combinatorial optimization algorithm. The key frame extraction is modeled as a combinatorial optimization problem. A fast dynamic programming algorithm based on a forward non-overlapping transfer matrix in polynomial time and a 0-1 integer linear programming algorithm based on an overlapping matrix is proposed to solve our maximization problem. In order to quantitatively evaluate our approach, a long video dataset named 'Animal world' is self-constructed, and the segmentation evaluation criterions are introduced. A good result is achieved on 'Animal world' dataset and a public available Keyframe-Sydney KFSYD dataset <sup>[<xref ref-type="bibr" rid="b1">1</xref>]</sup>.</p></abstract>

Key frame extraction based on quaternion Fourier transform with multiple features fusion

Key frame extraction, an important technique to improve video viewing efficiency and reduce video redundancy, has attracted considerable attention. Notwithstanding, existing key frame extraction for surveillance video not only fails to extract the local detail information of the target accurately but also preprocess the image with grayscale, so the integrity of the image information is also damaged. In order to solve the above-described problems, the paper presents a key frame extraction method based on the quaternion Fourier transform with the multiple features fusion. Initially, the method extracts dynamic and static features that can characterize the color image information separately. Then, the quaternion matrix is used to perform the quaternion Fourier transform on the different features to obtain the fused phase spectrum of the Fourier transform. Immediately after, the information containing the overall structure of the image is filtered by Gaussian filtering, and the quaternion Fourier inverse transform is performed to obtain the fused feature map. Finally, an adaptive key frame filtering criterion, which uses the fused feature map to extract the key frames is constructed in this paper for accurately extracting key frames. Notably, experimental results indicate that the proposed method ensures the integrity of the image information while accurately capturing the global and local motion state of the target.

An Accurate Urine Red Blood Cell Detection Method Based on Multi-Focus Video Fusion and Deep Learning with Application to Diabetic Nephropathy Diagnosis

Background and Objective: Detecting urine red blood cells (U-RBCs) is an important operation in diagnosing nephropathy. Existing U-RBC detection methods usually employ single-focus images to implement such tasks, which inevitably results in false positives and missed detections due to the abundance of defocused U-RBCs in the single-focus images. Meanwhile, the current diabetic nephropathy diagnosis methods heavily rely on artificially setting a threshold to detect the U-RBC proportion, whose accuracy and robustness are still supposed to be improved. Methods: To overcome these limitations, a novel multi-focus video dataset in which the typical shape of all U-RBCs can be captured in one frame is constructed, and an accurate U-RBC detection method based on multi-focus video fusion (D-MVF) is presented. The proposed D-MVF method consists of multi-focus video fusion and detection stages. In the fusion stage, D-MVF first uses the frame-difference data of multi-focus video to separate the U-RBCs from the background. Then, a new key frame extraction method based on the three metrics of information entropy, edge gradient, and intensity contrast is proposed. This method is responsible for extracting the typical shapes of U-RBCs and fusing them into a single image. In the detection stage, D-MVF utilizes the high-performance deep learning model YOLOv4 to rapidly and accurately detect U-RBCs based on the fused image. In addition, based on U-RBC detection results from D-MVF, this paper applies the K-nearest neighbor (KNN) method to replace artificial threshold setting for achieving more accurate diabetic nephropathy diagnosis. Results: A series of controlled experiments are conducted on the self-constructed dataset containing 887 multi-focus videos, and the experimental results show that the proposed D-MVF obtains a satisfactory mean average precision (mAP) of 0.915, which is significantly higher than that of the existing method based on single-focus images (0.700). Meanwhile, the diabetic nephropathy diagnosis accuracy and specificity of KNN reach 0.781 and 0.793, respectively, which significantly exceed the traditional threshold method (0.719 and 0.759). Conclusions: The research in this paper intelligently assists microscopists to complete U-RBC detection and diabetic nephropathy diagnosis. Therefore, the work load of microscopists can be effectively relieved, and the urine test demands of nephrotic patients can be met.

Visual video evaluation association modeling based on chaotic pseudo-random multi-layer compressed sensing for visual privacy-protected keyframe extraction

In current society, artificial intelligence processing technology offers convenient video monitoring, but also raises the risk of privacy leakage. Theoretically, the data used in intelligent video processing methods may directly convey visual information containing private content. For the above problem, this paper uses a multi-layer visual privacy-protected (VPP) coding method to blur private content in the video at the visual level, while avoiding the loss of important visual features contained in the video as much as possible. And this provides a guarantee of the quality of the subsequent keyframe extraction step. Then a visual evaluation algorithm is proposed for assessing the quality of VPP-encoded video privacy protection. And the experiment shows that the results are consistent with those of subjective evaluation. In addition, for VPP-encoded video, we propose an unsupervised two-layer clustering keyframe extraction method with corresponding performance evaluation index. Finally, an association model is established to balance the privacy protection quality and the keyframe extraction performance.

Keyframe Extraction Using Linear Rotation Invariant Coordinates

Keyframe extraction is a widely applied remedy for issues faced with 3D motion capture -based computer animation. In this paper, we propose a novel keyframe extraction method, where the motion is represented in linear rotation invariant coordinates and the dimensions covering 95% of the data are automatically selected using principal component analysis. Then, by K-means classification, the summarized data is clustered and a keyframe is extracted from each cluster based on cosine similarity. To validate the method, an online user study was conducted. The results of the user study show that 45% of the participants preferred the keyframes extracted using the proposed method, outperforming the alternative by 6%.

A Video Key Frame Extraction Method Based on Multiview Fusion

A massive amount of video data is stored in the real-time road monitoring system, especially in high-speed scenes. Traditional methods of video key frame extraction have the problems of large computation and long-time consumption. Thus, it is imperative to decrease the massive video data generated by monitoring and help researchers to study key frames. Aiming at the above problems, we propose an efficient key frame extraction method based on multiview fusion, where the autoencoder is used to compress the video data. Specifically, all the video frames of the video data are subjected to feature dimensionality reduction, and the features after dimensionality reduction are subjected to multiview fusion. Finally, dynamic programming and clustering are used to extract key frames. The experimental results show that the proposed method has lower computational complexity in extracting key frames, while the mutual information in the extracted key frames is large. It illustrates the reliability and efficiency of the proposed method, which provides technical support for subsequent video research.

Video key frame extraction based on scale and direction analysis

Currently, efficient and accurate key frame extraction for massive videos remains a challenge, especially in surveillance applications. To overcome the inaccuracy of traditional key frame extraction, this paper proposes a method for key frame extraction via analyzing scale and direction of motion trajectory on spatiotemporal slice (MTSS). The proposed method employs both local and global motion state changes of object as the metric to extract key frames and meanwhile gives a strategy which integrates coarse extraction with partial fine re-extraction of spatiotemporal slices to capture these state changes. Firstly, coarse extraction of spatiotemporal slices is employed to detect motion segments. Then partial fine re-extraction is performed on the motion segments to obtain MTSS. Finally, the frames at the inflexions of scale and direction of MTSS are extracted as key frames. The experimental results have demonstrated that the proposed method outperforms existing state-of-the-art methods in terms of accuracy for various videos while requiring a comparable or even less computation time.

Spatio-Temporal Deep Feature Fusion for Human Action Recognition

Action Recognition plays a vital role in many secure applications. The objective of this paper is to identify actions more accurately. This paper focuses on the two stream network in which keyframe extraction method is utilized before extracting spatial features. The temporal features are extracted using Attentive Correlated Temporal Feature (ACTF) which uses Long Short Term Memory (LSTM) for deep features. The spatial and temporal features are fused and classified using multi Support Vector Machine (multiSVM) classifier. Experiments are done on HMDB51 and UCF101 datasets. The results of the proposed method are compared with recent methods in terms of accuracy. The proposed method is proved to work better than other methods by achieving an accuracy of 96% for HMDB51 dataset and 98% for UCF101 dataset.

A video summarization framework based on activity attention modeling using deep features for smart campus surveillance system.

Like other business domains, digital monitoring has now become an integral part of almost every academic institution. These surveillance systems cover all the routine activities happening on the campus while producing a massive volume of video data. Selection and searching the desired video segment in such a vast video repository is highly time-consuming. Effective video summarization methods are thus needed for fast navigation and retrieval of video content. This paper introduces a keyframe extraction method to summarize academic activities to produce a short representation of the target video while preserving all the essential activities present in the original video. First, we perform fine-grain activity recognition using a realistic Campus Activities Dataset (CAD) by modeling activity attention scores using a deep CNN model. In the second phase, we use the generated attention scores for each activity category to extract significant video frames. Finally, we evaluate the inter-frame similarity index used to reduce the number of redundant frames and extract only the representative keyframes. The proposed framework is tested on different videos, and the experimental results show the performance of the proposed summarization process.

A Deep Learning and Clustering Extraction Mechanism for Recognizing the Actions of Athletes in Sports.

In sports, the essence of a complete technical action is a complete information structure pattern and the athlete's judgment of the action is actually the identification of the movement information structure pattern. Action recognition refers to the ability of the human brain to distinguish a perceived action from other actions and obtain predictive response information when it identifies and confirms it according to the constantly changing motion information on the field. Action recognition mainly includes two aspects: one is to obtain the required action information based on visual observation and the other is to judge the action based on the obtained action information, but the neuropsychological mechanism of this process is still unknown. In this paper, a new key frame extraction method based on the clustering algorithm and multifeature fusion is proposed for sports videos with complex content, many scenes, and rich actions. First, a variety of features are fused, and then, similarity measurement can be used to describe videos with complex content more completely and comprehensively; second, a clustering algorithm is used to cluster sports video sequences according to scenes, eliminating the need for shots in the case of many scenes. It is difficult and complicated to detect segmentation; third, extracting key frames according to the minimum motion standard can more accurately represent the video content with rich actions. At the same time, the clustering algorithm used in this paper is improved to enhance the offline computing efficiency of the key frame extraction system. Based on the analysis of the advantages and disadvantages of the classical convolutional neural network and recurrent neural network algorithms in deep learning, this paper proposes an improved convolutional network and optimization based on the recognition and analysis of human actions under complex scenes, complex actions, and fast motion compared to post-neural network and hybrid neural network algorithm. Experiments show that the algorithm achieves similar human observation of athletes' training execution and completion. Compared with other algorithms, it has been verified that it has very high learning rate and accuracy for the athlete's action recognition.

Key Frame Extraction Method of Music and Dance Video Based on Multicore Learning Feature Fusion

The purpose of video key frame extraction is to use as few video frames as possible to represent as much video content as possible, reduce redundant video frames, and reduce the amount of computation, so as to facilitate quick browsing, content summarization, indexing, and retrieval of videos. In this paper, a method of dance motion recognition and video key frame extraction based on multifeature fusion is designed to learn the complicated and changeable dancer motion recognition. Firstly, multiple features are fused, and then the similarity is measured. Then, the video sequences are clustered by the clustering algorithm according to the scene. Finally, the key frames are extracted according to the minimum amount of motion. Through the quantitative analysis and research of the simulation results of different models, it can be seen that the model proposed in this paper can show high performance and stability. The breakthrough of video clip retrieval technology is bound to effectively promote the inheritance and development of dance, which is of great theoretical significance and practical value.

Retracted] Key Frame Extraction Method for Minors’ Participation in Online Short‐Form Video from the Perspective of Government Administration

With the progress of the Internet era, the audience scope of online short‐form video and live broadcast platform is rapidly expanding. In this situation, the physical and mental health of minors is affected and harmed, and serious social problems have been caused. In view of the random images of minors participating in short online videos, the monitoring department should strengthen supervision and control and establish a good network atmosphere. The focus of surveillance is to identify short‐form videos in which minors participate, monitor the content of short‐form videos, and effectively identify minors. On the basis of analyzing and studying the existing methods of moving object extraction, the method of spatio‐temporal information combined with geometric curve evolution is used to improve the effect of moving object extraction. Firstly, the time dynamic information of video frames is fully utilized and the initial contour of the target is segmented by high‐order statistical algorithm. Based on the improved watershed algorithm, the spatial frames of the original image are segmented and the secondary contour of the target is obtained by combining the spatial and temporal information. The level set evolution algorithm is used to remove the noise and information void in the secondary target, and the final moving target is obtained. Through the extraction of key frames, it can effectively control the appearance of minors in short‐form videos, which can actively guide the benign development of the platform, and put forward governance suggestions based on the perspective of government management.

Key Frame Extraction of Surveillance Video Based on Fractional Fourier Transform

With the vigorous development of national infrastructure construction and public information construction, video surveillance systems have gradually penetrated various fields. The current key frame extraction technology has inadequate target details and inaccurate judgment of local actions. Addressing this problem, a key frame extraction method based on fractional Fourier transform is proposed. This method obtained the phase spectra information of different orders by performing fractional Fourier transform on the surveillance video frames. Next, the method designed an adaptive algorithm based on the golden section point to select the transformation order. Then, the phase spectrum information of two adjacent frames was used to characterize the changes in the global and local motion states of the target. The final step was to extract key frames based on this. Experimental results show that, compared with the previous methods, the key frames extracted by the method proposed in this paper can correctly capture the changes in the global and local motion states of the target.

Key Frame Extraction for Sports Training Based on Improved Deep Learning

With the rapid technological advances in sports, the number of athletics increases gradually. For sports professionals, it is obligatory to oversee and explore the athletics pose in athletes’ training. Key frame extraction of training videos plays a significant role to ease the analysis of sport training videos. This paper develops a sports actions’ classification system for accurately classifying athlete’s actions. The key video frames are extracted from the sports training video to highlight the distinct actions in sports training. Subsequently, a fully convolutional network (FCN) is used to extract the region of interest (ROI) pose detection of frames followed by the application of a convolution neural network (CNN) to estimate the pose probability of each frame. Moreover, a distinct key frame extraction approach is established to extract the key frames considering neighboring frames’ probability differences. The experimental results determine that the proposed method showed better performance and can recognize the athlete’s posture with an average classification rate of 98%. The experimental results and analysis validate that the proposed key frame extraction method outperforms its counterparts in key pose probability estimation and key pose extraction.

A novel keyframe extraction method for video classification using deep neural networks

Combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs) produces a powerful architecture for video classification problems as spatial–temporal information can be processed simultaneously and effectively. Using transfer learning, this paper presents a comparative study to investigate how temporal information can be utilized to improve the performance of video classification when CNNs and RNNs are combined in various architectures. To enhance the performance of the identified architecture for effective combination of CNN and RNN, a novel action template-based keyframe extraction method is proposed by identifying the informative region of each frame and selecting keyframes based on the similarity between those regions. Extensive experiments on KTH and UCF-101 datasets with ConvLSTM-based video classifiers have been conducted. Experimental results are evaluated using one-way analysis of variance, which reveals the effectiveness of the proposed keyframe extraction method in the sense that it can significantly improve video classification accuracy.

An Online Arrangement Method of Difficult Actions in Competitive Aerobics Based on Multimedia Technology

In accordance with the development trend of competitive aerobics’ arrangement structure, this paper studies the online arrangement method of difficult actions in competitive aerobics based on multimedia technology to improve the arrangement effect. RGB image, optical flow image, and corrected optical flow image are taken as the input modes of difficult action recognition network in competitive aerobics video based on top-down feature fusion. The key frames of input modes in competitive aerobics video are extracted by using the key frame extraction method based on subshot segmentation of a double-threshold sliding window and fully connected graph. Through forward propagation, the score vector of video relative to all categories is obtained, and the probability score of probability distribution is obtained after normalization. The human action recognition in competitive aerobics video is completed, and the online arrangement of difficult action in competitive aerobics is realized based on this. The experimental results show that this method has a high accuracy in identifying difficult actions in competitive aerobics video; the online arrangement of difficult actions in competitive aerobics has obvious advantages, meets the needs of users, and has strong practicability.

Multifeature fusion action recognition based on key frames

SummaryAs an important technology in computer vision, video‐based human action recognition has a great commercial value, which has attracted extensive attention in the field of computer vision and pattern recognition in both academia and industry. To date, there are a wide variety of applications of human action recognition, such as surveillance, robotics, health care, video searching, and human–computer interaction. However, there are many challenges involved in human action recognition in videos, such as cluttered backgrounds, occlusions, viewpoint variation, execution rate, and camera motion. However, data redundancy and single feature were largely limited the accuracy of human action recognition. In this article, adopting the key frame extraction and multifeature fusion techniques, a novel action recognition method was proposed, which can improve the recognition accuracy. The main works are as follows: 1) in order to solve the problem of data redundancy, a key frame extraction method based on node contribution weighting is proposed to extract video key frames; 2) different kinds of information flows are extracted from the obtained key frame sequences, and different convolutional neural networks are used to obtain corresponding classification results and merge, so as to better complement the information in different flows. Lastly, the experimental results show that our method improves the accuracy of action recognition.

Application Research of Key Frames Extraction Technology Combined with Optimized Faster R-CNN Algorithm in Traffic Video Analysis

The intelligent transportation system under the big data environment is the development direction of the future transportation system. It effectively integrates advanced information technology, data communication transmission technology, electronic sensing technology, control technology, and computer technology and applies them to the entire ground transportation management system to establish a real-time, accurate, and efficient comprehensive transportation management system that works on a large scale and in all directions. Intelligent video analysis is an important part of smart transportation. In order to improve the accuracy and time efficiency of video retrieval schemes and recognition schemes, this article firstly proposes a segmentation and key frame extraction method for video behavior recognition, using a multi-time scale dual-stream network to extract video features, improving the efficiency and efficiency of video behavior detection. On this basis, an improved algorithm for vehicle detection based on Faster R-CNN is proposed, and the Faster R-CNN network feature extraction layer is improved by using the principle of residual network, and a hole convolution is added to the network to filter out the redundant features of high-resolution video images to improve the problem of vehicle missed detection in the original algorithm. The experimental results show that the key frame extraction technology combined with the optimized Faster R-CNN algorithm model greatly improves the accuracy of detection and reduces the leakage. The detection rate is satisfactory.

Intelligent Sports Video Classification Based on Deep Neural Network (DNN) Algorithm and Transfer Learning.

Traditional text annotation-based video retrieval is done by manually labeling videos with text, which is inefficient and highly subjective and generally cannot accurately describe the meaning of videos. Traditional content-based video retrieval uses convolutional neural networks to extract the underlying feature information of images to build indexes and achieves similarity retrieval of video feature vectors according to certain similarity measure algorithms. In this paper, by studying the characteristics of sports videos, we propose the histogram difference method based on using transfer learning and the four-step method based on block matching for mutation detection and fading detection of video shots, respectively. By adaptive thresholding, regions with large frame difference changes are marked as candidate regions for shots, and then the shot boundaries are determined by mutation detection algorithm. Combined with the characteristics of sports video, this paper proposes a key frame extraction method based on clustering and optical flow analysis, and experimental comparison with the traditional clustering method. In addition, this paper proposes a key frame extraction algorithm based on clustering and optical flow analysis for key frame extraction of sports video. The algorithm effectively removes the redundant frames, and the extracted key frames are more representative. Through extensive experiments, the keyword fuzzy finding algorithm based on improved deep neural network and ontology semantic expansion proposed in this paper shows a more desirable retrieval performance, and it is feasible to use this method for video underlying feature extraction, annotation, and keyword finding, and one of the outstanding features of the algorithm is that it can quickly and effectively retrieve the desired video in a large number of Internet video resources, reducing the false detection rate and leakage rate while improving the fidelity, which basically meets people's daily needs.

Summarization of Wireless Capsule Endoscopy Video Using Deep Feature Matching and Motion Analysis

Conventional Wireless capsule endoscopy (WCE) video summary generation techniques apprehend an image by extracting hand crafted features, which are not essentially sufficient to encapsulate the semantic similarity of endoscopic images. Use of supervised methods for extraction of deep features from an image need an enormous amount of accurate labelled data for training process. To solve this, we use an unsupervised learning method to extract features using convolutional auto encoder. Furthermore, WCE images are classified into similar and dissimilar pairs using fixed threshold derived through large number of experiments. Finally, keyframe extraction method based on motion analysis is used to derive a structured summary of WCE video. Proposed method achieves an average F-measure of 91.1% with compression ratio of 83.12%. The results indicate that the proposed method is more efficient compared to existing WCE video summarization techniques.