Key Frame Extraction Research Articles

Automatic stenosis recognition from coronary angiography using convolutional neural networks

Background and objectiveCoronary artery disease, which is mostly caused by atherosclerotic narrowing of the coronary artery lumen, is a leading cause of death. Coronary angiography is the standard method to estimate the severity of coronary artery stenosis, but is frequently limited by intra- and inter-observer variations. We propose a deep-learning algorithm that automatically recognizes stenosis in coronary angiographic images. MethodsThe proposed method consists of key frame detection, deep learning model training for classification of stenosis on each key frame, and visualization of the possible location of the stenosis. Firstly, we propose an algorithm that automatically extracts key frames essential for diagnosis from 452 right coronary artery angiography movie clips. Our deep learning model is then trained with image-level annotations to classify the areas narrowed by over 50 %. To make the model focus on the salient features, we apply a self-attention mechanism. The stenotic locations are visualized using the activated area of feature maps with gradient-weighted class activation mapping. ResultsThe automatically detected key frame was very close to the manually selected key frame (average distance (1.70 ± 0.12) frame per clip). The model was trained with key frames on internal datasets, and validated with internal and external datasets. Our training method achieved high frame-wise area-under-the-curve of 0.971, frame-wise accuracy of 0.934, and clip-wise accuracy of 0.965 in the average values of cross-validation evaluations. The external validation results showed high performances with the mean frame-wise area-under-the-curve of (0.925 and 0.956) in the single and ensemble model, respectively. Heat map visualization shows the location for different types of stenosis in both internal and external data sets. With the self-attention mechanism, the stenosis could be precisely localized, which helps to accurately classify the stenosis by type. ConclusionsOur automated classification algorithm could recognize and localize coronary artery stenosis highly accurately. Our approach might provide the basis for a screening and assistant tool for the interpretation of coronary angiography.

Enhanced Video Classification System Using a Block-Based Motion Vector

The main objective of this work was to design and implement a support vector machine-based classification system to classify video data into predefined classes. Video data has to be structured and indexed for any video classification methodology. Video structure analysis involves shot boundary detection and keyframe extraction. Shot boundary detection is performed using a two-pass block-based adaptive threshold method. The seek spread strategy is used for keyframe extraction. In most of the video classification methods, selection of features is important. The selected features contribute to the efficiency of the classification system. It is very hard to find out which combination of features is most effective. Feature selection makes relevance to the proposed system. Herein, a support vector machine-based classifier was considered for the classification of video clips. The performance of the proposed system considered six categories of video clips: cartoons, commercials, cricket, football, tennis, and news. When shot level features and keyframe features, along with motion vectors, were used, 86% correct classification was achieved, which was comparable with the existing methods. The research concentrated on feature extraction where combination of selected features was given to a classifier to get the best classification performance.

Evaluation and Simulation of Sports Balance Training and Testing Equipment Based on Medical Video Image Analysis

Aiming at the shortcomings of video analysis system in the current process of balance force training, a new video analysis system for balance force training is proposed to analyze the test effect of balance force training. This paper evaluates and tests the function and utility of balance testing equipment by using the methods of sports training, sports biomechanics, mechanical design, sensors, video image analysis and computer-related research, combined with the latest research results of physical training at home and abroad. Firstly, several representative algorithms are selected in the frame difference method and background difference method. Then, these methods are tested according to the typical scene conditions, and the experimental results are used to compare and analyze the algorithms qualitatively and quantitatively, and the paper proposes the moving object detection and tracking based on GVF-Snake image model. The experimental results show that our proposed model has good detection effect compared with the background difference method based on Vibe algorithm, codebook model, Mixed Gauss model, adaptive Mixed Gauss model, median filter and adaptive background modeling. The system can accurately analyze the video and image information of sports training. The accuracy of key frame extraction is high, and the recall rate is high. It can be used to guide and evaluate balance force training. And the final evaluation results show the effectiveness of video image algorithm.

An automated approach to retrieve lecture videos using context based semantic features and deep learning

Video digitization is one of the emerging technologies holding significant importance over years in applications like video recording and video compression. There are different techniques available in the literature for the effective retrieval of videos. This research work presents a video retrieval scheme based on a deep learning strategy. Initially, the video archive is subjected to the keyframe extraction, for extracting useful keyframes from the video. The features extracted from the keyframes are stored in the feature database. The features are clustered using the Fuzzy C Means (FCM) algorithm. These clustered features have been provided to the deep learner for finding the optimal centroid for the incoming user query. For experimentation, the research has considered videos from different categories, and both the text query and the video query have been used for the retrieval. The experimental results demonstrate the efficiency of the proposed deep learning strategy in video retrieval and its achievement of improved values of 0.9620, 0.9682, and 0.9652 respectively for recall, precision, and F-measure.

Fast and robust key frame extraction method for gesture video based on high-level feature representation

In gesture video, the inner-frame difference is too subtle to be projected via low-level features, and the gesture frames, expressing semantic information, are distributed only among the tiny part of the whole video frame. This paper introduces a fast and robust key frame extraction method for gesture video, founded upon high-level feature representation to extract the gesture key frame precisely without affecting the semantic information. Firstly, a gesture video segmentation model is designed by employing SSD, which classify gesture video into the semantic scene and the static scene. And then, the 2D-DWT-based perceptual hash algorithm is studied to extract candidate static key frames. Afterward, the multi-channel gradient magnitude frequency histogram (HGMF-MC) based on improved VGG16 is developed as a new image descriptor. Finally, a key frame extraction mechanism based on HGMF-MC is proposed to generate gesture video summary of two scenes, respectively. Experiments consistently show the superiority of the proposed method on Chinese sign language, Cambridge, ChaLearn and CVRR-Hands gesture datasets. The results demonstrate that the method proposed is effective, which improves the video compression ratio and outperforms the state-of-the-art methods.

Emerging intelligent big data analytics for cloud and edge computing

Emerging intelligent big data analytics for cloud and edge computing

Key Frames Extraction Based on Optical-flow and Mutual Information Entropy

Key frames provide a suitable video summary and framework for video indexing, browsing and retrieval, and how to extract key frames is the core in the field of video retrieval. In order to solve the problem of redundancy and missing selection in key frame extraction, this paper proposed a key frame extraction algorithm based on optical flow and mutual information entropy. This algorithm integrates mutual information entropy and optical flow characteristics to extract key frames. First, the optical flow method calculated the total optical flow of each frame of the image, and selected the video frame with extreme optical flow difference in the neighbourhood as the key frame, and the other video frames were put into the candidate key frame set; by calculating the mutual information entropy of the key frame set, the minimum mutual information entropy is taken as the threshold; Then, The mutual information entropy of the candidate key frame set was calculated, and the image frames larger than the threshold value were put into the key frame set; Finally, redundant key frames are deleted through the measurement of inter-frame similarity, and the remaining key frames are the key frames to be extracted. The experimental results show that compared with other methods, the accuracy of extracted key frames by this approach is significantly improved on precision, recall and F-score, and the extracted key frames can reflect video content more effectively.

Retracted: Mathematical Model of Yield Forecast Based on Long and Short-Term Memory Image Neural Network

Yield prediction has always been the focus of stock investors' attention in stock investment. This research uses the virtual stock scene constructed by VR technology to explore the mathematical model of return rate prediction. First, neural nodes, activation functions and output functions are designed under the LSTM framework, and a recurrent neural network that can handle long-term sequences is constructed. Use VR technology to simulate stock data and extract key frames from the image sequence. Aiming at the timing of key curve nodes, this study uses a long and short-term memory neural network that is good at processing time series data to classify curve trends, and realizes the identification of abnormal fluctuations in virtual scenes of stock images. Then, according to the existing probability control asset allocation theory, the index of investment allocation is adjusted from the entire time series to a dynamic time series. According to the stock price decomposition of EEMD, the EEMD-LSTM model is constructed to predict the stock market price. Use EEMD to decompose the original sequence to obtain IMF and trend items containing various time-scale characteristics. Then, the LSTM neural network is used to predict and analyze each rate of return, and the analysis software Python is used to process and count the predicted results. At the same time, retrospective restrictions are regarded as riskier assets. Finally, the investment is made under conditions that do not allow shorting in the VR simulation scenario. The accuracy of the calculation is more than 70% except for the Shanghai and Shenzhen 300 Index reaching 66%. The research results show that in the case of different stock price forecasts, the prediction ability of LSTM neural network has been greatly improved compared with that of support vector machine, which fully reflects the advantages of LSTM in time series prediction.

KeyFrame extraction based on face quality measurement and convolutional neural network for efficient face recognition in videos

Indexing is the process of extracting a compact, significant and pertinent signature that describes the content of the data. This field has a broad spectrum of promising applications, such as the Face in Video Recognition (FiVR). Motivating the interest of researchers around the world. Since the video has a huge amount of data, the process of extracting the relevant frames becomes necessary and an essential step prior to performing face recognition. In this context, we propose a new method for extracting keyframes from videos based on face quality and deep learning for a face recognition task. The first step is the face detection using MTCNN detector, which detects five landmarks (the eyes, the two corners of the mouth and the nose). It limits face boundaries in a bounding box, and provides a confidence score. This method has two steps. The first step aims to generate the face quality score of each face in the data set prepared for the learning step. To generate quality scores, we use three face feature extractor including Gabor, LBP and HoG. The second step consist on training a deep Convolutional Neural Network in a supervised manner in order to select frames having the best face quality. The obtained results show the effectiveness of the proposed method compared to the methods of the state of the art.

Proposed Copyright Protection Systems for 3D Video Based on Key Frames

Recently, three Dimension (3D) multimedia technology has grown over a wide range, where the 3D used in many applications such as medical, television, cinema, game, education, etc. The developments also include the computer network, these evaluations make the store and distribute the media very easy and at the same time very critical, where any person can access these data and manipulate it or redistribute it with illegal copyrights. Therefore, it is necessary to find a watermarking technique to keep the security of these media. 3D multimedia, especially 3D Video contains a huge amount of data, trying achievement copyright protection for all these data consumes the computation, and slow down securing process, thus 3D video key frame extraction process is necessary. In this paper, the histogram-based key frame extraction technique is applied to extract essential and important data of the 3D video. A modified invisible, and blind watermark system is suggested in this research for 3D Video copyrights and authentication protection using Dual-Tree Complex Wavelet Transform (DTCWT). The experiment results show that 3D video key extraction system has given high compression ratio ( from 4:1 to 5:1) and the protection system could put an invisible watermark while keeping the quality of the 3D video, as well as the proposed watermark system, is robust where the watermark extracted successfully after applying a different types of attack.

Video Summarization using Keyframe Extraction Methods

Video summarization plays an important role in too many fields, such as video indexing, video browsing, video compression, video analyzing and so on. One of the fundamental units in the video structure analysis is the keyframe extraction, Keyframe provides meaningful frames from the video. The keyframe consists of the meaningful frame from the videos which help for video summarization. In this proposed model, we presented an approach that is based on Convolutional Neural Network, keyframe extraction from videos and static video summarization. First, the video should be converted to frames. Then we perform redundancy elimination techniques to reduce the redundancy from frames. Then extract the keyframes from video by using the Convolutional Neural Network(CNN) model. From the extracted keyframe, we form a video summarization.

Underwater Image Enhancement and Mosaicking System Based on A-KAZE Feature Matching

Feature extraction and matching is a key component in image stitching and a critical step in advancing image reconstructions, machine vision and robotic perception algorithms. This paper presents a fast and robust underwater image mosaicking system based on (2D)2PCA and A-KAZE key-points extraction and optimal seam-line methods. The system utilizes image enhancement as a preprocessing step to improve quality and allow for greater keyframe extraction and matching performance, leading to better quality mosaicking. The application focus of this paper is underwater imaging and it demonstrates the suitability of the developed system in advanced underwater reconstructions. The results show that the proposed method can address the problems of noise, mismatching and quality issues which are typically found in underwater image datasets. The results demonstrate the proposed method as scale-invariant and show improvements in terms of processing speed and system robustness over other methods found in the literature.

Gravitational search-based optimal deep neural network for occluded face recognition system in videos

Video surveillance is an effective method to improve public safety and privacy. Video surveillance technology has entered a stage where increasing video cameras are inexpensive but requiring open staff to evaluate the videos is expensive. The extensive researches conducted using machine learning for automatic face recognition doesn’t provide accurate results as of human evaluation. To enhance the biometric features of the security applications, automatic face recognition is used. Surveillance records incorporate various challenges for face recognition and face detection. For instance, facial recognition systems can be affected by the size of a face image, occlusion, posture, lighting conditions, and establishment, while recognition accuracy may be affected as a result of low objectives, occlusion, posture, light, and dimness. To conquer these obstacles, an effective face detection and recognition framework is proposed with optimal feature extraction methods. At first, the keyframes with face pictures are extracted using a strategy known as keyframe extraction using wavelet information. After the extraction of keyframes, the multi-angle movement feature, SURF feature, holo-entropy, and appearance features are used for feature extraction. Finally, the recognition can be done using optimal deep neural network based on the gravitational search algorithm. Therefore, the proposed method’s performance is evaluated using various benchmark video dataset. The efficiency of the proposed approach is evaluated by comparing it using sensitivity, specificity, accuracy, keyframe extraction time, etc.

Large-Scale Video Retrieval via Deep Local Convolutional Features

In this paper, we study the challenge of image-to-video retrieval, which uses the query image to search relevant frames from a large collection of videos. A novel framework based on convolutional neural networks (CNNs) is proposed to perform large-scale video retrieval with low storage cost and high search efficiency. Our framework consists of the key-frame extraction algorithm and the feature aggregation strategy. Specifically, the key-frame extraction algorithm takes advantage of the clustering idea so that redundant information is removed in video data and storage cost is greatly reduced. The feature aggregation strategy adopts average pooling to encode deep local convolutional features followed by coarse-to-fine retrieval, which allows rapid retrieval in the large-scale video database. The results from extensive experiments on two publicly available datasets demonstrate that the proposed method achieves superior efficiency as well as accuracy over other state-of-the-art visual search methods.

Sentiment key frame extraction in user-generated micro-videos via low-rank and sparse representation

It is a prevailing trend that the user-generated content (UGC) on social media is shifting toward mobile video and micro content. Sentiment analysis and emotion recognition extract the opinions expressed in UGC and are important to understand the fast-growing mobile micro-videos. Although extensive research efforts have been devoted to it, most of the existing studies pre-processed the video before extracting sentiment features, pre-sampling as an example. Due to the rather unstructured nature of user-generated videos and the sparsely expressed emotions, sampling before sentiment analysis can remove visual contents important to understand the video. In this paper, a novel unsupervised sentiment key frame extraction (SKFE) model based on low-rank and sparse representation is proposed. Sparsity is the important characteristic distinct to a video frame. The low-rank constraint is helpful to improve the robustness to noise and outliers. The l2,1-norm sparsity is adopted to improve the robustness via different indexes running through the feature dimensions and among the video frames. To better characterize the relationship between the local features and alleviate the sensitiveness of sparse coding, a regularization term based on the Laplacian matrix is introduced to preserve the consistency of sparse codes for similar local features. The experimental results on publicly available datasets demonstrate the effectiveness of the proposed SKFE model.

Video Summarization for Copyright Protection

Nowadays, a summarization of videos for copyright application is an interested research area in computer vision. Key frames extraction is crucial phase in video summarization. This work proposed a method for key frames extraction. The method consists of two operations for covering the content of the whole video sequence. Firstly, a set of candidates key frames are extracted based on adaptive thresholding of difference of histogram of adjacent frames of video data. Secondly, the extraction of a final key frames is done depending on computing Euclidian distance between two adjacent frames in candidates key frames set. The proposed method was tested with a set of standard and online television videos. The experimental results show the efficiency of the proposed system.

Keys for Action: An Efficient Keyframe-Based Approach for 3D Action Recognition Using a Deep Neural Network.

In this paper, we propose a novel and efficient framework for 3D action recognition using a deep learning architecture. First, we develop a 3D normalized pose space that consists of only 3D normalized poses, which are generated by discarding translation and orientation information. From these poses, we extract joint features and employ them further in a Deep Neural Network (DNN) in order to learn the action model. The architecture of our DNN consists of two hidden layers with the sigmoid activation function and an output layer with the softmax function. Furthermore, we propose a keyframe extraction methodology through which, from a motion sequence of 3D frames, we efficiently extract the keyframes that contribute substantially to the performance of the action. In this way, we eliminate redundant frames and reduce the length of the motion. More precisely, we ultimately summarize the motion sequence, while preserving the original motion semantics. We only consider the remaining essential informative frames in the process of action recognition, and the proposed pipeline is sufficiently fast and robust as a result. Finally, we evaluate our proposed framework intensively on publicly available benchmark Motion Capture (MoCap) datasets, namely HDM05 and CMU. From our experiments, we reveal that our proposed scheme significantly outperforms other state-of-the-art approaches.

Detection of shot boundaries and extraction of key frames for video retrieval

Detection of shot boundaries and extraction of key frames for video retrieval

A novel key-frame selection-based sign language recognition framework for the video data

ABSTRACT Sign language is a medium of communication for people with hearing disabilities. Static and dynamic gestures are identified in a video-based sign language recognition and translated them into humanly understandable phrases to achieve the communication objective. However, videos contain redundant Key-frames which require additional processing. Number of such Key-frames can be reduced. The selection of particular Key-frames without losing the required information is a challenging task. The Key-frame extraction algorithm is used which helps to speed-up the sign language recognition process by extracting essential key-frames. The proposed framework eliminates the computation overhead by picking up the distinct Key-frames for the recognition process. Discrete Wavelet Transform (DWT), Discrete Cosine Transform (DCT), and Histograms of Oriented Gradient (HOG) are used for unique features extraction. We used the bagged tree, boosted tree ensemble method, Fine KNN, and SVM for classification. We tested methodology on video-based datasets of Pakistani Sign Language. It achieved an overall 97.5% accuracy on 37 Urdu alphabets and 95.6% accuracy on 100 common words.

Network video summarization based on key frame extraction via superpixel segmentation

AbstractThe spread of insecure online video has been a serious social problem. The video summarization becomes one of key step for automatic filtering the expected video from the Internet. At present, the most existing video summarization methods are based on calculating the image similarity between video frames, so that the key frame can be selected properly. In this article, we introduce a superpixel segmentation based image similarity calculation, and then the metric is applied into video summarization. To identify the video key frames, we introduce superpixel segmentation to cluster the pixels locally by estimating the optical flow displacement field between successive frames, which can extract key frames and reduce video redundancy. On the VSUMM dataset and YouTube dataset, the experimental results demonstrate that the proposed method has clear advantages on both subjectively qualitative analysis and objectively quantitative evaluation comparing with the state of the art methods.