Key Frame Extraction Research Articles

Gated Recurrent Units and Recurrent Neural Network Based Multimodal Approach for Automatic Video Summarization

A typical video record aggregation system requires the concurrent performance of a large number of image processing tasks, including but not limited to image acquisition, pre-processing, segmentation, feature extraction, verification, and description. These tasks must be executed with utmost precision to ensure smooth system performance. Among these tasks, feature extraction and selection are the most critical. Feature extraction involves converting the large-scale image data into smaller mathematical vectors, and this process requires great skill. Various feature extraction models are available, including wavelet, cosine, Fourier, histogram-based, and edge-based models. The key objective of any feature extraction model is to represent the image data with minimal attributes and no loss of information. In this study, we propose a novel feature-variance model that detects differences in video features and generates feature-reduced video frames. These frames are then fed into a GRU-based RNN model, which classifies them as either keyframes or non-keyframes. Keyframes are then extracted to create a summarized video, while non-keyframes are reduced. Various key-frame extraction models are also discussed in this section, followed by a detailed analysis of the proposed summarization model and its results. Finally, we present some interesting observations about the proposed model and suggest ways to improve it.

Object of Interest and Unsupervised Learning-based Framework for an Effective Video Summarization Using Deep Learning

During this digital era, a large amount of visual data is being generated by various multimedia sources. A technique is urgently required to provide a clear and accurate summary of the original video to highlight the most informative segments of the video content. However, the varying video resolution, multi-dimensional feature representation, and massive storage create difficulties in key frame extraction techniques. As a result, many unnecessary features of the videos must be dropped to count their unique qualities. Therefore, a novel deep learning-based approach is proposed where frames are first retrieved using 25 frames per second; then, objects are detected on extracted frames using YOLOv5, and the frames with the target object only are processed further to overcome time consumption and high-speed computing hardware limitations. Further features are obtained using VGG-16 and object of Interest (OoI) based ResNet-50, respectively, and comparisons are performed to find the best solution. The extracted features are compressed using Principal Component Analysis (PCA) based on unsupervised Learning, which may efficiently minimize information loss, and potentially reduce dimension. By performing a comprehensive evaluation to obtain the best value of K using the Silhouette score, Candidate frames are extracted with the maximum mean and standard deviation from each K-means algorithm-based cluster. Pearson Correlation Coefficient (PCC) is used as a post- processing step to remove the redundant frames from the candidate frames and final keyframes extraction. The experiment was performed on the benchmark office dataset from industrial surveillance, which outperforms the state-of-the-art models in terms of recall score.

A multimodal fusion-based deep learning framework combined with keyframe extraction and spatial and channel attention for group emotion recognition from videos

A multimodal fusion-based deep learning framework combined with keyframe extraction and spatial and channel attention for group emotion recognition from videos

Human Action Recognition Using Key-Frame Attention-Based LSTM Networks

Human action recognition is a classical problem in computer vision and machine learning, and the task of effectively and efficiently recognising human actions is a concern for researchers. In this paper, we propose a key-frame-based approach to human action recognition. First, we designed a key-frame attention-based LSTM network (KF-LSTM) using the attention mechanism, which can be combined with LSTM to effectively recognise human action sequences by assigning different weight scale values to give more attention to key frames. In addition, we designed a new key-frame extraction method by combining an automatic segmentation model based on the autoregressive moving average (ARMA) algorithm and the K-means clustering algorithm. This method effectively avoids the possibility of inter-frame confusion in the temporal sequence of key frames of different actions and ensures that the subsequent human action recognition task proceeds smoothly. The dataset used in the experiments was acquired with an IMU sensor-based motion capture device, and we separately extracted the motion features of each joint using a manual method and then performed collective inference.

Key frame extraction method for lecture videos based on spatio-temporal subtitles.

Affected by the Corona Virus Disease 2019 (COVID-19), online lecture videos have witnessed an explosive growth. In the face of massive videos, this paper proposes a method for extracting key frames of lecture videos based on spatio-temporal subtitles, which can efficiently and quickly obtain effective information. Firstly, the spatio-temporal slices of subtitle area of the video sequence are extracted and spliced along the time axis to construct the video spatio-temporal subtitle. Then, the video spatio-temporal subtitle is processed in binarization, and the projection method is used to construct the SSPA curve of the video spatio-temporal subtitle. Finally, a selection method for steady-state key frame is designed, that is, the key frame extraction is realized by combining curve edge detection and subtitle existence threshold, which ensures the robustness of the proposed method. The test results of 8 videos show that the average value of the comprehensive index F1-score of the key frame extracted by the algorithm can reach 0.97, the average precision is 0.97, and the average recall rate is 0.98. It can effectively extract the key frames in lecture videos, and compared with other algorithms, the average running time is reduced to 0.072 of the original, which is helpful to extract video information quickly and accurately.

A Soft Sensor Model of Sintering Process Quality Index Based on Multi-Source Data Fusion

In complex industrial processes such as sintering, key quality variables are difficult to measure online and it takes a long time to obtain quality variables through offline testing. Moreover, due to the limitations of testing frequency, quality variable data are too scarce. To solve this problem, this paper proposes a sintering quality prediction model based on multi-source data fusion and introduces video data collected by industrial cameras. Firstly, video information of the end of the sintering machine is obtained via the keyframe extraction method based on the feature height. Secondly, using the shallow layer feature construction method based on sinter stratification and the deep layer feature extraction method based on ResNet, the feature information of the image is extracted at multi-scale of the deep layer and the shallow layer. Then, combining industrial time series data, a sintering quality soft sensor model based on multi-source data fusion is proposed, which makes full use of multi-source data from various sources. The experimental results show that the method effectively improves the accuracy of the sinter quality prediction model.

Static Video Summarization Using Video Coding Features with Frame-Level Temporal Subsampling and Deep Learning

There is an abundance of digital video content due to the cloud’s phenomenal growth and security footage; it is therefore essential to summarize these videos in data centers. This paper offers innovative approaches to the problem of key frame extraction for the purpose of video summarization. Our approach includes the extraction of feature variables from the bit streams of coded videos, followed by optional stepwise regression for dimensionality reduction. Once the features are extracted and their dimensionality is reduced, we apply innovative frame-level temporal subsampling techniques, followed by training and testing using deep learning architectures. The frame-level temporal subsampling techniques are based on cosine similarity and the PCA projections of feature vectors. We create three different learning architectures by utilizing LSTM networks, 1D-CNN networks, and random forests. The four most popular video summarization datasets, namely, TVSum, SumMe, OVP, and VSUMM, are used to evaluate the accuracy of the proposed solutions. This includes the precision, recall, F-score measures, and computational time. It is shown that the proposed solutions, when trained and tested on all subjective user summaries, achieved F-scores of 0.79, 0.74, 0.88, and 0.81, respectively, for the aforementioned datasets, showing clear improvements over prior studies.

ViSTORY: Effective Video Storyboard Generation with Visual Keyframes using Discrete Cosine Transform

Nowadays, multimedia content utility is increasing rapidly. Multimedia search engines like Google, Yahoo, Bing, etc., are available just a click away to all users. There are around 500-600 hours of video uploads per unit of time to the Internet. So, among other types of multimedia content, such as text and images, video is the most complicated content for indexing, browsing, and retrieval. Videos give more scope for implementation because of their complex and unstructured nature. This paper proposes a new method of video storyboard generation with keyframe extraction in spatial and frequency domains using Discrete Cosine Transform (DCT) for video summarization. It discusses the empirical appraisal of video visual keyframes with t-test analysis in comparison with spatial and frequency domains, resulting in a quick response to customer demands by providing static storyboards. This study proposes a new performance measure as matching frames by analyzing input videos and the standard benchmarks video dataset, i.e., Open Video Project (OVP) and SumMe. Among all the keyframe extraction techniques, DCT gives higher accuracy and a better matching rate.

A Two-Stage Image Frame Extraction Model -ISLKE for Live Gesture Analysis on Indian Sign Language

The new industry revolution focused on Smart and interconnected technologies along with the Robotics and Artificial Intelligence, Machine Learning, Data analytics etc. on the real time data to produce the value-added products. The ways the goods are being produced are aligned with the people’s life style which is witnessed in terms of wearable smart devices, digital assistants, self-driving cars etc. Over the last few years, an evident capturing of the true potential of Industry 4.0 in health service domain is also observed. In the same context, Sign Language Recognition- a breakthrough in the live video processing domain, helps the deaf and mute communities grab the attention of many researchers. From the research insights, it is clearly evident that precise extraction and interpretation of the gesture data along with an addressal of the prevailing limitations is a crucial task. This has driven the work to come out with a unique keyframe extraction model focusing on the preciseness of the interpretation. The proposed model ISLKE deals with a clustering-based two stage keyframe extraction process. It has experimented on daily usage vocabulary of Indian Sign Language (ISL) and attained an average accuracy of 96% in comparison to the ground-truth facts. It is also observed that with the two-stage approach, filtering of uninformative frames has reduced complexity and computational efforts. These key leads, help in the further development of commercial communication applications in order to reach the speech and hearing disorder communities.

A convolutional neural network based online teaching method using edge-cloud computing platform

Teaching has become a complex essential tool for students’ abilities, due to their different levels of learning and understanding. In the traditional offline teaching methods, dance teachers lack a target for students ‘classroom teaching. Furthermore, teachers have limited time, so they cannot take full care of each student’s learning needs according to their understanding and learning ability, which leads to the polarization of the learning effect. Because of this, this paper proposes an online teaching method based on Artificial Intelligence and edge calculation. In the first phase, standard teaching and student-recorded dance learning videos are conducted through the key frames extraction through a deep convolutional neural network. In the second phase, the extracted key frame images were then extracted for human key points using grid coding, and the fully convolutional neural network was used to predict the human posture. The guidance vector is used to correct the dance movements to achieve the purpose of online learning. The CNN model is distributed into two parts so that the training occurs at the cloud and prediction happens at the edge server. Moreover, the questionnaire was used to obtain the students’ learning status, understand their difficulties in dance learning, and record the corresponding dance teaching videos to make up for their weak links. Finally, the edge-cloud computing platform is used to help the training model learn quickly form vast amount of collected data. Our experiments show that the cloud-edge platform helps to support new teaching forms, enhance the platform’s overall application performance and intelligence level, and improve the online learning experience. The application of this paper can help dance students to achieve efficient learning.

Efficient key frame extraction and hybrid wavelet convolutional manta ray foraging for sports video classification

ABSTRACT Sports video classification (SVC) is now considered a challenging topic, therefore, developing an automatic sports scene classification technique has received tremendous interest. This research develops an efficient key frame extraction method and hybrid Wavelet Convolution Neural Network (WCNN) framework with optimization scheme to classify sports videos. Initially, input videos are converted into number of frames, and keyframes are extracted using Enhanced threshold with Discrete Wavelet Transform (ETDWT) method. Then, Cross Guided Bilateral Filter (CGBF) method eliminates the noise from the keyframe. After that, segmentation process is performed by the Fuzzy Equilibrium Optimizer (FEO) algorithm, and then motions are detected using the Farneback optical flow (OF) method. Finally, classification process is performed using Hybrid Wavelet Convolutional Manta Ray Foraging Optimization (HWCMRFO) algorithm to categorize different sports videos. The overall work is implemented using Python language. Simulation results proved that the proposed work achieved the highest accuracy (93.17%) compared to existing approaches.

Spatiotemporal Detection and Localization of Object Removal Video Forgery with Multiple Feature Extraction and Optimized Residual Network

Video forgery detection and localization is one of the most important issue due to the advanced editing software that provides strengthen to tools for manipulating the videos. Object based video tampering destroys the originality of the video. The main aim of the video forensic is to eradicate the forgeries from the original video that are useful in various applications. However, the research on detecting and localizing the object based video forgery with advanced techniques still remains the open and challenging issue. Many of the existing techniques have focused only on detecting the forged video under static background that cannot be applicable for detecting the forgery in tampered video. In addition to this, conventional techniques fail to extract the essential features in order to investigate the depth of the video forgery. Hence, this paper brings a novel technique for detecting and localizing the forged video with multiple features. The steps involved in this research are keyframe extraction, pre-processing, feature extraction and finally detection and localization of forged video. Initially, keyframe extraction uses the Gaussian mixture model (GMM) to extract frames from the forged videos. Then, the pre-processing stage is manipulated to convert the RGB frame into a grayscale image. Multi-features need to be extracted from the pre-processed frames to study the nature of the forged videos. In our proposed study, speeded up robust features (SURF), principal compound analysis histogram oriented gradients (PCA-HOG), model based fast digit feature (MBFDF), correlation of adjacent frames (CAF), the prediction residual gradient (PRG) and optical flow gradient (OFG) features are extracted. The dataset used for the proposed approach is collected from REWIND of about 40 forged and 40 authenticated videos. With the help of the DL approach, video forgery can be detected and localized. Thus, this research mainly focuses on detecting and localization of forged video based on the ResNet152V2 model hybrid with the bidirectional gated recurrent unit (Bi-GRU) to attain maximum accuracy and efficiency. The performance of this approach is finally compared with existing approaches in terms of accuracy, precision, F-measure, sensitivity, specificity, false-negative rate (FNR), false discovery rate (FDR), false-positive rate (FPR), Mathew’s correlation coefficient (MCC) and negative predictive value (NPV). The proposed approach assures the performance of 96.17% accuracy, 96% precision, 96.14% F-measure, 96.58% sensitivity, 96.5% specificity, 0.034 FNR, 0.04 FDR, 0.034 FPR, 0.92 MCC and 96% NPV, respectively. Along with is, the mean square error (MSE) and peak-to-signal-noise ratio (PSNR) for the GMM model attained about 104 and 27.95, respectively.

Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model.

Word-level sign language recognition (WSLR) is the backbone for continuous sign language recognition (CSLR) that infers glosses from sign videos. Finding the relevant gloss from the sign sequence and detecting explicit boundaries of the glosses from sign videos is a persistent challenge. In this paper, we propose a systematic approach for gloss prediction in WLSR using the Sign2Pose Gloss prediction transformer model. The primary goal of this work is to enhance WLSR's gloss prediction accuracy with reduced time and computational overhead. The proposed approach uses hand-crafted features rather than automated feature extraction, which is computationally expensive and less accurate. A modified key frame extraction technique is proposed that uses histogram difference and Euclidean distance metrics to select and drop redundant frames. To enhance the model's generalization ability, pose vector augmentation using perspective transformation along with joint angle rotation is performed. Further, for normalization, we employed YOLOv3 (You Only Look Once) to detect the signing space and track the hand gestures of the signers in the frames. The proposed model experiments on WLASL datasets achieved the top 1% recognition accuracy of 80.9% in WLASL100 and 64.21% in WLASL300. The performance of the proposed model surpasses state-of-the-art approaches. The integration of key frame extraction, augmentation, and pose estimation improved the performance of the proposed gloss prediction model by increasing the model's precision in locating minor variations in their body posture. We observed that introducing YOLOv3 improved gloss prediction accuracy and helped prevent model overfitting. Overall, the proposed model showed 17% improved performance in the WLASL 100 dataset.

Key-frame extraction based video watermarking using speeded up robust features and discrete cosine transform

Due to advancements in the internet and multimedia technologies, unauthorised users can easily modify video content. As a result, video authentication has been established as a viable solution for ensuring multimedia security. We propose a key-frame based video watermarking scheme based on discrete cosine transform (DCT). First, the pearson correlation coefficient (PCC) is used to detect the shot boundaries of the input video. To reduce the difficulties created by traditional video watermarking systems; an entropy measure is employed to detect key-frames from input video. Traditional schemes entail embedding the entire watermark into all frames of the video, which is inefficient and time-consuming. To improve the security, robustness, and imperceptibility of the proposed video watermarking scheme, speeded up robust feature points are extracted from each key-frame of the shot and used as reference points for embedding and detection of watermark. The embedded watermark is extracted blindly without using the original data during the extraction process. The results of the experiments reveal that the proposed technique effectively detects shot boundaries under a variety of camera operations and outperforms in terms of imperceptibility and resilience.

Key-frame extraction based video watermarking using speeded up robust features and discrete cosine transform

Due to advancements in the internet and multimedia technologies, unauthorised users can easily modify video content. As a result, video authentication has been established as a viable solution for ensuring multimedia security. We propose a key-frame based video watermarking scheme based on discrete cosine transform (DCT). First, the Pearson correlation coefficient (PCC) is used to detect the shot boundaries of the input video. To reduce the difficulties created by traditional video watermarking systems; an entropy measure is employed to detect key-frames from input video. Traditional schemes entail embedding the entire watermark into all frames of the video, which is inefficient and time-consuming. To improve the security, robustness, and imperceptibility of the proposed video watermarking scheme, speeded up robust feature points are extracted from each key-frame of the shot and used as reference points for embedding and detection of watermark. The embedded watermark is extracted blindly without using the original data during the extraction process. The results of the experiments reveal that the proposed technique effectively detects shot boundaries under a variety of camera operations and outperforms in terms of imperceptibility and resilience.

Deep Unsupervised Key Frame Extraction for Efficient Video Classification

Video processing and analysis have become an urgent task, as a huge amount of videos (e.g., YouTube, Hulu) are uploaded online every day. The extraction of representative key frames from videos is important in video processing and analysis since it greatly reduces computing resources and time. Although great progress has been made recently, large-scale video classification remains an open problem, as the existing methods have not well balanced the performance and efficiency simultaneously. To tackle this problem, this work presents an unsupervised method to retrieve the key frames, which combines the convolutional neural network and temporal segment density peaks clustering. The proposed temporal segment density peaks clustering is a generic and powerful framework, and it has two advantages compared with previous works. One is that it can calculate the number of key frames automatically. The other is that it can preserve the temporal information of the video. Thus, it improves the efficiency of video classification. Furthermore, a long short-term memory network is added on the top of the convolutional neural network to further elevate the performance of classification. Moreover, a weight fusion strategy of different input networks is presented to boost performance. By optimizing both video classification and key frame extraction simultaneously, we achieve better classification performance and higher efficiency. We evaluate our method on two popular datasets (i.e., HMDB51 and UCF101), and the experimental results consistently demonstrate that our strategy achieves competitive performance and efficiency compared with the state-of-the-art approaches.

Intercity Railway Risk Space Anomaly Detection Based on Train Predeparture Key Frame Extraction and IADN Network

Anomalies between the platform doors and train seriously threaten the safe operation of intercity railway train, and therefore, accurately selecting the appropriate time for anomaly detection in elaborate process of train access is vital. However, the scarcity of data samples, disadvantage of anomaly reconstruction, and unreliable reconstruction errors pose challenges to image reconstruction-based anomaly detection methods. To address these issues, this study proposes a method based on train predeparture key frame extraction and patch-level unsupervised network using image inpainting. The key frame extraction is designed to extract the key frame to assist subsequent detection by tracking the motion state of train door and converting states into the switching door signals. An unsupervised network, which named image-inpainting anomaly detection network (IADN), based on image-inpainting autoencoder (AE), local abnormal information enhancement, and global-attentive reconstruction error (GARE), is proposed for anomaly classification and localization. The network utilizes an image-inpainting AE based on masking strategy and dual-channel structure to reconstruct normal images and solve the anomaly reconstruction problem. Afterward, it proposes local abnormal information enhancement and GARE to enhance the difference between inputs and reconstructions to improve the accuracy. The proposed method is tested on the intercity railway risk space anomaly dataset, and the anomaly classification AUC, localization precision, and recall achieve 99.32%, 97.45%, and 96.29%, respectively, which outperforms state-of-the-art methods.

RETRACTED ARTICLE: Synchronous restoration of video key frame loss based on digital media communication protocol

This paper presents a synchronous recovery method for video key frame loss, aiming to analyze the experimental video through digital media feature extraction algorithm. By analyzing and designing effective communication protocols in real-time embedded systems, video data can be better processed. On this basis, key frames are restore synchronously through digital media communication protocol, and verified by comparing with other algorithms. Experimental results shows that the recall and precision of this algorithm for key frame extraction are 90.1% and 100% respectively, Among the three algorithms compared, the recall and precision of video key frame extraction based on single feature algorithm are the highest, 80.2% and 85.9% respectively. At the same time, the synchronization restoration time of this algorithm for lost key frames is 12.2 s, which took less time than other algorithms. It can be seen that the algorithm based on digital media feature extraction is of great significance for synchronous restoration of video key frame loss. It can be seen that the algorithm based on digital media feature extraction is of great significance for the synchronous recovery of video key frame loss, and can effectively promote the development of video data diversity and the improvement of information interaction. Under the digital media communication protocol, it can effectively synchronize the audio and video receiving end of multimedia information, and then recover and predict the lost frames in the video sequence. First, it can enhance the internal characteristics of video frames and the similarity and consistency of images between frames. Secondly, it can significantly improve the accuracy and efficiency of video data processing. Finally, from the perspective of practical development, the synchronous recovery of video key frame loss based on digital media communication protocol can effectively promote the sustainable development of media data, which has certain social value and practical significance in the current era of big data.

Human Action Recognition of Autonomous Mobile Robot Using Edge-AI

The development of autonomous mobile robots (AMRs) has brought with its requirements for intelligence and safety. Human action recognition (HAR) within AMR has become increasingly important because it provides interactive cognition between human and AMR. This study presents a full architecture for edge-artificial intelligence HAR (Edge-AI HAR) to allow AMR to detect human actions in real time. The architecture consists of three parts: a human detection and tracking network, a key frame extraction function, and a HAR network. The HAR network is a cascade of a DenseNet121 and a double-layer bidirectional long-short-term-memory (DLBiLSTM), in which the DenseNet121 is a pretrained model to extract spatial features from action key frames and the DLBiLSTM provides a deep two-directional LSTM inference to classify complicated time-series human actions. Edge-AI HAR undergoes two optimizations—ROS distributed computation and TensorRT structure optimization—to give a small model structure and high computational efficiency. Edge-AI HAR is demonstrated in two experiments using an AMR and is demonstrated to give an average precision of 97.58% for single action recognition and around 86% for continuous action recognition.

EFFECT OF KEYFRAMES EXTRACTION FROM THERMAL INFRARED VIDEO STREAM TO GENERATE DENSE POINT CLOUD OF THE BUILDING'S FACADE

Abstract. Keyframes extraction is required and effective for the 3D reconstruction of objects from a thermal video sequence to increase geometric accuracy, reduce the volume of aerial triangulation calculations, and generate the dense point cloud. The primary goal and focus of this paper are to assess the effect of keyframes extraction from the thermal infrared video sequence on the geometric accuracy of the dense point cloud generated. The method of keyframes extraction of thermal infrared video presented in this paper consists of three basic steps. (A) The ability to identify and remove blur frames from non-blur frames in a sequence of recorded frames. (B) The ability to apply the standard baseline condition between sequence frames to establish the overlap condition and prevent the creation of degeneracy conditions. (C) Evaluating degeneracy conditions and keyframes extraction using Geometric Robust Information Criteria (GRIC). The performance evaluation criteria for keyframes extraction in the generation of the thermal infrared dense point cloud in this paper are to assess the increase in density of the generated three-dimensional point cloud and reduce reprojection error. Based on the results and assessments presented in this paper, using keyframes increases the density of the thermal infrared dense point cloud by about 0.03% to 0.10% of points per square meter. It reduces the reprojection error by about 0.005% of pixels (2 times).