Key Frame Extraction Research Articles

SOZIL: Self-Optimal Zero-Shot Imitation Learning

Zero-shot imitation learning has demonstrated its superiority to learn complex robotic tasks with less human participation. Recent studies show convincing performance under the condition that the robot follows the demonstration strictly by the learned inverse model. However, these methods are difficult to achieve satisfactory performance in imitation when the demonstration is suboptimal, and the learning of the learned inverse models is vulnerable to label ambiguity issues. In this paper, we propose Self-Optimal Zero-shot Imitation Learning (SOZIL) to tackle these problems. The contribution of SOZIL is twofold. First, Goal Consistency Loss (GCL) is designed to learn the multi-step goal-conditioned policy from exploration data. By directly using the goal state as supervision, GCL solves the label ambiguity problem caused by trajectory and action diversity. Second, Estimation-based Keyframe Extraction (EKE) is developed to optimize demonstrations. We formulate the keyframe extraction process as a path optimization problem under suboptimal control. By predicting the performance of the learned policy in executing transitions of any two states, EKE creates a directed graph containing all candidate paths and extracts keyframes by solving the graph’s shortest path problem. Furthermore, the proposed method is evaluated with various simulated and real-world robotic manipulating experiments such as cable harness assembly, rope manipulation, and block moving. Experimental results show that SOZIL achieves a superior success rate and manipulation efficiency than baselines.

Server node video processing based on feature depth analysis algorithm

ABSTRACT The complex and diverse server video data leads to the problem of effective retrieval of these data. The current shot edge detection algorithm and key frame extraction algorithm in server node video processing have problems such as poor extraction performance and poor adaptability. Therefore, the research combined the feature depth analysis to improve the two, and the performance was verified by experiments. The shot detection algorithm is verified by modifying the secondary detection model. This method can detect lens mutation, gradual change and other phenomena well, and the accuracy rate can reach 99.7%. The precision under the gradient lens is 92.08%, far higher than 63.50% and 85.39% of ISIFT and CS-DFS. In the verification experiment using Convolution Neural Networks (CNNs) key frame extraction algorithm, the number of key frame extractions of the proposed algorithm can reach up to 88 frames. Compared with other methods, the accuracy of the algorithm studied can reach 99.67%, which is higher than the comparison algorithm. In general, the improved algorithm proposed in the study has high adaptability to edge detection and the ability to express key frame video, and has high practicability in actual server node video processing.

A study on table tennis landing point detection algorithm based on spatial domain information

To address the limitations of computer vision-assisted table tennis ball detection, which heavily relies on vision acquisition equipment and exhibits slow processing speed, we propose a real-time calculation method for determining the landing point of table tennis balls. This novel approach is based on spatial domain information and reduces the dependency on vision acquisition equipment. This method incorporates several steps: employing dynamic color thresholding to determine the centroid coordinates of all objects in the video frames, utilizing target area thresholding and spatial Euclidean distance to eliminate interference balls and noise, optimizing the total number of video frames through keyframe extraction to reduce the number of operations for object recognition and landing point detection, and employing the four-frame difference slope method and polygonal area determination to detect the landing point and area of the target object, thereby obtaining precise coordinates and their corresponding areas. Experimental results on the above method on the Jetson Nano development board show that the dynamic color thresholding method achieves a detection speed of 45.3 fps. The keyframe extraction method correctly identifies the landing point frames with an accuracy rate exceeding 93.3%. In terms of drop point detection, the proposed method achieves 78.5% overall accuracy in detecting table tennis ball drop points while ensuring real-time detection. These experiments validate that the proposed method has the ability to detect table tennis ball drop points in real time and accurately in low frame rate vision acquisition devices and real environments.

HY-LSTM: A new time series deep learning architecture for estimation of pedestrian time to cross in advanced driver assistance system

Advanced driver assistance systems (ADASs), particularly pedestrian protection systems (PPSs), have emerged as a hot research topic with the goal of enhancing traffic safety. The development of reliable on-board pedestrian detection systems is a critical problem for PPSs. It is extremely difficult to provide the required resilience of this type of system due to the fluctuating look of pedestrians (e.g., varied attire, changing size, aspect ratio, and dynamic shape) and the unstructured surroundings. The detection of pedestrians has gained huge focus among researchers because of its huge applications in the domain of automated vehicles. In the previous decades, the majority of examinations are done to obtain better solutions for detecting pedestrians, but fewer of them concentrated on determining the pedestrian. It is one of crucial interest to study regarding transport safety as it implies minimizing the count of traffic collisions and protecting pedestrians who are more susceptible to accidents. Predicting pedestrian conduct is critical for road safety, traffic management systems, ADAS, and autonomous vehicles in general. The fundamental problem in the field of self-driving and smart automobiles is identifying impediments, particularly people, and taking action to avoid collisions with them. Various studies have been conducted in this sector by many researchers, yet there are still many mistakes in the proper identification of pedestrians. Hence, this paper devises an approach to estimate pedestrian time for crossing in an advanced driving assistance system (ADAS). The inputted videos undergo keyframe extraction wherein crucial keyframes are extracted. The deep joint segmentation is further applied for identifying the pedestrian followed by intention classification. Then, the estimation of pedestrian time to cross is predicted using Hybrid-Long short term memory (HY-LSTM), and it is a new time series model obtained by unifying LSTM and Object-based convolution neural network (OCNN), where the layer and hyperparameters of OCNN are optimally derived using Gradient Chef Based Optimization (GCBO). The proposed GCBO-HY-LSTM outperformed showing the least Mean absolute error (MAE) of 0.038, Mean square error (MSE) of 0.029, and Root Mean square error (RMSE) of 0.170.

An Enhanced Spatio-Temporal Human Detected Keyframe Extraction

Due to the immense availability of Closed-Circuit Television surveillance, it is quite difficult for crime investigation due to its huge storage and complex background. Content-based video retrieval is an excellent method to identify the best Keyframes from these surveillance videos. As the crime surveillance reports numerous action scenes, the existing keyframe extraction is not exemplary. At this point, the Spatio-temporal Histogram of Oriented Gradients - Support Vector Machine feature method with the combination of Background Subtraction is appended over the recovered crime video to highlight the human presence in surveillance frames. Additionally, the Visual Geometry Group trains these frames for the classification report of human-detected frames. These detected frames are processed to extract the keyframe by manipulating an inter-frame difference with its threshold value to favor the requisite human-detected keyframes. Thus, the experimental results of HOG-SVM illustrate a compression ratio of 98.54%, which is preferable to the proposed work's compression ratio of 98.71%, which supports the criminal investigation.

Application of Multimedia Data Feature Extraction Technology in Teaching Classical Oil Painting

The cross-modal oil painting image generated by traditional methods makes it easy to miss the important information of the target part, and the generated image lacks realism. This paper combines the feature extraction technology of multimedia data with the generation confrontation network in deep learning, puts forward a generation model of classic oil painting, and applies it to university teaching. Firstly, the key frame extraction algorithm is used to extract the key frames in the video, and the channel attention network is introduced into the pre-trained ResNet-50 network to extract the static features of 2D images in short oil painting videos. Then, the depth feature mapping is carried out in the time dimension by using the double-stream I3D network, and the feature representation is enhanced by combining static and dynamic features. Finally, the high-dimensional features in the depth space are mapped to the two-dimensional space by using the opposition generation network to generate classic oil painting pictures.

Early Fire Detection Using Long Short-Term Memory-Based Instance Segmentation and Internet of Things for Disaster Management.

Fire outbreaks continue to cause damage despite the improvements in fire-detection tools and algorithms. As the human population and global warming continue to rise, fires have emerged as a significant worldwide issue. These factors may contribute to the greenhouse effect and climatic changes, among other detrimental consequences. It is still challenging to implement a well-performing and optimized approach, which is sufficiently accurate, and has tractable complexity and a low false alarm rate. A small fire and the identification of a fire from a long distance are also challenges in previously proposed techniques. In this study, we propose a novel hybrid model, called IS-CNN-LSTM, based on convolutional neural networks (CNN) to detect and analyze fire intensity. A total of 21 convolutional layers, 24 rectified linear unit (ReLU) layers, 6 pooling layers, 3 fully connected layers, 2 dropout layers, and a softmax layer are included in the proposed 57-layer CNN model. Our proposed model performs instance segmentation to distinguish between fire and non-fire events. To reduce the intricacy of the proposed model, we also propose a key-frame extraction algorithm. The proposed model uses Internet of Things (IoT) devices to alert the relevant person by calculating the severity of the fire. Our proposed model is tested on a publicly available dataset having fire and normal videos. The achievement of 95.25% classification accuracy, 0.09% false positive rate (FPR), 0.65% false negative rate (FNR), and a prediction time of 0.08 s validates the proposed system.

Analyzing audiovisual data for understanding user's emotion in human−computer interaction environment

PurposeAlthough numerous signal modalities are available for emotion recognition, audio and visual modalities are the most common and predominant forms for human beings to express their emotional states in daily communication. Therefore, how to achieve automatic and accurate audiovisual emotion recognition is significantly important for developing engaging and empathetic human–computer interaction environment. However, two major challenges exist in the field of audiovisual emotion recognition: (1) how to effectively capture representations of each single modality and eliminate redundant features and (2) how to efficiently integrate information from these two modalities to generate discriminative representations.Design/methodology/approachA novel key-frame extraction-based attention fusion network (KE-AFN) is proposed for audiovisual emotion recognition. KE-AFN attempts to integrate key-frame extraction with multimodal interaction and fusion to enhance audiovisual representations and reduce redundant computation, filling the research gaps of existing approaches. Specifically, the local maximum–based content analysis is designed to extract key-frames from videos for the purpose of eliminating data redundancy. Two modules, including “Multi-head Attention-based Intra-modality Interaction Module” and “Multi-head Attention-based Cross-modality Interaction Module”, are proposed to mine and capture intra- and cross-modality interactions for further reducing data redundancy and producing more powerful multimodal representations.FindingsExtensive experiments on two benchmark datasets (i.e. RAVDESS and CMU-MOSEI) demonstrate the effectiveness and rationality of KE-AFN. Specifically, (1) KE-AFN is superior to state-of-the-art baselines for audiovisual emotion recognition. (2) Exploring the supplementary and complementary information of different modalities can provide more emotional clues for better emotion recognition. (3) The proposed key-frame extraction strategy can enhance the performance by more than 2.79 per cent on accuracy. (4) Both exploring intra- and cross-modality interactions and employing attention-based audiovisual fusion can lead to better prediction performance.Originality/valueThe proposed KE-AFN can support the development of engaging and empathetic human–computer interaction environment.

EI-RNN-based text generation for the static and dynamic isolated sign language videos

SLR, which assists hearing-impaired people to communicate with other persons by sign language, is considered as a promising method. However, as the features of some of the static SL could be the same as the feature in a single frame of dynamic Isolated Sign Language (ISL), the generation of accurate text corresponding to the SL is necessary during the SLR. Therefore, Edge-directed Interpolation-based Recurrent Neural Network (EI-RNN)-centered text generation with varied features of the static and dynamic Isolated SL is proposed in this article. Primarily, ISL videos are converted to frames and pre-processed with key frame extraction and illumination control. After that, the foreground is separated with the Symmetric Normalised Laplacian-centered Otsu Thresholding (SLOT) technique for finding accurate key points in the human pose. The human pose’s key points are extracted with the Media Pipeline Holistic (MPH) pipeline approach and to improve the features of the face and hand sign, the resultant frame is fused with the depth image. After that, to differentiate the static and dynamic actions, the action change in the fused frames is determined with a correlation matrix. After that, to engender the output text for the respective SL, features are extracted individually as of the static and dynamic frames. It is obtained from the analysis that when analogized to the prevailing models, the proposed EI-RNN’s translation accuracy is elevated by 2.05% in INCLUDE 50 Indian SL based Dataset and Top 1 Accuracy 2.44% and Top 10 accuracy, 1.71% improved in WLASL 100 American SL.

Keyframe-guided Video Swin Transformer with Multi-path Excitation for Violence Detection

Abstract Violence detection is a critical task aimed at identifying violent behavior in video by extracting frames and applying classification models. However, the complexity of video data and the suddenness of violent events present significant hurdles in accurately pinpointing instances of violence, making the extraction of frames that indicate violence a challenging endeavor. Furthermore, designing and applying high-performance models for violence detection remains an open problem. Traditional models embed extracted spatial features from sampled frames directly into a temporal sequence, which ignores the spatio-temporal characteristics of video and limits the ability to express continuous changes between adjacent frames. To address the existing challenges, this paper proposes a novel framework called ACTION-VST. First, a keyframe extraction algorithm is developed to select frames that are most likely to represent violent scenes in videos. To transform visual sequences into spatio-temporal feature maps, a multi-path excitation module is proposed to activate spatio-temporal, channel and motion features. Next, an advanced Video Swin Transformer-based network is employed for both global and local spatio-temporal modeling, which enables comprehensive feature extraction and representation of violence. The proposed method was validated on two large-scale datasets, RLVS and RWF-2000, achieving accuracies of over 98 and 93%, respectively, surpassing the state of the art.

Action recognition method based on lightweight network and rough-fine keyframe extraction

To address the issues of large number of parameters and low recognition accuracy of action recognition networks, we propose an effective action recognition method based on lightweight network and rough-fine keyframe extraction. The method consists of three modules. The first module proposes a keyframe extraction network based on grayscale and feature descriptors, and employs a rough-fine idea to extract video keyframe. It reduces the redundancy of keyframe and enhances their ability to express action semantics. The second module introduces an attention-based feature extraction network, which combines decoupling ideas with attention mechanisms to enhance the accuracy of the action recognition network, while significantly reducing the network parameters. The third module is an improved attention module which optimizes the representation of local information. Finally, addition of the residual module fuses feature information between different convolutional layers. Experiments on two different datasets show that the number of parameters in the proposed method is only 6.4M. On publicly available datasets of HMDB51 and UCF101, the method achieves recognition accuracy of 75.69% and 93.18% without pre-training, respectively. The proposed method is valid and feasible on multiple public datasets.

Analysis of the prospect of the development of university innovation and entrepreneurship courses based on the chain-type intelligent education model

Abstract Based on the chained intelligent education model, this paper provides practical training and practice activities for the development of university innovation and entrepreneurship courses in colleges and universities by constructing a university innovation and entrepreneurship learning platform to achieve resource sharing of multi-professional experimental bases, thus ensuring that the development of innovation and entrepreneurship courses in all majors in colleges and universities is satisfied. Then the learning platform system is designed and implemented, and the system technical support includes classroom key frame extraction based on deep neural network, S-T graph and Rt-Ch frame feature analysis, and classroom face dataset. Finally, the construction guarantee measures and management suggestions of the university innovation and entrepreneurship platform are proposed. The platform can stimulate students’ entrepreneurial spirit, improve their entrepreneurial ability, expand the knowledge related to innovation and entrepreneurship, and enhance the practical experience. The statistical analysis of the development of the course affecting innovation and entrepreneurship through the survey analysis method yielded a score of 2.17 (standard score of 54.27) for the item of guiding other students in the innovation and entrepreneurship course, which is the lowest score among the 27 questions, indicating that students’ main consciousness has not yet been formed, and their consciousness and ability of active learning and guiding others to learn are to be improved. This study improves students’ innovation spirit, entrepreneurial consciousness and entrepreneurial ability through the transfer of innovative and entrepreneurial knowledge, and it has a guiding reference value for college students to grow into high-quality and innovative talents.

Auto encoder with mode‐based learning for keyframe extraction in video summarization

AbstractThe exponential increase in video consumption has created new difficulties for browsing and navigating through video more effectively and efficiently. Researchers are interested in video summarization because it offers a brief but instructive video version that helps users and systems save time and effort when looking for and comprehending relevant content. Key frame extraction is a method of video summarization that only chooses the most important frames from a given video. In this article, a novel supervised learning method ‘TC‐CLSTM Auto Encoder with Mode‐based Learning’ using temporal and spatial features is proposed for automatically choosing keyframes or important sub‐shots from videos. The method was able to achieve an average F‐score of 84.35 on TVSum dataset. Extensive tests on benchmark data sets show that the suggested methodology outperforms state‐of‐the‐art methods.

Non-redundant shift-invariant complex wavelet transform and fractional gorilla troops optimization-based deep convolutional neural network for video watermarking

With the development of multimedia processing approaches along with the Internet leads to various issues, like data tampering, data piracy, and illicit dissemination. The quick progression of fast communication networks for digital video transmission has formulated the prerequisite copyright protection for these media. On the other hand, the development of Internet technologies has directed the accessibility of images, audio, and videos in different structures. Moreover, unlicensed users manipulate the utilization of multimedia by broadcasting them on numerous internet sites to get money fraudulently without the involvement of the original copyright holder. Besides, watermarking is a process utilized to hide the watermark sign present in the multimedia statistics that are not perceptible to an intruder for manipulating any information. This paper devised a novel video watermarking technique which uses an optimized deep learning technique. Here, the binary pixel map generation is performed using DCNN for self-embedding in input video. In addition, weights as well as bias of DCNN are trained by designed Fractional Gorilla Troops Optimization-Deep Convolutional Neural Network algorithm, thus FGTO_DCNN algorithm is developed. Accordingly, the FGTO technique is newly introduced by integrating FC along with GTO algorithm. Moreover, non-redundant Shift Invariant Complex Wavelet Transform (SICWT) is employed for extracted key frame and thus watermark image is fed into embedding process. After that, inverse non-redundant SICWT is applied for embedding video, which is further passed to key frame extraction in extraction stage. Finally, non-redundant SICWT is utilized for obtaining extracted watermark image. The developed FGTO-driven DCNN model attained improved performance than other existing video watermarking techniques with Mean Square Error (MSE) of 0.0437, Normalized Correlation Coefficient (NCC) of 0.9998, high Peak Signal-to-Noise Ratio (PSNR) value of 51.93 dB, and high Weighted Signal-to-Noise Ratio (WSNR) of 49.86 dB.

Key frame extraction method with global information balance

Key frame extraction method with global information balance

Novel error correction-based key frame extraction technique for dynamic hand gesture recognition

Novel error correction-based key frame extraction technique for dynamic hand gesture recognition

Key-Frame Extraction for Reducing Human Effort in Object Detection Training for Video Surveillance

This paper presents a supervised learning scheme that employs key-frame extraction to enhance the performance of pre-trained deep learning models for object detection in surveillance videos. Developing supervised deep learning models requires a significant amount of annotated video frames as training data, which demands substantial human effort for preparation. Key frames, which encompass frames containing false negative or false positive objects, can introduce diversity into the training data and contribute to model improvements. Our proposed approach focuses on detecting false negatives by leveraging the motion information within video frames that contain the detected object region. Key-frame extraction significantly reduces the human effort involved in video frame extraction. We employ interactive labeling to annotate false negative video frames with accurate bounding boxes and labels. These annotated frames are then integrated with the existing training data to create a comprehensive training dataset for subsequent training cycles. Repeating the training cycles gradually improves the object detection performance of deep learning models to monitor a new environment. Experiment results demonstrate that the proposed learning approach improves the performance of the object detection model in a new operating environment, increasing the mean average precision (mAP@0.5) from 54% to 98%. Manual annotation of key frames is reduced by 81% through the proposed key-frame extraction method.

Direct-Optimization-Based DC Dictionary Learning With the MCP Regularizer.

Direct-optimization-based dictionary learning has attracted increasing attention for improving computational efficiency. However, the existing direct optimization scheme can only be applied to limited dictionary learning problems, and it remains an open problem to prove that the whole sequence obtained by the algorithm converges to a critical point of the objective function. In this article, we propose a novel direct-optimization-based dictionary learning algorithm using the minimax concave penalty (MCP) as a sparsity regularizer that can enforce strong sparsity and obtain accurate estimation. For solving the corresponding optimization problem, we first decompose the nonconvex MCP into two convex components. Then, we employ the difference of the convex functions algorithm and the nonconvex proximal-splitting algorithm to process the resulting subproblems. Thus, the direct optimization approach can be extended to a broader class of dictionary learning problems, even if the sparsity regularizer is nonconvex. In addition, the convergence guarantee for the proposed algorithm can be theoretically proven. Our numerical simulations demonstrate that the proposed algorithm has good convergence performances in different cases and robust dictionary-recovery capabilities. When applied to sparse approximations, the proposed approach can obtain sparser and less error estimation than the different sparsity regularizers in existing methods. In addition, the proposed algorithm has robustness in image denoising and key-frame extraction.

Sign Language Recognition using Deep CNN with Normalised Keyframe Extraction and Prediction using LSTM

Sign Language Recognition using Deep CNN with Normalised Keyframe Extraction and Prediction using LSTM

Key Frame Extraction Based on Real-Time Person Availability Using YOLO

Keyframe extraction plays a crucial role in summarizing lengthy videos, particularly in the context of surveillance footage with a fixed field of view that records events over extended periods. The process of manually reviewing such videos can be time-consuming and challenging to extract essential information effectively. To address this issue, a study was conducted to evaluate four distinct methods for keyframe extraction, with the aim of determining the most suitable approach for creating a people-based dataset. The four methods assessed in the study were absolute difference, entropy, optical flow, and object detection-based video summarization using the YOLO (You Only Look Once) algorithm. Each method offers a unique approach to identify keyframes that encapsulate critical instances within the video footage. Among all the four evaluated methods, the object detection-based video summarization approach stood out as particularly promising. This method employed the YOLO algorithm, which utilizes advanced object detection techniques to identify and track people within the video frames. With this approach only fewest numbers of frames are extracted but still capturing all the relevant instances featuring people. The results of this study suggest that object detection-based video summarization using the YOLO algorithm is a highly effective method for keyframe extraction in surveillance videos. By significantly reducing the number of frames while preserving all relevant instances, this approach offers a time-efficient solution for reviewing and analyzing extensive video footage, ultimately facilitating the creation of a people-based dataset for further research and applications in various domains.