Sequence Of Key Frames Research Articles

A Binocular Camera Calibration Method in Froth Flotation Based on Key Frame Sequences and Weighted Normalized Tilt Difference

3D froth information is a significant indicator of working condition recognition in froth flotation. To extract it accurately, the first and key step is camera calibration. Although there have been many camera calibration methods proposed in these years, they cannot handle well for binocular camera calibration in froth flotation. The main reason is that there is an open environment without support surface or special geometric shapes for camera calibration in froth flotation. Therefore, we propose a novel binocular camera calibration method based on key frame sequences and weighted normalized tilt difference. First, we extract a high-quality frame sequence by filtering out incomplete and out-of-focus images from the froth video. Then, we propose a method to measure the tilt degree of the planar plate and extract the key frame sequences of the planar plate based on the tilt difference. After that, we calculate the camera parameters by weighted normalized tilt difference under various poses. The proposed binocular camera calibration method uses the froth video to obtain accurate and robust camera parameters, and does not require auxiliary equipment. Experiments have demonstrated the effectiveness and flexibility of the proposed method in froth flotation.

Human Action Recognition Using Key-Frame Attention-Based LSTM Networks

Human action recognition is a classical problem in computer vision and machine learning, and the task of effectively and efficiently recognising human actions is a concern for researchers. In this paper, we propose a key-frame-based approach to human action recognition. First, we designed a key-frame attention-based LSTM network (KF-LSTM) using the attention mechanism, which can be combined with LSTM to effectively recognise human action sequences by assigning different weight scale values to give more attention to key frames. In addition, we designed a new key-frame extraction method by combining an automatic segmentation model based on the autoregressive moving average (ARMA) algorithm and the K-means clustering algorithm. This method effectively avoids the possibility of inter-frame confusion in the temporal sequence of key frames of different actions and ensures that the subsequent human action recognition task proceeds smoothly. The dataset used in the experiments was acquired with an IMU sensor-based motion capture device, and we separately extracted the motion features of each joint using a manual method and then performed collective inference.

Machine Vision based Unsupervised Summarization of Wireless Capsule Endoscopy Video

Wireless capsule endoscopy is a medical procedure that uses a small, pill-sized capsule with a tiny camera inside to examine the inside of the digestive system. The capsule contains a camera that takes pictures as it moves through the digestive tract. The images are transmitted to a small recording device that the patient wears on a belt or shoulder strap. The recording device stores the images, which are later downloaded and examined by a doctor. The pill-sized camera records a video for eight to twelve hours. The purpose of this work is to investigate and put into practise a technique for summarising the capsule endoscopy footage. The method clusters the video frames in order to accomplish this by looking for local maxima in the neighbouring distance graph that is generated from the colour histograms of each video frame. The method then determines the centroid frame for each cluster and produces a key-frame for each cluster. Visualising the conclusion of this investigation is its second objective. This is accomplished by placing each cluster on a timeline according to the size and calibre of the keyframe. The summary of the capsule endoscopy video is generated as a sequence of keyframes. These keyframes are representative frames of the video. The work also proposes a technique to assess the quality and the quantity of the keyframes extracted.

Adaptive Weighted Flow Net Algorithm for Human Activity Recognition Using Depth Learned Features

Human Activity Recognition (HAR) from video data collections is the core application in vision tasks and has a variety of utilizations including object detection applications, video-based behavior monitoring, video classification, and indexing, patient monitoring, robotics, and behavior analysis. Although many techniques are available for HAR in video analysis tasks, most of them are not focusing on behavioral analysis. Hence, a new HAR system analysis the behavioral activity of a person based on the deep learning approach proposed in this work. The most essential aim of this work is to recognize the complex activities that are useful in many tasks that are based on object detection, modelling of individual frame characteristics, and communication among them. Moreover, this work focuses on finding out the human actions from various video resolutions, invariant human poses, and nearness of multi objects. First, we identify the key and essential frames of each activity using histogram differences. Secondly, Discrete Wavelet Transform (DWT) is used in this system to extract coefficients from the sequence of key-frames where the activity is localized in space. Finally, an Adaptive Weighted Flow Net (AWFN) algorithm is proposed in this work for effective video activity recognition. Moreover, the proposed algorithm has been evaluated by comparing it with the existing Visual Geometry Group (VGG-16) convolution neural networks for making performance comparisons. This work focuses on competent deep learning-based feature extraction to discriminate the activities for performing the classification accuracy. The proposed model has been evaluated with VGG-16 using a combination of regular UCF-101 activity datasets and also in very challenging Low-quality videos such as HMDB51. From these investigations, it is proved that the proposed AWFN approach gives higher detection accuracy of 96%. It is approximately 0.3% to 7.88% of higher accuracy than state-of-art methods.

Generating Upper‐Body Motion for Real‐Time Characters Making their Way through Dynamic Environments

AbstractReal‐time character animation in dynamic environments requires the generation of plausible upper‐body movements regardless of the nature of the environment, including non‐rigid obstacles such as vegetation. We propose a flexible model for upper‐body interactions, based on the anticipation of the character's surroundings, and on antagonistic controllers to adapt the amount of muscular stiffness and response time to better deal with obstacles. Our solution relies on a hybrid method for character animation that couples a keyframe sequence with kinematic constraints and lightweight physics. The dynamic response of the character's upper‐limbs leverages antagonistic controllers, allowing us to tune tension/relaxation in the upper‐body without diverging from the reference keyframe motion. A new sight model, controlled by procedural rules, enables high‐level authoring of the way the character generates interactions by adapting its stiffness and reaction time. As results show, our real‐time method offers precise and explicit control over the character's behavior and style, while seamlessly adapting to new situations. Our model is therefore well suited for gaming applications.

Extracting keyframes of breast ultrasound video using deep reinforcement learning.

Ultrasound (US) plays a vital role in breast cancer screening, especially for women with dense breasts. Common practice requires a sonographer to recognize key diagnostic features of a lesion and record a single or several representative frames during the dynamic scanning before performing the diagnosis. However, existing computer-aided diagnosis tools often focus on the final diagnosis process while neglecting the influence of the keyframe selection. Moreover, the lesions could have highly-irregular shapes, varying sizes, and locations during the scanning. The recognition of diagnostic characteristics associated with the lesions is challenging and also faces severe class imbalance. To address these, we proposed a reinforcement learning-based framework that can automatically extract keyframes from breast US videos of unfixed length. It is equipped with a detection-based nodule filtering module and a novel reward mechanism that can integrate anatomical and diagnostic features of the lesions into keyframe searching. A simple yet effective loss function was also designed to alleviate the class imbalance issue. Extensive experiments illustrate that the proposed framework can benefit from both innovations and is able to generate representative keyframe sequences in various screening conditions.

Two-Stage Frame Extraction in Video Analysis for Accurate Prediction of Object Tracking by Improved Deep Learning

Recently, the information extraction from graphics and video summarizing using keyframes have benefited from a recent look at the visual content-based method. Analysis of keyframes in a movie may be done by extracting visual elements from the video clips. In order to accurately anticipate the path of an item in real-time, the visible components are utilized. The frame variations with low-level properties such as color and structure are the basis of the rapid and reliable approach. This research work contains 3 phases: preprocessing, two-stage extraction, and video prediction module. Besides, this framework on object track estimation uses the probabilistic deterministic process to arrive at an estimate of the object. Keyframes for the whole video sequence are extracted using a proposed two-stage feature extraction approach by CNN feature extraction. An alternate sequence is first constructed by comparing the color characteristics of neighboring frames in the original series to those of the generated one. When an alternate arrangement is compared to the final keyframe sequence, it is found that there are substantial structural changes between consecutive frames. Three keyframe extraction techniques based on on-time behavior have been employed in this study. A keyframe extraction optimization phase termed as "Adam" optimizer, dependent on the number of final keyframes is then introduced. The proposed technique outperforms the prior methods in computational cost and resilience across a wide range of video formats, video resolutions, and other parameters. Finally, this research compares SSIM, MAE, and RMSE performance metrics with the traditional approach.

A Hybrid Deep Learning Model for Recognizing Actions of Distracted Drivers.

With the rapid spreading of in-vehicle information systems such as smartphones, navigation systems, and radios, the number of traffic accidents caused by driver distractions shows an increasing trend. Timely identification and warning are deemed to be crucial for distracted driving and the establishment of driver assistance systems is of great value. However, almost all research on the recognition of the driver’s distracted actions using computer vision methods neglected the importance of temporal information for action recognition. This paper proposes a hybrid deep learning model for recognizing the actions of distracted drivers. Specifically, we used OpenPose to obtain skeleton information of the human body and then constructed the vector angle and modulus ratio of the human body structure as features to describe the driver’s actions, thereby realizing the fusion of deep network features and artificial features, which improve the information density of spatial features. The K-means clustering algorithm was used to preselect the original frames, and the method of inter-frame comparison was used to obtain the final keyframe sequence by comparing the Euclidean distance between manually constructed vectors representing frames and the vector representing the cluster center. Finally, we constructed a two-layer long short-term memory neural network to obtain more effective spatiotemporal features, and one softmax layer to identify the distracted driver’s action. The experimental results based on the collected dataset prove the effectiveness of this framework, and it can provide a theoretical basis for the establishment of vehicle distraction warning systems.

A Temporal Directed Graph Convolution Network for Traffic Forecasting Using Taxi Trajectory Data

Traffic forecasting plays a vital role in intelligent transportation systems and is of great significance for traffic management. The main issue of traffic forecasting is how to model spatial and temporal dependence. Current state-of-the-art methods tend to apply deep learning models; these methods are unexplainable and ignore the a priori characteristics of traffic flow. To address these issues, a temporal directed graph convolution network (T-DGCN) is proposed. A directed graph is first constructed to model the movement characteristics of vehicles, and based on this, a directed graph convolution operator is used to capture spatial dependence. For temporal dependence, we couple a keyframe sequence and transformer to learn the tendencies and periodicities of traffic flow. Using a real-world dataset, we confirm the superior performance of the T-DGCN through comparative experiments. Moreover, a detailed discussion is presented to provide the path of reasoning from the data to the model design to the conclusions.

Interp-SUM: Unsupervised Video Summarization with Piecewise Linear Interpolation.

This paper addresses the problem of unsupervised video summarization. Video summarization helps people browse large-scale videos easily with a summary from the selected frames of the video. In this paper, we propose an unsupervised video summarization method with piecewise linear interpolation (Interp-SUM). Our method aims to improve summarization performance and generate a natural sequence of keyframes with predicting importance scores of each frame utilizing the interpolation method. To train the video summarization network, we exploit a reinforcement learning-based framework with an explicit reward function. We employ the objective function of the exploring under-appreciated reward method for training efficiently. In addition, we present a modified reconstruction loss to promote the representativeness of the summary. We evaluate the proposed method on two datasets, SumMe and TVSum. The experimental result showed that Interp-SUM generates the most natural sequence of summary frames than any other the state-of-the-art methods. In addition, Interp-SUM still showed comparable performance with the state-of-art research on unsupervised video summarization methods, which is shown and analyzed in the experiments of this paper.

Multifeature fusion action recognition based on key frames

SummaryAs an important technology in computer vision, video‐based human action recognition has a great commercial value, which has attracted extensive attention in the field of computer vision and pattern recognition in both academia and industry. To date, there are a wide variety of applications of human action recognition, such as surveillance, robotics, health care, video searching, and human–computer interaction. However, there are many challenges involved in human action recognition in videos, such as cluttered backgrounds, occlusions, viewpoint variation, execution rate, and camera motion. However, data redundancy and single feature were largely limited the accuracy of human action recognition. In this article, adopting the key frame extraction and multifeature fusion techniques, a novel action recognition method was proposed, which can improve the recognition accuracy. The main works are as follows: 1) in order to solve the problem of data redundancy, a key frame extraction method based on node contribution weighting is proposed to extract video key frames; 2) different kinds of information flows are extracted from the obtained key frame sequences, and different convolutional neural networks are used to obtain corresponding classification results and merge, so as to better complement the information in different flows. Lastly, the experimental results show that our method improves the accuracy of action recognition.

No-Reference Quality Evaluation of Stereoscopic Video Based on Spatio-Temporal Texture

Due to the wide application of stereoscopic display technology, stereoscopic video quality assessment (SVQA) is facing great challenges, but worthwhile. Stereoscopic videos contain a great deal of information, which involves not only the spatial domain but also the spatio-temporal domain. Motion in stereoscopic video plays a critical role in quality perception, while the existing SVQA methods rarely refer to motion factors, and the performance of these methods is restrained. In this article, a novel SVQA based on motion perception is introduced and its performance is superior to that of existing excellent methods. Particularly, to appropriately reduce the amount of data processing, we extract the key-frame sequences according to the influence of movement intensity on binocular visual quality perception. The binocular summation and difference operations are implemented on extracted sequences, and then spatial texture and spatio-temporal texture statistic measurement are extracted simultaneously with local binary patterns from three orthogonal planes (LBP-TOP). Experiments are implemented on two publicly available databases and the results demonstrate the effectiveness and robustness of our algorithm for various categories of distortion stereoscopic video pairs.

A New Automatic Visual Scene Segmentation Algorithm for Flash Movie

Flash movie retrieval system can improve the utilization of Flash movie on the Internet. The key step of a content-based Flash movie retrieval system is visual scene segmentation that directly affects the retrieval effect. In this paper, an adaptive threshold method for visual scene segmentation based on frame difference of color histogram is proposed. Firstly, all key frame sequences of a Flash movie are obtained; then the region-weighted color histogram difference of adjacent key frames is calculated; lastly, the visual scene is classified by comparing the result with the average difference. In the process of visual scene segmentation, the spatial characteristics of color are considered and the regional weighting coefficient of key frames is determined by comparing the experiments. The proposed algorithm replaces the traditional fixed global threshold with a variable adaptive threshold. The experiments show that the proposed algorithm has a better detection effect than the fixed threshold algorithm. This algorithm can easily be implemented with moderate computational complexity. The proposed algorithm can be used to extract visual features of the visual scene, generate dynamic summary, and finally can be applied to content-based Flash animation retrieval system. Moreover, the proposed algorithm can also be used in non-flash applications.

A kinetic energy model based on machine vision for recognition of aggressive behaviours among group-housed pigs

The aim of this study is to build a kinetic energy model to recognise aggressive behaviours in pigs. Pig mixing experiments were performed 10 times. In each experiment, 7 pigs from 3 pens were mixed and then videos of the first 3 h after mixing and the 3 h after 24 h were selected as experimental data. A 30 h data of 5 groups of pigs were used as training set and a 30 h data of another 5 groups of pigs as test set. Firstly, according to connected area and adhesion index, aggressive pigs were separated from the herd and then key frame sequences with aggression were extracted. Secondly, head and 4 kink points of pigs were selected as feature points. By analysing motion of these points, kinetic energy of 2 aggressive pigs was calculated and kinetic energy difference between adjacent frames was selected as features. Finally, these features were trained by using hierarchical clustering to obtain the thresholds of high and medium aggression. These thresholds were used to design the rule of aggression recognition. By using the proposed algorithm, high aggression could be recognised with an accuracy of 95.8%, a sensitivity of 92.8%, specificity of 96.7% and precision of 90.2%, and medium aggression could be recognised with an accuracy of 92.3%, a sensitivity of 92.6%, specificity of 91.9% and precision of 93.0%. The results indicate that this algorithm can be used to recognise aggressive behaviours of pigs.

Dynamic Hand Gesture Tracking Method Based on Key Frame and Video Perceptual Hashing

Aiming at the common problems such as low tracking accuracy, robustness and poor real-time ability for the traditional tracking algorithm, a novel dynamic hand gesture tracking method based on key frame and video perceptual hashing has been proposed in this paper. Firstly, the proposed method is used to frame the video sequence and extract the key frames, in order to obtain a sequence of key frames. Secondly, the video perceptual hashing is used to extract the perceptual feature of the key frame and target template, thus the hash sequence is generated. Finally, the hamming distance is used to compare the similarities between the key frame and target template, therefore, the real-time tracking is carried out by using an online adaptive template update method to ensure the continuity of the hand gesture tracking. Experimental results show that the proposed method can realize the accurate and real-time tracking of the dynamic deformation gestures, and effectively reduce the time complexity of the algorithm, as well as ensure the stability and continuity of the moving target tracking.

Near-duplicate Video Detection Algorithm Based on Global GSP Feature and Local ScSIFT Feature Fusion

The main problem with near-duplicate video detection is the high computational complexity and the low efficiency. Near-duplicate video detection methods based on global feature is running fast but with low accuracy, on the contrary, methods based on local feature is accurate, but the calculation is large and time-consuming. Therefore, a near-duplicate video detection algorithm combining global GSP feature and local ScSIFT feature is proposed. Firstly, the video clips and the query ones are discretized into a set of key frame sequences, and the temporal information is recorded at the same time. Secondly, by filtering the video clips on global Gaussian-Scale pyramid feature, similar video clips are selected as the candidates to assure the high recall. Then, combined with the temporal features of the keyframes, the candidate videos are further detected by the local ScSIFT feature to obtain a higher precision. Experimental results show that the proposed algorithm can improve the accuracy of near-duplicate video detection on the basis of guaranteeing the timeliness of the algorithm.

Image motion feature extraction for recognition of aggressive behaviors among group-housed pigs

The aim of this study is to develop a computer vision-based method to automatically detect aggressive behaviors among pigs. Ten repetitions of the same experiment were performed. In each of the experiment, 7 piglets were mixed from three litters in two pigpens and captured on video for a total of 6h. From these videos, the first 3h of video after mixing were recorded asa training set, and the 3h of video after 24h were recorded asa validation set. Connected area and adhesion index were used to locate aggressive pigs and to extract key frame sequences. The two pigs in aggression were regarded asa whole rectangle according to their characteristics of continuous and large-proportion adhesion. The acceleration feature was extracted by analyzing the displacement change of four sides of this rectangle between adjacent frames, and hierarchical clustering was used to calculate its threshold. Based on this feature, the recognition rules of medium and high aggression were designed. Testing 10 groups of pigs, the accuracy of recognizing medium aggression was 95.82% with a sensitivity of 90.57% and with a specificity of 96.95%, and the accuracy of recognizing high aggression was 97.04% with a sensitivity of 92.54% and with a specificity of 97.38%. These results indicate that the acceleration can be used to recognize pigs’ aggressive behaviors.

A Fast and Robust Key Frame Extraction Method for Video Copyright Protection

The paper proposes a key frame extraction method for video copyright protection. The fast and robust method is based on frame difference with low level features, including color feature and structure feature. A two-stage method is used to extract accurate key frames to cover the content for the whole video sequence. Firstly, an alternative sequence is got based on color characteristic difference between adjacent frames from original sequence. Secondly, by analyzing structural characteristic difference between adjacent frames from the alternative sequence, the final key frame sequence is obtained. And then, an optimization step is added based on the number of final key frames in order to ensure the effectiveness of key frame extraction. Compared with the previous methods, the proposed method has advantage in computation complexity and robustness on several video formats, video resolution, and so on.

Energy-efficient medium access approach for wireless body area network based on body posture

Energy efficiency has become one of the most important issues in wireless body area network (WBAN). In this paper, an energy-efficient medium access control (MAC) protocol for WBAN is proposed based on human body posture under walking scenery. Due to person's movements, WBAN is a dynamic network, which means that traditional static protocols are no more suitable for it. For solving this problem, firstly, the feature of human walking at a constant speed is analyzed and we divide a spell of movements into a sequence of key frames just like a video constituted by numbers of continuous frames. As a result, the dynamic walking process is translated into several static postures, which the static MAC protocol could be used for. Secondly, concerning the performance of network lifetime, we design a posture-aware approach for lifetime maximization (PA-DPLM). With analytical and simulation results provided, we demonstrate that PA-DPLM protocol is energy-efficient and can be used under constant speed walking scenery.

Key Frame Extraction from Video

A complete overview of key frame extraction techniques has been provided. It has been found out that such techniques usually have three phases, namely shot boundary detection as a pre-processing phase, main phase of key frame detection, where visual, structural, audio and textual features are extracted from each frame, then processed and analyzed with artificial intelligence methods, and the last post-processing phase lies in removal of duplicates if they occur in the resulting sequence of key frames. Estimation techniques and available test video collections have been also observed. At the end, conclusions concerning drawbacks of the examined procedure and basic tendencies of its development have been marked.