Number Of Key Frames Research Articles

An optimization method of human skeleton keyframes selection for action recognition

In the action recognition field based on the characteristics of human skeleton joint points, the selection of keyframes in the skeleton sequence is a significant issue, which directly affects the action recognition accuracy. In order to improve the effectiveness of keyframes selection, this paper proposes inflection point frames, and transforms keyframes selection into a multi-objective optimization problem based on it. First, the pose features are extracted from the input skeleton joint point data, which used to construct the pose feature vector of each frame in time sequence; then, the inflection point frames in the sequence are determined according to the flow of momentum of each body part. Next, the pose feature vectors are input into the keyframes multi-objective optimization model, with the fusion of domain information and the number of keyframes; finally, the output keyframes are input to the action classifier. To verify the effectiveness of the method, the MSR-Action3D, the UTKinect-Action and Florence3D-Action, and the 3 public datasets, are chosen for simulation experiments and the results show that the keyframes sequence obtained by this method can significantly improve the accuracy of multiple action classifiers, and the average recognition accuracy of the three data sets can reach 94.6%, 97.6% and 94.2% respectively. Besides, combining the optimized keyframes with deep learning classifier on the NTU RGB + D dataset can make the accuracies reaching 83.2% and 93.7%.

Analysis in Making Animation Movements Using Traditional Methods (Keyframe) and Motion Capture Methods

The keyframe method for making 3D animation movements is a traditional technique used by animators around the world to create certain animation movements based zon the 12 principles of animation. Meanwhile, motion capture is a motion recording technique used to describe the process of recording movements and interpreting these movements into a digital model. Motion capture itself is commonly used in the military, entertainment, sports, and other fields. In this study, we analyze the use of the two methods, the results of which can be used as a suggestion for the use of the two methods for the production of animated films. Three aspects were used to analyze the two methods, namely the number of keyframes, the level of refinement and detail, the final process and cleanup.
 Keywords: keyframe, motion capture

Deep Unsupervised Key Frame Extraction for Efficient Video Classification

Video processing and analysis have become an urgent task, as a huge amount of videos (e.g., YouTube, Hulu) are uploaded online every day. The extraction of representative key frames from videos is important in video processing and analysis since it greatly reduces computing resources and time. Although great progress has been made recently, large-scale video classification remains an open problem, as the existing methods have not well balanced the performance and efficiency simultaneously. To tackle this problem, this work presents an unsupervised method to retrieve the key frames, which combines the convolutional neural network and temporal segment density peaks clustering. The proposed temporal segment density peaks clustering is a generic and powerful framework, and it has two advantages compared with previous works. One is that it can calculate the number of key frames automatically. The other is that it can preserve the temporal information of the video. Thus, it improves the efficiency of video classification. Furthermore, a long short-term memory network is added on the top of the convolutional neural network to further elevate the performance of classification. Moreover, a weight fusion strategy of different input networks is presented to boost performance. By optimizing both video classification and key frame extraction simultaneously, we achieve better classification performance and higher efficiency. We evaluate our method on two popular datasets (i.e., HMDB51 and UCF101), and the experimental results consistently demonstrate that our strategy achieves competitive performance and efficiency compared with the state-of-the-art approaches.

Compare Between Histogram Similarity and Histogram Differencing For More Brief Key Frames Extraction from Video Stream

Speedy increase in the employ of the content of multimedia is very commonly perceived in existing generations. Video is one of the extreme exceedingly recovered data of multimedia which can provide quick fixes and immediate gratification for whole the kinds of user information willingness. Detection of Key frame from Videos and classification of videos are the necessary protocols in the complements tasks of video retrieval. The aim of this research is to compare between histogram similarity and histogram differencing for more brief key frames extraction from video stream. The suggested system for key frame extraction has three steps. Firstly, the video frames series will take and the characteristics for points of interest are elicited utilizing SUSAN detector. After eliciting interest characteristics from all video frames images, secondly, K-Means clustering technique was utilized for these features to construct the clusters with a number of interest points. Thirdly, a histogram builds for each video frame based on numbers of features in each cluster. X-axis of a histogram represents the cluster number and y-axis represents the number of features in each cluster. For key frames extraction, numbers for each cluster can be offered and a query histogram can be constructed based on entered clusters’ numbers. A query histogram was matched with every video frame histogram using Manhattan distance to discover the similar histogram to query histogram. After chosen similar histogram, it can extract all key frames from that video frames. The experimental results show that the number of key frames extracted from the video is very brief when using the histogram similarity compared to the histogram differencing. For example, the number of key frames extracted from a car video is 70 when using the histogram similarity, while it is 180 when using the histogram differencing.

Research on improving LIO-SAM based on Intensity Scan Context

During indoor mobile mapping, localization plays an important role. It is difficult for IMU, odometer, Lidar or SLAM algorithm singly to meet the high efficiency, real-time, accurate and robust performance. So the fusion of the data measured by these different methods is a current research hotspot. However, most of the researches still focus on the loosely coupled fusion based on filtering methods, and the data cannot be fully utilized. In this paper, based on LOM-SAM framework, the Intensity Scan Context is introduced to extract keyframes and detect loop closure which will provide a closed-loop factor, together with IMU pre-integration factor and Lidar odometry factor to construct the factor graph. Then incremental smoothing optimization algorithm is used to get high precision trajectory, realize the tightly-coupled Lidar and IMU positioning. The results show that the number of keyframes is reduced, the elevation error is effectively decreased, and the real-time performance is improved without decreasing the accuracy of LIO-SAM.

The adaptive key frames extraction method based on representativeness and independence features

For the existing traditional method to extract key frames from the video sequence with gradual change of content, there are problems of having too much redundancy or too much loss in the extraction results and other issues, a new key frames extracting method for adaptively extracting the video key frames has proposed in the paper. The experimental results have shown that the proposed method can effectively extract video key frames, especially for the video with gradual change of content, which can express more succinctly and comprehensively. The experiments on a large number of videos containing various scenes have shown that the key frames extracted by the proposed algorithm can more fully express the main content of video. When dealing with content gradient video, the effect is better. The content adaptively determines the number of key frames based on video.

The adaptive key frames extraction method based on representativeness and independence features

For the existing traditional method to extract key frames from the video sequence with gradual change of content, there are problems of having too much redundancy or too much loss in the extraction results and other issues, a new key frames extracting method for adaptively extracting the video key frames has proposed in the paper. The experimental results have shown that the proposed method can effectively extract video key frames, especially for the video with gradual change of content, which can express more succinctly and comprehensively. The experiments on a large number of videos containing various scenes have shown that the key frames extracted by the proposed algorithm can more fully express the main content of video. When dealing with content gradient video, the effect is better. The content adaptively determines the number of key frames based on video.

Endoscope localization and gastrointestinal feature map construction based on monocular SLAM technology

Aiming at the real intestinal environment, the commonly used feature extraction algorithms are compared and analyzed. The performances of SIFT, BRISK, ORB and FREAK are evaluated from three aspects: feature extraction time, correct matching rate and attitude estimation accuracy under rotation and scaling transformation. ORB binary feature operator has the advantages of fast feature extraction, high correct matching rate, and small estimation error. Based on the measured data of gastrointestinal endoscopy, the classical SLAM algorithm is modified in the aspects of pose adjustment and spatial point location calculation. A framework of gastrointestinal SLAM (Simultaneous Localization and Mapping) algorithm is constructed by introducing the local pose optimization algorithm based on visual correlation graph and triangular measurement algorithm with the minimum geometric distance. In order to tackle the problem of a large amount of accumulated data of spatial points and keyframes, a keyframe judgment strategy and spatial map point deletion strategy is designed to control the growth of keyframes and map points. The feature map of the human colon is reconstructed based on endoscopic video data, and the trajectory of the endoscope is estimated, which verifies the feasibility of the algorithm in the actual intestinal environment. In order to get the real value of map and trajectory conveniently, a simulated gastrointestinal system is built to validate the effectiveness of the algorithm, and the effects of the image frame rate and a number of keyframes on positioning accuracy and feature map accuracy are evaluated. Based on the simulated gastrointestinal system, the SLAM algorithm of the gastrointestinal tract is compared with the traditional visual mileage calculation method. When the endoscope moves in a straight line, keyframe storage of the proposed algorithm is only 1/8 of that of the latter, and the positioning accuracy is improved three times.

Compressing animated meshes with fine details using local spectral analysis and deformation transfer

Geometry-centric shape animation, usually represented as dynamic meshes with fixed connectivity and time-deforming geometry, is becoming ubiquitous in digital entertainment and other relevant graphics applications. However, digital animation with fine details, which requires more diversity of texture on meshed geometry, always consumes a significant amount of storage space, and compactly storing and efficiently transmitting these meshes still remain technically challenging. In this paper, we propose a novel key-frame-based dynamic meshes compression method, wherein we decompose the meshes into the low-frequency and high-frequency parts by applying piece-wise manifold harmonic bases to reduce spatial-temporal redundancy of primary poses and by using deformation transfer to recover high-frequency details. First of all, we partition the animated meshes into several clusters with similar poses, and the primary poses of meshes in each cluster can be characterized as a linear combination of manifold harmonic bases derived from the key-frame of that cluster. Second, we recover the geometric details on each primary pose using the deformation transfer technique which reconstructs the details from the key-frames. Thus, we only need to store a very small number of key-frames and a few harmonic coefficients for compressing time-varying meshes, which would reduce a significant amount of storage in contrast with traditional methods where bases were stored explicitly. Finally, we employ the state-of-the-art static mesh compression method to store the key-frames and apply a second-order linear prediction coding to the harmonics coefficients to further reduce the spatial-temporal redundancy. Our comprehensive experiments and thorough evaluations on various datasets have manifested that, our novel method could obtain a high compression ratio while preserving high-fidelity geometry details and guaranteeing limited human perceived distortion rate simultaneously, as quantitatively characterized by the popular Karni–Gotsman error and our newly devised local rigidity error metrics.

Sign Language Recognition based on Key Frame

This paper presents a new real-time recognition system as for Chinese sign language. In order to decrease massive unnecessary data, including the body of model, the differential method is used to extract the arms of model. The sign language videos are stored in the different groups, where the number of key frames is same, to decrease matching process. Hu invariant moments models are implying to extract the image’s feature and obtain the key frames.

Classification and comparison of on-line video summarisation methods

Many methods exist for generating keyframe summaries of videos. However, relatively few methods consider on-line summarisation, where memory constraints mean it is not practical to wait for the full video to be available for processing. We propose a classification (taxonomy) for on-line video summarisation methods based upon their descriptive and distinguishing properties such as feature space for frame representation, strategies for grouping time-contiguous frames, and techniques for selecting representative frames. Nine existing on-line methods are presented within the terms of our taxonomy and subsequently compared by testing on two synthetic data sets and a collection of short videos. We find that success of the methods is largely independent of techniques for grouping time-contiguous frames and for measuring similarity between frames. On the other hand, decisions about the number of keyframes and the selection mechanism may substantially affect the quality of the summary. Finally, we remark on the difficulty in tuning the parameters of the methods “on-the-fly”, without knowledge of the video duration, dynamic or content.

Estimating Audience Engagement to Predict Movie Ratings

While watching movies, audience members exhibit both subtle and coarse gestures (e.g., smiles, head-pose change, fidgeting, stretching) which convey sentiment (i.e., engaged or disengaged) during feature length movies. Noticing these behaviors using computer vision systems is a very challenging problem—especially in a movie theatre environment. The environment is dark and contains views of people at different scales and viewpoints. Feature length movies typically run 80-120 minutes, and tracking people uninterrupted for this duration is still an unsolved problem. Facial expressions of audience members are subtle, short, and sparse; making it difficult to detect and recognize activities. Finally, annotating audience sentiment at the frame-level is prohibitively time consuming. To circumvent these issues, we use an infrared illuminated test-bed to obtain a visually uniform input of audiences watching feature length movies. We present a method which can automatically detect the change in behavior (key-gestures) using “key-frames”, which can convey audience sentiment. As the number of key-frames are many orders of magnitudes lower than the number of frames, the annotation problem is reduced to assigning a sentiment label for each key-frame. Using these discovered key-gestures, we create a movie rating classifier from crowd-sourced ratings and demonstrate its predictive capability. Our dataset consists of over 50 hours of audience behavior collected across 237 subjects.

Mossar: motion segmentation by using splitting and remerging strategies

This paper presents a novel approach for motion segmentation by using strategies of splitting and remerging. The presented approach, Mossar, hybridizes two existing ones to obtain their potential advantages while covering weaknesses: (1) velocity-based, one of the most widely used approaches that has fairly low accuracy but provides computational simplicity and (2) graph-based, a state-of-the-art approach that provides outstanding accuracy, yet bears high computational complexity and a burden in setting of thresholds. An initial set of key frames is generated by a velocity-based splitting process and then fed into a graph-based remerging process for refinement. We present mechanisms that improve key-frames capturing in the velocity-based approach as well as details on how the graph-based approach is modified and later applied to remerging. The proposed approach also allows users to interactively add or reduce the number of key frames to control segmentation hierarchy without the need to change threshold values and re-run segmentation, as usually done in existing approaches. Our experimental results show that the presented hybrid approach, compared to both velocity-based and graph-based, demonstrates superior performance in terms of accuracy and in comparison to graph-based, our approach has not only less complexity but also a lesser number of thresholds, the values of which can be much more simply determined.

A Clustering Algorithm for Key Frame Extraction Based on Density Peak

Aiming at the problem of video key frame extraction, a density peak clustering algorithm is proposed, which uses the HSV histogram to transform high-dimensional abstract video image data into quantifiable low-dimensional data, and reduces the computational complexity while capturing image features. On this basis, the density peak clustering algorithm is used to cluster these low-dimensional data and find the cluster centers. Combining the clustering results, the final key frames are obtained. A large number of key frame extraction experiments for different types of videos show that the algorithm can extract different number of key frames by combining video content, overcome the shortcoming of traditional key frame extraction algorithm which can only extract a fixed number of key frames, and the extracted key frames can represent the main content of video accurately.

A Meaningful Compact Key Frames Extraction in Complex Video Shots

Key frame extraction is an essential technique in the computer vision field. The extracted key frames should brief the salient events with an excellent feasibility, great efficiency, and with a high-level of robustness. Thus, it is not an easy problem to solve because it is attributed to many visual features. This paper intends to solve this problem by investigating the relationship between these features detection and the accuracy of key frames extraction techniques using TRIZ. An improved algorithm for key frame extraction was then proposed based on an accumulative optical flow with a self-adaptive threshold (AOF_ST) as recommended in TRIZ inventive principles. Several video shots including original and forgery videos with complex conditions are used to verify the experimental results. The comparison of our results with the-state-of-the-art algorithms results showed that the proposed extraction algorithm can accurately brief the videos and generated a meaningful compact count number of key frames. On top of that, our proposed algorithm achieves 124.4 and 31.4 for best and worst case in KTH dataset extracted key frames in terms of compression rate, while the the-state-of-the-art algorithms achieved 8.90 in the best case.

A 3D stroke based representation of sign language signs using key maximum curvature points and 3D chain codes

Sign Language is a visual spatial language used by deaf and dumb community to convey their thoughts and ideas with the help of hand gestures and facial expressions. This paper proposes a novel 3D stroke based representation of dynamic gestures of Sign Language Signs incorporating local as well as global motion information. The dynamic gesture trajectories are segmented into strokes or sub-units based on Key Maximum Curvature Points (KMCPs) of the trajectory. This new representation has helped us in uniquely representing the signs with fewer number of key frames. We extract 3D global features from global trajectories using a scheme of representing strokes as 3D codes, which involves dividing strokes into smaller units (stroke subsegment vectors or SSVs), and representing them as belonging to one of the 22 partitions. These partitions are obtained using a discretisation procedure which we call an equivolumetric partition (EVP) of sphere. The codes representing the strokes are referred to as an EVP code. In addition to global hand motion and local hand motion, facial expressions are also considered for non-manual signs to interpret the meaning of words completely. In contrast to existing methods, our method of stroke based representation has less expensive training phase since it only requires the training of key stroke features and stroke sequences of each word.

A robust incremental clustering-based facial feature tracking

Emerging significance of person-independent, emotion specific facial feature tracking has been actively tracked in the machine vision society for decades. Among distinct methods, the Constrained Local Model (CLM) has shown significant results in person-independent feature tracking. In this paper, we propose an automatic, efficient, and robust method for emotion specific facial feature detection and tracking from image sequences. A novel tracking system along with 17-point feature model on the frontal face region has also been proposed to facilitate the tracking of human basic facial expressions. The proposed feature tracking system keeps patch images and face shapes till certain number of key frames incorporating CLM-based tracker. After that, incremental patch and shape clustering algorithms is applied to build appearance model and structure model of similar patches and similar shapes respectively. The clusters in each model are built and updated incrementally and online, controlled by amount of facial muscle movement. The overall performance of the proposed Robust Incremental Clustering-based Facial Feature Tracking (RICFFT) is evaluated on the FGnet database and the Extended Cohn-Kanade (CK+) database. RICFFT demonstrates mean tracking accuracy of 97.45% and 96.64% for FGnet and CK+ database respectively. Also, RICFFT is more robust by minimizing average shape distortion error of 0.20% and 1.86% for FGnet and CK+ (apex frame) database, as compared with classic method CLM.

Editorial Issue 27.6

This issue contains four papers. In the first paper, Mingfei Wang, Jinyuan Jia, Ning Xie, and Chenxi Zhang, from Tongji University in Shanghai, China, aim to solve the challenging problems of choosing the data supplier for data-dispatching services in distributed virtual environments based on peer-to-peer networks. They propose a dynamic node-organizing mechanism that aims by applying the avatar's behavioral characteristics to the neighbor maintenance mechanism and scene data transmission. They have conducted extensive simulation experiments that simulate avatar behaviors in a popular online game. The results show that their proposed mechanism achieved a substantial alleviation of neighbor churn and reduced information exchange, which improves the transmission efficiency in DVEs. In the second paper, Masaki Oshita, from the Kyushu Institute of Technology, Japan, proposes an interactive character motion control interface that uses hands. Using their hands and fingers, the user can control a large number of degrees of freedom (DOFs) at the same time. The author applied principal component analysis (PCA) to a set of sample poses and assigned the extracted principal components to each DOF of the hands (such as the hand positions and finger bending/extending angles). The author developed methods for computing the feature vector, for applying PCA, and for pose and action synthesis. In addition, he introduced a pose transition method for performing a step motion when necessary to prevent foot sliding. He presents his experimental results and demonstrates the effectiveness of his interface. In the third paper, Wenwu Yang, Xun Wang, Wangbin Kou, Bailin Yang, and Guozheng Wang, from Zhejiang Gongshang University, Hangzhou, China, present a topology-aware method based on moving least squares (MLS) deformation approach for 2D characters. First, a Laplace equation is solved to obtain a set of weights, which are called as harmonic weights. Then, the MLS deformation is performed by using the harmonic weights as the deformation influence of the user-specified controls. Finally, the possible distortion in the traditional MLS deformation can be effectively avoided, since the harmonic weights spread the deformation of the controls in a localized and topology-aware way. In addition, a simple but effective area-preserving variant of MLS deformation is proposed, which is suitable for the editing of incompressible objects. Mohammadali Hajizadeh and Hossein Ebrahimnezhad, from Sahand University of Technology, Islamic Republic of Iran, propose a key frame-based technique for 3D dynamic mesh compression. First, key frames are extracted from the animated sequence. Extracted key frames are then linearly combined using blending weights to predict the vertex locations of the other frames. These blending weights play a key role in the proposed algorithm since the prediction performance and the required number of key frames are greatly dependent on these weights. They present a novel method in order to compute the optimum blending weight, which makes it possible to predict location of the vertices of the non-key frames with the minimum number of key frames. The residual prediction errors are finally quantized and encoded using Huffman coding and another heuristic method. Experimental results on different test sequences with various sizes, topologies, and geometries demonstrate the privileged performance of the proposed method compared to the previous techniques.

Scalable Video Event Retrieval by Visual State Binary Embedding

With the exponential increase of media data on the web, fast media retrieval is becoming a significant research topic in multimedia content analysis. Among the variety of techniques, learning binary embedding (hashing) functions is one of the most popular approaches that can achieve scalable information retrieval in large databases, and it is mainly used in the near-duplicate multimedia search. However, till now most hashing methods are specifically designed for near-duplicate retrieval at the visual level rather than the semantic level. In this paper, we propose a visual state binary embedding (VSBE) model to encode the video frames, which can preserve the essential semantic information in binary matrices, to facilitate fast video event retrieval in unconstrained cases. Compared with other video binary embedding models, one advantage of our proposed VSBE model is that it only needs a limited number of key frames from the training videos for hash function training, so the computational complexity is much lower in the training phase. At the same time, we apply the pairwise constraints generated from the visual states to sketch the local properties of the events at the semantic level, so accuracy is also ensured. We conducted extensive experiments on the challenging TRECVID MED dataset, and have proved the superiority of our proposed VSBE model.

An Automatic Three-Dimensional Scene Reconstruction System Using Crowdsourced Geo-Tagged Videos

Automatic 3-D scene reconstruction is a useful technique in modern intelligent systems. Scene reconstruction from video sequences requires a selection of representative video frames. Most previous works employ the content-based techniques to automatically extract key frames. These methods take no frame geo-information into account and may be computationally heavy. In this paper, we propose a new key frame selection scheme based on the video geographic cues. Currently, an increasing number of user-generated videos are collected, which is a trend driven by the popularity of smartphones. In addition, it is convenient to acquire and fuse various sensor data (e.g., the geo-spatial metadata) for creating the geo-tagged mobile videos. Nowadays, large repositories of media content are automatically geo-tagged. Our proposed technique utilizes these underlying geo-metadata to select the most representative frames. We first eliminate irrelevant frames in which the candidate 3-D object does not appear. Then, a fixed number of key frames are selected. The criterion is that the selected key frames can maximally cover the candidate 3-D object/scene. Comprehensive experiments demonstrate the high quality of the reconstructed 3-D objects. Moreover, the execution time is reduced by 90%.