Temporal Templates Research Articles

Object Tracking Algorithm Based on Integrated Multi-Scale Templates Guided by Judgment Mechanism

The object tracking algorithm TransT, based on Transformer, achieves significant improvements in accuracy and success rate by fusing the extracted features of convolutional neural networks with the structure of Transformer. However, when dealing with the deformation of the object’s appearance, the algorithm exhibits issues such as insufficient tracking accuracy and drift, which directly affect the stability of the algorithm. In order to overcome this problem, this paper demonstrates how to expand scene information at the scale level during the fusion process and, on this basis, achieve accurate recognition and positioning. The predicted results are promptly fed back to the subsequent tracking process, from which temporal templates are embedded. Starting from both location and time can effectively improve the adaptive ability of the tracking model. In the final experimental comparison results, the algorithm proposed in this paper can adapt well to the situation of object deformation, and the overall performance of the tracking model has also been improved.

Temporally Local Weighting-Based Phase-Locked Time-Shift Data Augmentation Method for Fast-Calibration SSVEP-BCI.

Various training-based spatial filtering methods have been proposed to decode steady-state visual evoked potentials (SSVEPs) efficiently. However, these methods require extensive calibration data to obtain valid spatial filters and temporal templates. The time-consuming data collection and calibration process would reduce the practicality of SSVEP-based brain-computer interfaces (BCIs). Therefore, we propose a temporally local weighting-based phase-locked time-shift (TLW-PLTS) data augmentation method to augment training data for calculating valid spatial filters and temporal templates. In this method, the sliding window strategy using the SSVEP response period as a time-shift step is to generate the augmented data, and the time filter which maximises the temporally local covariance between the original template signal and the sine-cosine reference signal is used to suppress the temporal noise in the augmented data. For the performance evaluation, the TLW-PLTS method was incorporated with state-of-the-art training-based spatial filtering methods to calculate classification accuracies and information transfer rates (ITRs) using three SSVEP datasets. Compared with state-of-the-art training-based spatial filtering methods and other data augmentation methods, the proposed TLW-PLTS method demonstrates superior decoding performance with fewer calibration data, which is promising for the development of fast-calibration BCIs.

A deep neural framework for real-time vehicular accident detection based on motion temporal templates

Vehicular accident prediction and detection has recently garnered curiosity and large amounts of attention in machine learning applications and related areas, due to its peculiar and fascinating application potentials in the development of Intelligent Transportation Systems (ITS) that play a pivotal role in the success of emerging smart cities. In this paper, we present a new vision-based framework for real-time vehicular accident prediction and detection, based on motion temporal templates and fuzzy time-slicing. The presented framework proceeds in a stepwise fashion, starting with automatically detecting moving objects (i.e., on-road vehicles or roadside pedestrians), followed by dynamically keep tracking of the detected moving objects based on temporal templates, clustering and supervised learning. Then, an extensive set of local features is extracted from the temporal templates of moving objects. Finally, an effective deep neural network (DNN) model is trained on the extracted features to detect abnormal vehicle behavioral patterns and thus predict an accident just before it occurs. The experiments on real-world vehicular accident videos demonstrate that the framework can yield mostly promising results by achieving a hit rate of 98.5% with a false alarm rate of 4.2% that compare very favorably to those from existing approaches, while still being able to deliver delay guarantees for realtime traffic monitoring and surveillance applications.

Human and action recognition using adaptive energy images

In this paper, we propose a new temporal template approach for action recognition and person identification based on motion sequence information in masked depth video streams obtained from RGB-D data. This new representation creates a membership function that models the change in motion based on the correlation between frames that occur during motion flow. The energy images created with this function emphasize the intervals of motion with more change, while the intervals with less change are suppressed. To understand the distinctive features, the obtained energy images by using the proposed function are given as input to the convolutional neural networks and different handcrafted classifiers. The proposed method was observed on the BodyLogin, NATOPS, and SBU Kinect datasets and compared with the existing temporal templates and recent methods. The results indicate that the proposed method provides both higher performance and better motion representation.

A novel ADHD classification method based on resting state temporal templates (RSTT) using spatiotemporal attention auto-encoder

A novel ADHD classification method based on resting state temporal templates (RSTT) using spatiotemporal attention auto-encoder

Human Activity Recognition Based on Transfer Learning with Spatio-Temporal Representations

A Gait History Image (GHI) is a spatial template that accumulates regions of motion into a single image in which moving pixels are brighter than others. A new descriptor named Time-Sliced Averaged Gradient Boundary Magnitude (TAGBM) is also designed to show the time variations of motion. The spatial and temporal information of each video can be condensed using these templates. Based on this opinion, a new method is proposed in this paper. Each video is split into N and M groups of consecutive frames, and the GHI and TAGBM are computed for each group, resulting spatial and temporal templates. Transfer learning with the fine-tuning technique has been used for classifying these templates. This proposed method achieves the recognition accuracies of 96.50%, 92.30% and 97.12% for KTH, UCF Sport and UCF-11 action datasets, respectively. Also it is compared with state-of-the-art approaches and the results show that the proposed method has the best performance.

Two-stream spatiotemporal feature fusion for human action recognition

Human action recognition is still a challenging topic in the computer vision field that has attracted a large number of researchers. It has a significant importance in varieties of applications such as intelligent video surveillance, sports analysis, and human–computer interaction. Recent works attempt to exploit the progress in deep learning architecture to learn spatial and temporal features from action video. However, it remains unclear how to combine spatial and temporal information with convolutional neural network. In this paper, we propose a novel human action recognition method by fusing spatial and temporal features learned from a simple unsupervised convolutional neural network called principal component analysis network (PCANet) in combination with bag-of-features (BoF) and vector of locally aggregated descriptors (VLAD ) encoding schemes. Firstly, both spatial and temporal features are learned via PCANet using a subset of frames and temporal templates for each video, while their dimensionality is reduced using whitening transformation (WT). The temporal templates are calculated using short-time motion energy images (ST-MEI) based on frame differencing. Then, the encoding scheme is applied to represent the final dual spatiotemporal PCANet features by feature fusion. Finally, the support vector machine (SVM) classifier is exploited for action recognition. Extensive experiments have been performed on two popular datasets, namely KTH and UCF sports, to evaluate the performance of proposed method. Our experimental results using leave-one-out evaluation strategy demonstrate that the proposed method presents satisfactory and comparable results on both datasets.

Human activity recognition by using MHIs of frame sequences

A motion history image MHI is a temporal template that collapses temporal motion information into a single image in which intensity is a function of recency of motion. In recent years, the popularity of deep learning architectures for human activity recognition has encouraged us to explore the effectiveness of combining them and MHIs. Based on this, two new methods are introduced in this paper. In the first method, which is called the basic method, each video splits into N groups of consecutive frames, and the MHI is calculated for each group. Transfer learning with the fine-tuning technique is used for classifying these temporal templates. The experimental results show that some misclassification errors are created because of the similarities between these temporal templates; these errors can be corrected by detecting specific objects in the scenes. Thus, spatial information consisting of a single frame is also added to the second method, called the proposed method. By converting video classification problems into image classification problems in the proposed method, less memory is needed and the time complexity is greatly reduced. They are implemented and compared with state-of-the-art approaches on two data sets. The results show that the proposed method significantly outperforms the others. It achieves recognition accuracies of 92% and 92.4% for the UCF Sport and UCF-11 action data sets, respectively.

Human activity recognition by using MHIs of frame sequences

A motion history image (MHI) is a temporal template that collapses temporal motion information into a single image in which intensity is a function of recency of motion. In recent years, the popularity of deep learning architectures for human activity recognition has encouraged us to explore the effectiveness of combining them and MHIs. Based on this, two new methods are introduced in this paper. In the first method, which is called the basic method, each video splits into N groups of consecutive frames, and the MHI is calculated for each group. Transfer learning with the fine-tuning technique is used for classifying these temporal templates. The experimental results show that some misclassification errors are created because of the similarities between these temporal templates; these errors can be corrected by detecting specific objects in the scenes. Thus, spatial information consisting of a single frame is also added to the second method, called the proposed method. By converting video classification problems into image classification problems in the proposed method, less memory is needed and the time complexity is greatly reduced. They are implemented and compared with state-of-the-art approaches on two data sets. The results show that the proposed method significantly outperforms the others. It achieves recognition accuracies of 92% and 92.4% for the UCF Sport and UCF-11 action data sets, respectively.

Multiple Batches of Motion History Images (MB-MHIs) for Multi-view Human Action Recognition

The recognition of human actions recorded in a multi-camera environment faces the challenging issue of viewpoint variation. Multi-view methods employ videos from different views to generate a compact view-invariant representation of human actions. This paper proposes a novel multi-view human action recognition approach that uses multiple low-dimensional temporal templates and a reconstruction-based encoding scheme. The proposed approach is based upon the extraction of multiple 2D motion history images (MHIs) of human action videos over non-overlapping temporal windows, constructing multiple batches of motion history images (MB-MHIs). Then, two kinds of descriptions are computed for these MHIs batches based on (1) a deep residual network (ResNet) and (2) histogram of oriented gradients (HOG) to effectively quantify a change in gradient. ResNet descriptions are average pooled at each batch. HOG descriptions are processed independently at each batch to learn a class-based dictionary using a K-spectral value decomposition algorithm. Later, the sparse codes of feature descriptions are obtained using an orthogonal matching pursuit approach. These sparse codes are average pooled to extract encoded feature vectors. Then, encoded feature vectors at each batch are fused to form a final view-invariant feature representation. Finally, a linear support vector machine classifier is trained for action recognition. Experimental results are given on three versions of a multi-view dataset: MuHAVi-8, MuHAVi-14, and MuHAVi-uncut. The proposed approach shows promising results when tested for a novel camera. Results on deep features indicate that action representation by MB-MHIs is more view-invariant than single MHIs.

Human Action Recognition using Rule based Fuzzy Motion Feature Templates

This paper proposes a technique for human activity recognition in a video stream. To achieve high accuracy in activity recognition results, the method in its initial step deploys temporal template matching to recognize activities. As temporal templates are susceptible to get affected by speed, style and performance pattern of activity, so it becomes difficult to accurately differentiate among closely similar activities (e.g walking, running and jogging). The confusion in recognizing activities is reconciled by subsequent rule based activity distinction. The proposed method recognizes the human activities in video on various bench-marked data sets including KTH Dataset and Weizmann Dataset. Experimental results demonstrate the novelty of method with a wide spectrum of varied conditions. The average accuracy of the method is 97.20% under standard conditions.

Transactional Behavior Verification in Business Process as a Service Configuration

Business Process as a Service (BPaaS) is an emerging type of cloud service that offers configurable and executable business processes to clients over the Internet. As BPaaS is still in early years of research, many open issues remain. Managing the configuration of BPaaS builds on areas such as software product lines and configurable business processes. The problem has concerns to consider from several perspectives, such as the different types of variable features, constraints between configuration options, and satisfying the requirements provided by the client. In our approach, we use temporal logic templates to elicit transactional requirements from clients that the configured service must adhere to. For formalizing constraints over configuration, feature models are used. To manage all these concerns during BPaaS configuration, we develop a structured process that applies formal methods while directing clients through specifying transactional requirements and selecting configurable features. The Binary Decision Diagram (BDD) analysis is then used to verify that the selected configurable features do not violate any constraints. Finally, model checking is applied to verify the configured service against the transactional requirement set. We demonstrate the feasibility of our approach with several validation scenarios and performance evaluations.

Human action recognition by means of subtensor projections and dense trajectories

In last years, most human action recognition works have used dense trajectories features, to achieve state-of-the-art results. Histograms of Oriented Gradients (HOG), Histogram of Optical Flow (HOF) and Motion Boundary Histograms (MBH) features are extracted from regions and being tracked across the frames.The goal of this paper is to improve the performance obtained by means of Improved Dense Trajectories (IDTs), adding new features based on temporal templates. We construct these templates considering a video sequence as a third-order tensor and computing three different projections. We use several functions for projecting the fibers from the video sequences, and combined them by means of sum pooling.As a first contribution of our work, we present in detail the method based on tensor projections. First, we have assessed the results obtained using only template based action recognition. Next, in order to achieve state-of-art recognition rates, we have fused our features with those of IDTs.This is the second contribution of the article.Experiments on four different public datasets have shown that this technique improves IDTs performance and that the results outperform the ones obtained by most of the state-of-the-art techniques for action recognition.

Human identification based on temporal lifting using 5/3 wavelet filters and radon transform

In this paper, a spatio-temporal gait recognition system is proposed to overcome the limitations associated with the most existing temporal template approaches such as gait energy image (GEI). These approaches do not preserve the whole temporal information in a gait sequence. They are also sensitive to changes in various conditions such as carrying and clothing. These limitations influence the performance of any gait recognition system. To address this problem, a temporal template approach based on lifting 5/3 wavelet filters is presented. In the proposed method named 5/3 gait image (5/3GI), the contour is first extracted from each image in a gait sequence. The gait contour images are then decomposed using 5/3 temporal wavelet filters into two temporal templates at the last temporal decomposition stage. These two templates are subjected to Radon transform for feature extraction. The principal component analysis (PCA) is subsequently applied to the Radon templates in the reference database to identify a subset of Radon template coefficients that carry the most important information suitable for gait recognition. Experimental results on USF HumanID and CASIA gait databases demonstrate that the proposed method achieves a better recognition performance than the most existing methods in the literature especially under walking variations.

P.2.b.017 - Screening for depression in medical students and sports academy students

In this paper, we propose a new temporal template approach for action recognition and person identification based on motion sequence information in masked depth video streams obtained from RGB-D data. This new representation creates a membership function that models the change in motion based on the correlation between frames that occur during motion flow. The energy images created with this function emphasize the intervals of motion with more change, while the intervals with less change are suppressed. To understand the distinctive features, the obtained energy images by using the proposed function are given as input to the convolutional neural networks and different handcrafted classifiers. The proposed method was observed on the BodyLogin, NATOPS, and SBU Kinect datasets and compared with the existing temporal templates and recent methods. The results indicate that the proposed method provides both higher performance and better motion representation.

Action recognition based on binary patterns of action-history and histogram of oriented gradient

In this paper, we have focused on the view-based spatio-temporal template matching approach for human action detection and classification. We have proposed an approach for human activity modeling that describes human motions as a texture pattern. We have combined two relatively simple feature extractors for obtaining a system to get more accurate result. In this method, video sequences are converted into temporal templates called Motion History Image (MHI), which can preserve dominant motion information. The local features are described with Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) descriptors. LBP operator is texture operator that encodes the direction of motion from the non-monotonous areas of MHI images. HOG is used as feature descriptor and extracts the features from LBP. These descriptors are used to train with Support Vector Machine (SVM) classifier to recognize various action classes. This proposed method has been tested on the KTH Action Dataset (which is one of the most widely used benchmark datasets for human action classification), and on the Pedestrian Action Dataset. Our method has shown 86.67 % recognition rate in the 6-classes of KTH Action Dataset and 94.3 % accuracy in the 7-classes of Pedestrian Action Dataset. Based on the complexity of datasets, both the results are quite satisfactory.

Cathodal Transcranial Direct Current Stimulation (tDCS) to the Right Cerebellar Hemisphere Affects Motor Adaptation During Gait.

The cerebellum appears to play a key role in the development of internal rules that allow fast, predictive adjustments to novel stimuli. This is crucial for adaptive motor processes, such as those involved in walking, where cerebellar dysfunction has been found to increase variability in gait parameters. Motor adaptation is a process that results in a progressive reduction in errors as movements are adjusted to meet demands, and within the cerebellum, this seems to be localised primarily within the right hemisphere. To examine the role of the right cerebellar hemisphere in adaptive gait, cathodal transcranial direct current stimulation (tDCS) was administered to the right cerebellar hemisphere of 14 healthy adults in a randomised, double-blind, crossover study. Adaptation to a series of distinct spatial and temporal templates was assessed across tDCS condition via a pressure-sensitive gait mat (ProtoKinetics Zeno walkway), on which participants walked with an induced 'limp' at a non-preferred pace. Variability was assessed across key spatial-temporal gait parameters. It was hypothesised that cathodal tDCS to the right cerebellar hemisphere would disrupt adaptation to the templates, reflected in a failure to reduce variability following stimulation. In partial support, adaptation was disrupted following tDCS on one of the four spatial-temporal templates used. However, there was no evidence for general effects on either the spatial or temporal domain. This suggests, under specific conditions, a coupling of spatial and temporal processing in the right cerebellar hemisphere and highlights the potential importance of task complexity in cerebellar function.

Research on Method for Uyghur temporal Word Recognition

Researches concerning to the temporal expressions in minority languages, particularly the Uyghur temporal word recognition, has not previously been conducted. this paper stated for the first time the relevant progresses and significances on domestic and overseas researches. We analyzed the formation of simple and compound temporal words in modern Uyghur language. We discussed the constitutive rule for Uyghur temporal expressions and put forward a new rule for combined temporal expression templates on the basis of dictionary and regular expressions. We then designed a suitable recognition algorithm and implemented an extraction system for the Uyghur temporal expressions. In the end, the feasibility and usability of the complete recognition method and results were discussed.

On Not Moving Well Enough

In this article I investigate ethnographically how people in the outskirts of Sarajevo attempted to reason their way through a widespread sense of persistent “pattering in place” in postwar, postsocialist, post-Yugoslav Bosnia and Herzegovina (Bosna i Hercegovina [BiH]). Concerns with household futures were explicitly contextualized within the everyday geopolitics of life in a semiprotectorate presumably on the “Road into Europe.” Rather than conceiving of their predicament in terms of “crisis,” my interlocutors diagnosed and criticized spatiotemporal entrapment through a politicizing understanding of the nesting of these different scales. Yet this politicization ultimately had depoliticizing effects, encouraging waiting rather than collective action. At this particular historical conjuncture, I have discerned an economy of temporal reasoning where yearnings for what were called “normal lives” evoked linear, forward movement as an imperative. Acknowledging that yearnings have their own histories, I investigate how a specific valuation of existential mobility along linear temporal templates shaped up at the intersection of, on the one hand, past futures—recalled from lives in Yugoslav socialist BiH and during the 1992–1995 war—and, on the other hand, futures projected as part of BiH’s ongoing “Road into Europe.”

Action performance evaluation in video sequences

In this paper, we propose a computer vision scheme for action performance evaluation in video sequences. Different from action recognition algorithms that aim at the classification of an unknown action from a set of predefined reference actions, the action evaluation system scores the action performance of a trainee with respect to professional trainers. It can be applied to sports, exercise fitness and rehabilitation evaluation. The global spatiotemporal representation extended from the Motion History Image is used to describe space–time changes of an action in videos. A self-comparison mechanism that reconstructs the trainee’s action as a linear combination of the trainers’ temporal templates is used to measure the deviation. An exponential utility function converts the reconstruction error into an understandable score between 0% and 100%. Test results on table-tennis scenarios have shown the effectiveness of the proposed method. It is also computationally very fast for on-line, real-time feedback of action performance.