Action Labels Research Articles

A Novel Double-Layer Framework for Action Prediction

Action prediction aims to infer the category of an action before it is fully executed. It is a challenging task since neither sufficient discriminative information nor the definite progress state of action can be obtained in an incomplete video. In this paper, we propose a novel double-layer learning framework for predicting the category of action from partial observations. Particularly, in the first layer of the framework, an unsupervised semantic reasoning method is presented for exploiting semantic information of an input incomplete video as well as inferring the future semantic information using the prior knowledge provided by training full videos. In the second layer of the framework, a discriminative action prediction model introduces a latent variable to indicate the progress state of the input video and captures the relationship among the actions, video observations, the semantic information, and the latent progress state for predicting the action label of the input video. Extensive experimental results on UT-I #1, UT-I #2, and UCF Sports datasets demonstrate the superiority of our method in predicting actions at the early stage.

A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation.

Action recognition has become a rapidly developing research field within the last decade. But with the increasing demand for large scale data, the need of hand annotated data for the training becomes more and more impractical. One way to avoid frame-based human annotation is the use of action order information to learn the respective action classes. In this context, we propose a hierarchical approach to address the problem of weakly supervised learning of human actions from ordered action labels by structuring recognition in a coarse-to-fine manner. Given a set of videos and an ordered list of the occurring actions, the task is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. We address this problem by combining a framewise RNN model with a coarse probabilistic inference. This combination allows for the temporal alignment of long sequences and thus, for an iterative training of both elements. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes as well as by the introduction of a regularizing length prior. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.

Adversarial Action Prediction Networks.

Different from after-the-fact action recognition, action prediction task requires action labels to be predicted from partially observed videos containing incomplete action executions. It is challenging because these partial videos have insufficient discriminative information, and their temporal structure is damaged. We study this problem in this paper, and propose an efficient and powerful deep network for learning representative and discriminative features for action prediction. Our approach exploits abundant sequential context information in full videos to enrich the feature representations of partial videos. This information is encoded in latent representations using a variational autoencoder (VAE), which are encouraged to be progress-invariant. Decoding such latent representations using another VAE, we can reconstruct missing information in the features extracted from partial videos. An adversarial learning scheme is adopted to differentiate the reconstructed features from the features directly extracted from full videos in order to well align their distributions. A multi-class classifier is also used to encourage the features to be discriminative. Our network jointly learns features and classifiers, and generates the features particularly optimized for action prediction. Extensive experimental results on UCF101, Sports-1M and BIT datasets demonstrate that our approach remarkably outperforms state-of-the-art methods, and shows significant speedup over these methods. Results also show that actions differ in their prediction characteristics; some actions can be correctly predicted even though only the beginning 10% portion of videos is observed.

Sparse representation based classification scheme for human activity recognition using smartphones

The availability of built-in sensors in mobile devices have paved a way for researchers to accurately determine human activities through these sensors. In this paper, we present a novel action recognition system based on sparse representation wherein, eight different human activities were classified. Our proposed classifier employs data of accelerometer, gyroscope, magnetometer and orientation sensor equipped in smartphones for recognizing human activities. Time-domain and frequency-domain features are derived from the acquired sensor data. We have introduced a novel algorithm for fusing the data from the four sensors using a sparse representation based technique that aid in achieving the best classification performance. In the proposed algorithm, if the majority of the sensors indicate a particular class as the output, then that specific class is assigned as the actual test class. However, if there is a disagreement between the classified output of different sensors, then a novel weighted fusion scheme is introduced to fuse the scores and the residue produced by different sensors. The weight used in fusion is chosen to be the standard deviation of the score vector. Thus, the features of excellent sensors are made to bestow more on to the result of action recognition. Finally, the action label is recognized based on an activity metric that maximizes the score while minimizing the residue. The performance analysis of the proposed system is performed using leave-one-subject-out approach. Performance evaluation metrics like recall, precision, specificity, F-score and accuracy are utilized in projecting the performance of the proposed system. It was shown that the proposed system attained a high overall accuracy of about 97.13%.

Interacting Hidden Markov Models for Video Understanding

People, cars and other moving objects in videos generate time series data that can be labeled in many ways. For example, classifiers can label motion tracks according to the object type, the action being performed, or the trajectory of the motion. These labels can be generated for every frame as long as the object stays in view, so object tracks can be modeled as Markov processes with multiple noisy observation streams. A challenge in video recognition is to recover the true state of the track (i.e. its class, action and trajectory) using Markov models without (a) counter-factually assuming that the streams are independent or (b) creating a fully coupled Hidden Markov Model (FCHMM) with an infeasibly large state space. This paper introduces a new method for labeling sequences of hidden states. The method exploits external consistency constraints among streams without modeling complex joint distributions between them. For example, common sense semantics suggest that trees cannot walk. This is an example of an external constraint between an object label (“tree”) and an action label (“walk”). The key to exploiting external constraints is a new variation of the Viterbi algorithm which we call the Viterbi–Segre (VS) algorithm. VS restricts the solution spaces of factorized HMMs to marginal distributions that are compatible with joint distributions satisfying sets of external constraints. Experiments on synthetic data show that VS does a better job of estimating true states with the given observations than the traditional Viterbi algorithm applied to (a) factorized HMMs, (b) FCHMMs, or (c) partially-coupled HMMs that model pairwise dependencies. We then show that VS outperforms factorized and pairwise HMMs on real video data sets for which FCHMMs cannot feasibly be trained.

Video representation by dense trajectories motion map applied to human activity recognition

ABSTRACTThis paper introduces an efficient video representation method based on the dense trajectory motion map (DTM). We utilize the salient features of dense trajectories and motion descriptor to integrate the discriminative information of a video into a map. Firstly, we extract the dense trajectories features by using dense optical flow then multiple descriptors are computed along trajectories to capture appearance and motion information. This result is then integrated into frames difference to integrate entire discriminative information and motion energy to get our first motion map. For the final DTM each generated motion map will be integrated with the absolute frame difference of next two frames till the end of entire video. Finally, we process the resultant DTM by exploring the efficient long-term recurrent convolutional network module for encoding and action label generation. The developed approach is shown better and had comparable recognition results over the existing methods when applied to the publically available human action datasets.

A discriminative structural model for joint segmentation and recognition of human actions

Achieving joint segmentation and recognition of continuous actions in a long-term video is a challenging task due to the varying durations of actions and the complex transitions of multiple actions. In this paper, a novel discriminative structural model is proposed for splitting a long-term video into segments and annotating the action label of each segment. A set of state variables is introduced into the model to explore discriminative semantic concepts shared among different actions. To exploit the statistical dependences among segments, temporal context is captured at both the action level and the semantic concept level. The state variables are treated as latent information in the discriminative structural model and inferred during both training and testing. Experiments on multi-view IXMAS and realistic Hollywood datasets demonstrate the effectiveness of the proposed method.

Towards Automatic Learning of Procedures From Web Instructional Videos

The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation---to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Exploring 3D Human Action Recognition: from Offline to Online.

With the introduction of cost-effective depth sensors, a tremendous amount of research has been devoted to studying human action recognition using 3D motion data. However, most existing methods work in an offline fashion, i.e., they operate on a segmented sequence. There are a few methods specifically designed for online action recognition, which continually predicts action labels as a stream sequence proceeds. In view of this fact, we propose a question: can we draw inspirations and borrow techniques or descriptors from existing offline methods, and then apply these to online action recognition? Note that extending offline techniques or descriptors to online applications is not straightforward, since at least two problems—including real-time performance and sequence segmentation—are usually not considered in offline action recognition. In this paper, we give a positive answer to the question. To develop applicable online action recognition methods, we carefully explore feature extraction, sequence segmentation, computational costs, and classifier selection. The effectiveness of the developed methods is validated on the MSR 3D Online Action dataset and the MSR Daily Activity 3D dataset.

Semi- and weakly-supervised human pose estimation

For human pose estimation in still images, this paper proposes three semi- and weakly-supervised learning schemes. While recent advances of convolutional neural networks improve human pose estimation using supervised training data, our focus is to explore the semi- and weakly-supervised schemes. Our proposed schemes initially learn conventional model(s) for pose estimation from a small amount of standard training images with human pose annotations. For the first semi-supervised learning scheme, this conventional pose model detects candidate poses in training images with no human annotation. From these candidate poses, only true-positives are selected by a classifier using a pose feature representing the configuration of all body parts. The accuracies of these candidate pose estimation and true-positive pose selection are improved by action labels provided to these images in our second and third learning schemes, which are semi- and weakly-supervised learning. While the first and second learning schemes select only poses that are similar to those in the supervised training data, the third scheme selects more true-positive poses that are significantly different from any supervised poses. This pose selection is achieved by pose clustering using outlier pose detection with Dirichlet process mixtures and the Bayes factor. The proposed schemes are validated with large-scale human pose datasets.

YOLO based Human Action Recognition and Localization

Human action recognition in video analytics has been widely studied in recent years. Yet, most of these methods assign a single action label to video after either analyzing a complete video or using classifier for each frame. But when compared to human vision strategy, it can be deduced that we (human) require just an instance of visual data for recognition of scene. It turns out that small group of frames or even single frame from the video are enough for precise recognition. In this paper, we present an approach to detect, localize and recognize actions of interest in almost real-time from frames obtained by a continuous stream of video data that can be captured from a surveillance camera. The model takes input frames after a specified period and is able to give action label based on a single frame. Combining results over specific time we predicted the action label for the stream of video. We demonstrate that YOLO is effective method and comparatively fast for recognition and localization in Liris Human Activities dataset.

Real-time activity recognition for energy efficiency in buildings

More than half of the electricity in residential and commercial buildings is consumed by lighting systems and appliances. Consumption by these service systems is directly associated with occupant activities. By recognizing activities and identifying the associated possible energy savings, more effective strategies can be developed to design better buildings and automation systems. In line with this motivation, using inductive and deductive reasoning, we introduce a framework to detect occupant activities and potential wasted energy consumption and peak-hour usage that could be shifted to non-peak hours in real-time. Our framework consists of three sub-algorithms for action detection, activity recognition and waste estimation. As the real-time input, the action detection algorithm receives the data from the sensing system, consisting of plug meters and sensors, to detect the occurred actions (e.g., turning on an appliance) via our unsupervised clustering models. Detected actions are then used by the activity recognition algorithm to recognize the activities (e.g., preparing food) through semantic reasoning on our constructed ontology. Based on the recognized activities, the waste estimation algorithm identifies the potential waste and estimates the potential savings. To evaluate the performance of our framework, an experimental study was carried out in an office with five occupants and in two single-occupancy apartments for two weeks. Following the experiment, the performance of the action detection and activity recognition algorithms was evaluated using the ground truth labels for actions and activities. Average accuracy was 97.6% for action detection using Gaussian Mixture Model with Principal Components Analysis and 96.7% for activity recognition. In addition, 35.5% of the consumption of an appliance or lighting system in average was identified as potential savings.

Discovering Rhetoric Agreement between a Request and Response

To support a natural flow of a conversation between humans and automated agents, rhetoric structures of each message has to be analyzed. We classify a pair of paragraphs of text as appropriate for one to follow another, or inappropriate, based on both topic and communicative discourse considerations. To represent a multi-sentence message with respect to how it should follow a previous message in a conversation or dialogue, we build an extension of a discourse tree for it. Extended discourse tree is based on a discourse tree for RST relations with labels for communicative actions, and also additional arcs for anaphora and ontology-based relations for entities. We refer to such trees as Communicative Discourse Trees (CDTs). We explore syntactic and discourse features that are indicative of correct vs incorrect request-response or question-answer pairs. Two learning frameworks are used to recognize such correct pairs: deterministic, nearest-neighbor learning of CDTs as graphs, and a tree kernel learning of CDTs, where a feature space of all CDT sub-trees is subject to SVM learning. We form the positive training set from the correct pairs obtained from Yahoo Answers, social network, corporate conversations including Enron emails, customer complaints and interviews by journalists. The corresponding negative training set is artificially created by attaching responses for different, inappropriate requests that include relevant keywords. The evaluation showed that it is possible to recognize valid pairs in 70% of cases in the domains of weak request-response agreement and 80% of cases in the domains of strong agreement, which is essential to support automated conversations. These accuracies are comparable with the benchmark task of classification of discourse trees themselves as valid or invalid, and also with classification of multi-sentence answers in factoid question-answering systems. The applicability of proposed machinery to the problem of chatbots, social chats and programming via NL is demonstrated. We conclude that learning rhetoric structures in the form of CDTs is the key source of data to support answering complex questions, chatbots and dialogue management.

Comparing Pictures and Videos for Teaching Action Labels to Children With Communication Delays

Children with communication delays often display difficulties labeling stimuli in their environment, particularly related to actions. Research supports direct instruction with video and picture stimuli for increasing children’s action labeling repertoires; however, no studies have compared which type of stimuli results in more efficient, maintainable, or generalizable learning for children with communication delays. In the present study, we used a single-case adapted alternating treatments design to compare picture and video stimuli for teaching action labels to three preschool-aged children receiving special education services in a self-contained setting for communication delays. Results indicate that all children acquired target action labels using both picture and video stimuli, and all children generalized some targets to in vivo stimuli without direct instruction. Efficiency, generalization, and maintenance measures varied across children. Implications and future directions for researchers and practitioners are provided.

Transcreation as a translation service: Process requirements and client expectations

In order to remain competitive, many translation companies are expanding their fields of activity to include technical communication, layout, terminology, and other related services, and are thus becoming full-service multilingual and cultural communication vendors. As translation agencies expand their services and competencies — in line with the broad definition of translation used in modern-day translation studies — translating marketing communications materials is one potential area into which they might expand. However, international advertising campaigns for multinational companies are often handled by or with advertising agencies, frequently without the involvement of translation agencies (Anholt 2000). Capturing this additional field brings with it new process requirements, especially for translation agencies which have previously focused more on technical and/or business translation. Depending on the creative brief, the translation of marketing communications materials or advertising campaigns frequently involves a comprehensive rewriting and recreation of the message (Bondarenko 2009; Torresi 2010). The term “transcreation” is often used to emphasise the extensive adaptation thereby required (Ray and Kelly 2010; Rike 2013). However, we argue that the need for a term such as “transcreation” — or “localisation”, for that matter — does not stem from a higher degree of adaptation, but from the specific service processes offered under this translatorial action label. In this paper, we look at the challenges translation agencies may face when entering this field. In an exploratory empirical study, we investigated the expectations of 10 regular clients of an Austrian translation agency regarding the translation (or transcreation) of marketing communications materials. For this purpose, 13 qualitative interviews were carried out with agency clients in Austria, Germany, Italy, and Switzerland. The interviewees were asked about their interest in such a service in principle, its potential added value, and their expectations of the product and process. The study also looked at their expectations of the briefing and review processes. The results serve to outline the necessary modifications to the processes required to offer this specific translation service.

Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos

Every moment counts in action recognition. A comprehensive understanding of human activity in video requires labeling every frame according to the actions occurring, placing multiple labels densely over a video sequence. To study this problem we extend the existing THUMOS dataset and introduce MultiTHUMOS, a new dataset of dense labels over unconstrained internet videos. Modeling multiple, dense labels benefits from temporal relations within and across classes. We define a novel variant of long short-term memory (LSTM) deep networks for modeling these temporal relations via multiple input and output connections. We show that this model improves action labeling accuracy and further enables deeper understanding tasks ranging from structured retrieval to action prediction.

Discriminative Part Selection For Human Action Recognition

Semantic parts have shown a powerful discriminative capacity for action recognition. However, many existing methods select parts according to predefined heuristic rules, which may cause the correlation among parts to be lost, or do not appropriately consider the cluttered candidate part space, which may result in weak generalizability of the resulting action labels. Therefore, better consideration of the correlation among parts and refinement of the candidate space will lead to a more discriminative action representation. This paper achieves improved performance by more elegantly addressing these two factors. First, considering the cluttered nature of the candidate space, we propose a recursive part elimination strategy for iterative refinement of the candidate parts. In each iteration, we eliminate the parts with the lowest weights, which are deemed to be noise. Second, we measure the discriminative capabilities of the candidates and select the top-ranked parts by applying a maximum margin model, which can alleviate overfitting while simultaneously improving generalizability and correlation extraction. Finally, using the selected parts, we extract mid-level features. We report experiments conducted on four datasets (KTH, Olympic Sports, UCF50, and HMDB51). The proposed method can achieve significant improvements compared with other recent methods, including a lower computational cost, a faster speed, and higher accuracy.

Dual many-to-one-encoder-based transfer learning for cross-dataset human action recognition

The emergence of large-scale human action datasets poses a challenge to efficient action labeling. Hand labeling large-scale datasets is tedious and time consuming; thus a more efficient labeling m...

Action Recognition in Still Images With Minimum Annotation Efforts.

We focus on the problem of still image-based human action recognition, which essentially involves making prediction by analyzing human poses and their interaction with objects in the scene. Besides image-level action labels (e.g., riding, phoning), during both training and testing stages, existing works usually require additional input of human bounding boxes to facilitate the characterization of the underlying human-object interactions. We argue that this additional input requirement might severely discourage potential applications and is not very necessary. To this end, a systematic approach was developed in this paper to address this challenging problem of minimum annotation efforts, i.e., to perform recognition in the presence of only image-level action labels in the training stage. Experimental results on three benchmark data sets demonstrate that compared with the state-of-the-art methods that have privileged access to additional human bounding-box annotations, our approach achieves comparable or even superior recognition accuracy using only action annotations in training. Interestingly, as a by-product in many cases, our approach is able to segment out the precise regions of underlying human-object interactions.

Teaching Young Children With Social-Communication Delays to Label Actions Using Videos and Language Expansion Models

Young children with autism spectrum disorder (ASD) and related social-communication delays may display difficulties commenting on actions that occur in their natural environment. One method for increasing early conversation skills is direct instruction. Using video examples of actions may increase the salient features of instructional targets and, as such, may be an effective stimulus for presenting actions during instruction. The present study used a multiple-probe design across participants replicated across behaviors to evaluate the effectiveness of a 0- to 4-s progressive time-delay (PTD) procedure using video presentation of actions to teach three preschool-aged children with ASD and related social-communication delays to label actions. In addition, the teacher provided language expansion on all instructional targets, as well as assessed generalization to novel stimuli (videos and pictures or photographs). Results indicate that all participants acquired action labels. Two of the three participants generalized responses to novel videos and pictures while expanding their responses without direct instruction. Implications for teachers targeting action labels in early childhood settings are provided.