Representation Of Human Activities Research Articles

MHAiR: A Dataset of Audio-Image Representations for Multimodal Human Actions

Audio-image representations for a multimodal human action (MHAiR) dataset contains six different image representations of the audio signals that capture the temporal dynamics of the actions in a very compact and informative way. The dataset was extracted from the audio recordings which were captured from an existing video dataset, i.e., UCF101. Each data sample captured a duration of approximately 10 s long, and the overall dataset was split into 4893 training samples and 1944 testing samples. The resulting feature sequences were then converted into images, which can be used for human action recognition and other related tasks. These images can be used as a benchmark dataset for evaluating the performance of machine learning models for human action recognition and related tasks. These audio-image representations could be suitable for a wide range of applications, such as surveillance, healthcare monitoring, and robotics. The dataset can also be used for transfer learning, where pre-trained models can be fine-tuned on a specific task using specific audio images. Thus, this dataset can facilitate the development of new techniques and approaches for improving the accuracy of human action-related tasks and also serve as a standard benchmark for testing the performance of different machine learning models and algorithms.

Actions are characterized by ‘canonical moments’ in a sequence of movements

Understanding what others are doing is an essential aspect of social cognition that depends on our ability to quickly recognize and categorize their actions. To effectively study action recognition we need to understand how actions are bounded, where they start and where they end. Here we borrow a conceptual approach – the notion of ‘canonicality' – introduced by Palmer and colleagues in their study of object recognition and apply it to the study of action recognition. Using a set of 50 video clips sourced from stock photography sites, we show that many everyday actions – transitive and intransitive, social and non-social, communicative – are characterized by ‘canonical moments' in a sequence of movements that are agreed by participants to ‘best represent’ a named action, as indicated in a forced choice (Exp 1, n = 142) and a free choice (Exp 2, n = 125) paradigm. In Exp 3 (n = 102) we confirm that canonical moments from action sequences are more readily named as depicting specific actions and, mirroring research in object recognition, that such canonical moments are privileged in memory (Exp 4, n = 95). We suggest that ‘canonical moments', being those that convey maximal information about human actions, are integral to the representation of human action.11This research was first presented at the European Conference on Visual Perception (Brady et al., 2021).

Semantic representations of human actions across vision and language

Semantic representations of human actions across vision and language

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

Two‐stream spatiotemporal networks for skeleton action recognition

AbstractSkeleton‐based neural networks have been considered a focus for human action recognition (HAR). It is noteworthy that the existing skeleton‐based methods are not capable of combining the spatial and temporal features reasonably to derive more effective high‐level representations, and it continues to be a challenging task of learning and representing the skeleton action discriminatively. In this study, a novel two‐stream spatiotemporal network (TSTN) is proposed, which is capable of processing the spatial and temporal features respectively and collectively to achieve a better representation and understanding of human action. The temporal branch stacks three gate recurrent unit (GRU) blocks in a new architecture to encode the temporal correlations from different aspects of human action, achieving high‐level temporal semantic feature expressions. The spatial branch encodes the spatial features with multi‐stacked graph convolutional network (GCN) blocks. Self‐attention mechanisms incorporated with the graph structure of the skeleton are explored to add weight influence and structural hints to further enhance the performance. The experimental results verify the effectiveness and superiority of the proposed model in skeleton action recognition; the model reaches state‐of‐the‐art on specific datasets.

Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework.

Vision-based human activity recognition (HAR) has emerged as one of the essential research areas in video analytics. Over the last decade, numerous advanced deep learning algorithms have been introduced to recognize complex human actions from video streams. These deep learning algorithms have shown impressive performance for the video analytics task. However, these newly introduced methods either exclusively focus on model performance or the effectiveness of these models in terms of computational efficiency, resulting in a biased trade-off between robustness and computational efficiency in their proposed methods to deal with challenging HAR problem. To enhance both the accuracy and computational efficiency, this paper presents a computationally efficient yet generic spatial-temporal cascaded framework that exploits the deep discriminative spatial and temporal features for HAR. For efficient representation of human actions, we propose an efficient dual attentional convolutional neural network (DA-CNN) architecture that leverages a unified channel-spatial attention mechanism to extract human-centric salient features in video frames. The dual channel-spatial attention layers together with the convolutional layers learn to be more selective in the spatial receptive fields having objects within the feature maps. The extracted discriminative salient features are then forwarded to a stacked bi-directional gated recurrent unit (Bi-GRU) for long-term temporal modeling and recognition of human actions using both forward and backward pass gradient learning. Extensive experiments are conducted on three publicly available human action datasets, where the obtained results verify the effectiveness of our proposed framework (DA-CNN+Bi-GRU) over the state-of-the-art methods in terms of model accuracy and inference runtime across each dataset. Experimental results show that the DA-CNN+Bi-GRU framework attains an improvement in execution time up to 167× in terms of frames per second as compared to most of the contemporary action-recognition methods.

TFC-GCN: Lightweight Temporal Feature Cross-Extraction Graph Convolutional Network for Skeleton-Based Action Recognition

For skeleton-based action recognition, graph convolutional networks (GCN) have absolute advantages. Existing state-of-the-art (SOTA) methods tended to focus on extracting and identifying features from all bones and joints. However, they ignored many new input features which could be discovered. Moreover, many GCN-based action recognition models did not pay sufficient attention to the extraction of temporal features. In addition, most models had swollen structures due to too many parameters. In order to solve the problems mentioned above, a temporal feature cross-extraction graph convolutional network (TFC-GCN) is proposed, which has a small number of parameters. Firstly, we propose the feature extraction strategy of the relative displacements of joints, which is fitted for the relative displacement between its previous and subsequent frames. Then, TFC-GCN uses a temporal feature cross-extraction block with gated information filtering to excavate high-level representations for human actions. Finally, we propose a stitching spatial–temporal attention (SST-Att) block for different joints to be given different weights so as to obtain favorable results for classification. FLOPs and the number of parameters of TFC-GCN reach 1.90 G and 0.18 M, respectively. The superiority has been verified on three large-scale public datasets, namely NTU RGB + D60, NTU RGB + D120 and UAV-Human.

Recognition of human action for scene understanding using world cup optimization and transfer learning approach.

Understanding human activities is one of the vital steps in visual scene recognition. Human daily activities include diverse scenes with multiple objects having complex interrelationships with each other. Representation of human activities finds application in areas such as surveillance, health care systems, entertainment, automated patient monitoring systems, and so on. Our work focuses on classifying scenes into different classes of human activities like waving hands, gardening, walking, running, etc. The dataset classes were pre-processed using the fuzzy color stacking technique. We adopted the transfer learning concept of pretrained deep CNN models. Our proposed methodology employs pretrained AlexNet, SqueezeNet, ResNet, and DenseNet for feature extraction. The adaptive World Cup Optimization (WCO) algorithm is used halfway to select the superior dominant features. Then, these dominant features are classified by the fully connected classifier layer of DenseNet 201. Evaluation of the performance matrices showed an accuracy of 94.7% with DenseNet as the feature extractor and WCO for feature selection compared to other models. Also, our proposed methodology proved to be superior to its counterpart without feature selection. Thus, we could improve the quality of the classification model by providing double filtering using the WCO feature selection process.

Radar Human Activity Recognition with an Attention-Based Deep Learning Network

Radar-based human activity recognition (HAR) provides a non-contact method for many scenarios, such as human-computer interaction, smart security, and advanced surveillance with privacy protection. Feeding radar-preprocessed micro-Doppler signals into a deep learning (DL) network is a promising approach for HAR. Conventional DL algorithms can achieve high performance in terms of accuracy, but the complex network structure causes difficulty for their real-time embedded application. In this study, an efficient network with an attention mechanism is proposed. This network decouples the Doppler and temporal features of radar preprocessed signals according to the feature representation of human activity in the time-frequency domain. The Doppler feature representation is obtained in sequence using the one-dimensional convolutional neural network (1D CNN) following the sliding window. Then, HAR is realized by inputting the Doppler features into the attention-mechanism-based long short-term memory (LSTM) as a time sequence. Moreover, the activity features are effectively enhanced using the averaged cancellation method, which improves the clutter suppression effect under the micro-motion conditions. Compared with the traditional moving target indicator (MTI), the recognition accuracy is improved by about 3.7%. Experiments based on two human activity datasets confirm the superiority of our method compared to traditional methods in terms of expressiveness and computational efficiency. Specifically, our method achieves an accuracy close to 96.9% on both datasets and has a more lightweight network structure compared to algorithms with similar recognition accuracy. The method proposed in this article has great potential for real-time embedded applications of HAR.

SMAM: Self and Mutual Adaptive Matching for Skeleton-Based Few-Shot Action Recognition.

This paper focuses on skeleton-based few-shot action recognition. Since skeleton is essentially a sparse representation of human action, the feature maps extracted from it, through a standard encoder network in the few-shot condition, may not be sufficiently discriminative for some action sequences that look partially similar to each other. To address this issue, we propose a self and mutual adaptive matching (SMAM) module to convert such feature maps into more discriminative feature vectors. Our method, named as SMAM-Net, first leverages both the temporal information associated with each individual skeleton joint and the spatial relationship among them for feature extraction. Then, the SMAM module adaptively measures the similarity between labeled and query samples and further carries out feature matching within the query set to distinguish similar skeletons of various action categories. Experimental results show that the SMAM-Net outperforms other baselines on the large-scale NTU RGB+D 120 dataset in the tasks of one-shot and five-shot action recognition. We also report our results on smaller datasets including NTU RGB+D 60, SYSU and PKU-MMD to demonstrate that our method is reliable and generalises well on different datasets. Codes and the pretrained SMAM-Net will be made publicly available.

VW-SC3D: A Sparse 3D CNN-Based Spatial–Temporal Network with View Weighting for Skeleton-Based Action Recognition

In recent years, human action recognition has received increasing attention as a significant function of human–machine interaction. The human skeleton is one of the most effective representations of human actions because it is highly compact and informative. Many recent skeleton-based action recognition methods are based on graph convolutional networks (GCNs) as they preserve the topology of the human skeleton while extracting features. Although many of these methods give impressive results, there are some limitations in robustness, interoperability, and scalability. Furthermore, most of these methods ignore the underlying information of view direction and rely on the model to learn how to adjust the view from training data. In this work, we propose VW-SC3D, a spatial–temporal model with view weighting for skeleton-based action recognition. In brief, our model uses a sparse 3D CNN to extract spatial features for each frame and uses a transformer encoder to obtain temporal information within the frames. Compared to GCN-based methods, our method performs better in extracting spatial–temporal features and is more adaptive to different types of 3D skeleton data. The sparse 3D CNN makes our model more computationally efficient and more flexible. In addition, a learnable view weighting module enhances the robustness of the proposed model against viewpoint changes. A test on two different types of datasets shows a competitive result with SOTA methods, and the performance is even better in view-changing situations.

TRANSFORMATIONS OF MOTIVES OF RELIGIOUS DRAMA IN THE LITERATURE OF THE 20TH-21ST CENTURIES: THE PHILOSOPHIC ASPECT

The article studies the transformation of motives of religious drama in the literature of the 20th-21st centuries from the point of view of religious philosophy, namely the neo-Thomism paradigm. The concept of transformation is interpreted as a transition to the use of meaning-generating models close to drama as a kind of literature in the context of the visual turn in the culture of the 20th-21st centuries. The actualization of action as a form-creating means along with the explicit or implicit accentuation of the integrity of the world picture is interpreted as a search for metaphysical grounds for the aesthetic systems of modernism and postmodernism and the poetics of individual authors and their works. The article substantiates the philosophical aspect of the interpretation of religious drama accepted in scientific literature as a metagenre for the literature of the 20th-21st centuries. The study outlines the universal character of generic methods and means of expressiveness of religious drama for literature in the conditions of the visual turn.
 The aesthetics of literature between phenomenological and metaphysical approaches to human action may be inclined to the fluctuation (metaxis) between them, which conditions the stylistic and genre openness of religious drama. In this context, its elements are used both in traditional genres and to rethink the genre and stylistic boundaries of the representation of human action in art in general and in literature in particular. Motives of religious drama (journey, overcoming obstacles, battle, feat as a leap of faith, love, despair, hatred, boredom as existential aspects of the human condition, service, sacrifice) in the works of P. Claudel, H. Chubai and D. Barthelmy are presented with significant differences, but with the preservation of paradigmatic features – symbolism and metaphysics as an indication of the ultimate integrity, unattainable neither rationally nor sensually. The universalism of the motives of religious drama is interpreted as the present ambivalence, which brings together high and low, profane and sacred, parody and grotesque with pathos, etc.

ConvNet-based performers attention and supervised contrastive learning for activity recognition

Human activity recognition based on generated sensor data plays a major role in a large number of applications such as healthcare monitoring and surveillance system. Yet, accurately recognizing human activities is still challenging and active research due to people’s tendency to perform daily activities in a different and multitasking way. Existing approaches based on the recurrent setting for human activity recognition have some issues, such as the inability to process data parallelly, the requirement for more memory and high computational cost albeit they achieved reasonable results. Convolutional Neural Network processes data parallelly, but, it breaks the ordering of input data, which is significant to build an effective model for human activity recognition. To overcome these challenges, this study proposes causal convolution based on performers-attention and supervised contrastive learning to entirely forego recurrent architectures, efficiently maintain the ordering of human daily activities and focus more on important timesteps of the sensors’ data. Supervised contrastive learning is integrated to learn a discriminative representation of human activities and enhance predictive performance. The proposed network is extensively evaluated for human activities using multiple datasets including wearable sensor data and smart home environments data. The experiments on three wearable sensor datasets and five smart home public datasets of human activities reveal that our proposed network achieves better results and reduces the training time compared with the existing state-of-the-art methods and basic temporal models.

When proxy-driven learning is no better than random: The consequences of representational incompleteness.

Machine learning is widely used for personalisation, that is, to tune systems with the aim of adapting their behaviour to the responses of humans. This tuning relies on quantified features that capture the human actions, and also on objective functions—that is, proxies – that are intended to represent desirable outcomes. However, a learning system’s representation of the world can be incomplete or insufficiently rich, for example if users’ decisions are based on properties of which the system is unaware. Moreover, the incompleteness of proxies can be argued to be an intrinsic property of computational systems, as they are based on literal representations of human actions rather than on the human actions themselves; this problem is distinct from the usual aspects of bias that are examined in machine learning literature. We use mathematical analysis and simulations of a reinforcement-learning case study to demonstrate that incompleteness of representation can, first, lead to learning that is no better than random; and second, means that the learning system can be inherently unaware that it is failing. This result has implications for the limits and applications of machine learning systems in human domains.

Colloquy with Juan José García-Noblejas on understanding communication from the perspective of Aristotle's Poetics

In this interview we will explore García-Noblejas’ thought on public communication, which is deeply marked by his interpretation of Aristotle’s Poetics and influenced by some contemporary thinkers. For García-Noblejas, communication is a service and a form of practical knowledge with five dimensions: ethics, politics, rhetoric, aesthetics, and poetics. Poetics is the most important of these, since its symbolic perspective of the representation of human actions allows us to deeply understand the narrative and dramatic texts typical of journalism, advertising, and propaganda, as well as written and audiovisual fiction. Poetics is the art of creating “possible worlds” in which the reader/spectator recognizes himself (or herself) and with which he dialogues starting from his own identity. García-Noblejas defends public communication that is based on the gift of self and looks to community fruitfulness. He also discusses some frequent pathologies in the media. Finally, he proposes his theory of the Second Navigation to discern the meaning of a text and explains why it is necessary to give a transcendent meaning to the Aristotelian concept of catharsis.

When Measurement Misleads

The discipline of information retrieval (IR) has a long history of examination of how best to measure performance. In particular, there is an extensive literature on the practice of assessing retrieval systems using batch experiments based on collections and relevance judgements. However, this literature has only rarely considered an underlying principle: that measured scores are inherently incomplete as a representation of human activity, that is, there is an innate gap between measured scores and the desired goal of human satisfaction. There are separate challenges such as poor experimental practices or the shortcomings of specific measures, but the issue considered here is more fundamental - straightforwardly, in batch experiments the human-machine gap cannot be closed. In other disciplines, the issue of the gap is well recognised and has been the subject of observations that provide valuable perspectives on the behaviour and effects of measures and the ways in which they can lead to unintended consequences, notably Goodhart's law and the Lucas critique. Here I describe these observations and argue that there is evidence that they apply to IR, thus showing that blind pursuit of performance gains based on optimisation of scores, and analysis based solely on aggregated measurements, can lead to misleading and unreliable outcomes.

Two distinct networks containing position-tolerant representations of actions in the human brain.

Humans can recognize others' actions in the social environment. This action recognition ability is rarely hindered by the movement of people in the environment. The neural basis of this position tolerance for observed actions is not fully understood. Here, we aimed to identify brain regions capable of generalizing representations of actions across different positions and investigate the representational content of these regions. In a functional magnetic resonance imaging experiment, participants viewed point-light displays of different human actions. Stimuli were presented in either the upper or the lower visual field. Multivariate pattern analysis and a surface-based searchlight approach were employed to identify brain regions that contain position-tolerant action representation: Classifiers were trained with patterns in response to stimuli presented in one position and were tested with stimuli presented in another position. Results showed above-chance classification in the left and right lateral occipitotemporal cortices, right intraparietal sulcus, and right postcentral gyrus. Further analyses exploring the representational content of these regions showed that responses in the lateral occipitotemporal regions were more related to subjective judgments, while those in the parietal regions were more related to objective measures. These results provide evidence for two networks that contain abstract representations of human actions with distinct representational content.

The Effect of Using VR Box in Learning Effective Shadow Boxing

The main purpose of the current study was to prepare exercises for the use of virtual reality, called (3D recording with 360 degrees) as well as to identify the effect of using virtual reality on learning effective shadow boxing. The researcher used the experimental method for its suitability for the nature of the study. (20) Undergraduate male students were chosen from the second year of college of physical education and sports science, students at Salahaddin University –Erbil. (n=20, age 19 ± 1.45). The sample was selected using a simple random selection approach, which resulted in ten (10) students from class (D) being assigned to an experimental group that wore virtual reality spectacles. As a control group, there were ten students in class (A). The educational program for the experimental group who wore the (VR) goggles had two instructional units and the same for the control group but with no (VR Box) each week for both groups (experimental and control) for a total of four weeks. The results showed the effectiveness of virtual reality in learning effective shadow boxing; which presence of statistically significant differences between the experimental and control groups in the post-test and in favor of the experimental group. Reference: Al Gburi ,I., Ibrahim .(2017) ‘the impact of virtual reality technology on improving cognitive achievement and learning some basic gymnastics skills for students’ unpublished master thesis. University of Karbala. Bao,T., Carender, WJ,. Kinnaird ,C., Barone, VJ., Peethambaran, G., Whitney, SL., Kabeto, M. ,Seidler RD., Sienko, KH.(2018) ‘ Effects of long-term balance training with vibrotactile sensory augmentation among community-dwelling healthy older adults: a randomized preliminary study’. J NeuroEng Rehab,15(1):5. Barry, G., van Schaik, P., MacSween, A.(2016)‘ Exergaming (XBOX Kinect™) versus traditional gym-based exercise for postural control, flow and technology acceptance in healthy adults: a randomised controlled trial’. BMC Sports Sci Med Rehabilitation , 8, 25 (2016). https://doi.org/10.1186/s13102-016-0050-0 Bliss, JP., Tidwell, PD., Guest, MA. (1997) ‘The effectiveness of virtual reality for administering spatial navigation training to firefighters’. Presence: Teleoperators & Virtual Environments. 6(1), pp.73–86. Düking, P., Holmberg ,H-C. and Sperlich, B. (2018) ‘ The Potential Usefulness of Virtual Reality Systems for Athletes: A Short SWOT Analysis’. Physiol. 9:128. doi: 10.3389/fphys.2018.00128. Eaves, D. L., Breslin, G., & Van Schaik, P. (2011) ‘The short-term effects of real-time virtual reality feedback on motor learning in dance’. Presence: Teleoperators and Virtual Environments, 20(1), 62-77. Mohamed, A.,Shawky. A., Muhammad, Abdul Hamid seed, M, Amir. (2019)’The effect of using virtual reality glasses on learning some basic skills in volleyball’, Journal of Physical Education and Sports Sciences.Vol 42,( Benha University - Faculty of Physical Education for Boys - Dec.2019 Part (9). McConville, KMV.,Virk, S.(2012)‘ Evaluation of an electronic video game for improvement of balance’. Virtual Reality. 16(4):315–23. https://doi.org/10.1007/s10055-012-0212-7. Miles, HC., Pop, SR., Watt, SJ., Lawrence, GP., John, NW. (2012) ‘A review of virtual environments for training in ball sports’. Computer Graph, 36, pp. 714–726. Riva, G., Wiederhold, BK., Mantovani, F. (2019) ‘ Neuroscience of virtual reality: from virtual exposure to embodied medicine’, Cyberpsychology, Behav Soc Netw, Jan 1; 22(1): 82–96. Published online 2019 Jan 14. doi: 1089/cyber.2017.29099.gri Sarteep, S., Salah. (2020) ‘The effectiveness of a designed device for measuring and improving choice reaction time among players of physical education and sport Science College’. International Journal of Physical Education, Sports and Health 7(6): 257-260. Sarteep, S.,Salah.(2021) ‘The Study Of Mental Imagery Dimensions And Their Relationship To Perform DIVE On Floor Exercise In gymnastics’. Indian Journal Physical Education, Sports and Applied Science,Vol11, NO 1, Jan,pp.19-23. Seymour,. NE., Gallagher, AG., Roman, SA., O’brien, MK., Bansal, VK., Andersen, DK. (2002)‘Virtual reality training improves operating room performance: Results of a randomized, double-blinded study’. Annals of Surgery. 2002 Oct; 236(4):458-63; discussion 463-4. doi: 10.1097/00000658-200210000-00008. PMID: 12368674; PMCID: PMC1422600. Stefan ,C., Michalski,.ID., Ancret, S., Dimitrios, S. , Tyler, J., Ross , Mark, B., Tobias, L.( 2019),‘ Getting your game on: Using virtual reality to improve real table tennis skills’.https://doi.org/10.1371/journal.pone.0222351. Stevens, J. A. (2005) ‘Interference effects demonstrate distinct roles for visual and motor imagery during the mental representation of human action’. Cognition, 95(3), 329-350 White, A., and Hardy, L. (1995) ‘Use of different imagery perspectives on the learning and performance of different motor skills’, British Journal of Psychology, 86 (Pt. 2), 169-180.

Self-Supervised Human Activity Representation for Embodied Cognition Assessment

Physical activities, according to the embodied cognition theory, are an important manifestation of cognitive functions. As a result, in this paper, the Activate Test of Embodied Cognition (ATEC) system is proposed to assess various cognitive measures. It consists of physical exercises with different variations and difficulty levels designed to provide assessment of executive and motor functions. This work focuses on obtaining human activity representation from recorded videos of ATEC tasks in order to automatically assess embodied cognition performance. A self-supervised approach is employed in this work that can exploit a small set of annotated data to obtain an effective human activity representation. The performance of different self-supervised approaches along with a supervised method are investigated for automated cognitive assessment of children performing ATEC tasks. The results show that the supervised learning approach performance decreases as the training set becomes smaller, whereas the self-supervised methods maintain their performance by taking advantage of unlabeled data.

Generalized Pose Decoupled Network for Unsupervised 3D Skeleton Sequence-Based Action Representation Learning.

Human action representation is derived from the description of human shape and motion. The traditional unsupervised 3-dimensional (3D) human action representation learning method uses a recurrent neural network (RNN)-based autoencoder to reconstruct the input pose sequence and then takes the midlevel feature of the autoencoder as representation. Although RNN can implicitly learn a certain amount of motion information, the extracted representation mainly describes the human shape and is insufficient to describe motion information. Therefore, we first present a handcrafted motion feature called pose flow to guide the reconstruction of the autoencoder, whose midlevel feature is expected to describe motion information. The performance is limited as we observe that actions can be distinctive in either motion direction or motion norm. For example, we can distinguish "sitting down" and "standing up" from motion direction yet distinguish "running" and "jogging" from motion norm. In these cases, it is difficult to learn distinctive features from pose flow where direction and norm are mixed. To this end, we present an explicit pose decoupled flow network (PDF-E) to learn from direction and norm in a multi-task learning framework, where 1 encoder is used to generate representation and 2 decoders are used to generating direction and norm, respectively. Further, we use reconstructing the input pose sequence as an additional constraint and present a generalized PDF network (PDF-G) to learn both motion and shape information, which achieves state-of-the-art performances on large-scale and challenging 3D action recognition datasets including the NTU RGB+D 60 dataset and NTU RGB+D 120 dataset.