Representation Of Human Activities Research Articles

A new bi-level deep human action representation structure based on the sequence of sub-actions

A new bi-level deep human action representation structure based on the sequence of sub-actions

Shared representations of human actions across vision and language

Humans can recognize and communicate about many actions performed by others. How are actions organized in the mind, and is this organization shared across vision and language? We collected similarity judgments of human actions depicted through naturalistic videos and sentences, and tested four models of action categorization, defining actions at different levels of abstraction ranging from specific (action verb) to broad (action target: whether an action is directed towards an object, another person, or the self). The similarity judgments reflected a shared organization of action representations across videos and sentences, determined mainly by the target of actions, even after accounting for other semantic features. Furthermore, language model embeddings predicted the behavioral similarity of action videos and sentences, and captured information about the target of actions alongside unique semantic information. Together, our results show that action concepts are similarly organized in the mind across vision and language, and that this organization reflects socially relevant goals.

Estimating built-up area carbon emissions through addressing regional development disparities with population and nighttime light data

Carbon emission inventories in urban built-up areas are critical for formulating effective emission reduction policies. Nighttime light data, reflecting socioeconomic activities, is routinely used as a proxy for estimating emissions. However, the reliability of this proxy is limited in areas with minimal or absent lighting and demonstrates variability according to the urban development. To address these limitations, a novel approach was developed. Firstly, it enhances the representation of human activities under different light values through integrating light values and population distributions into multidimensional proxy features. Secondly, cities with different socio-economic development were clustered to minimize the development disparities within the same cluster. Four clusters were detected including ‘Agriculture-dependent & Low-density’, ‘Agriculture-dependent & High-density’, ‘Industry-oriented’, and ‘Tertiary-oriented’. Then A multilayer perceptron regressor, refined with a moving average method, was employed to model the nonlinear relationships between emissions and these proxy features. The results show a significant increase in modeling accuracy when regional development disparities are incorporated through this clustering strategy. Additionally, the Shapley Additive Explanatory method was utilized to analyze the influence of regional development heterogeneity on the estimation model. It demonstrated that areas with high light intensity and population significantly influence emission estimates in all clusters, except for the ‘Industry-oriented cluster’. This finding underscores the varying effectiveness of proxy features across different regions, emphasizing the importance of considering regional development disparities. Finally, based on the estimated inventory obtained, spatio-temporal hot spot detection pinpointed eastern and central China as significant emission hot spots. This research not only enhances the data resources for managing carbon emissions but also underscores the need for inter-city cooperation strategies to achieve China's carbon reduction targets.

MHAiR: A Dataset of Audio-Image Representations for Multimodal Human Actions

Audio-image representations for a multimodal human action (MHAiR) dataset contains six different image representations of the audio signals that capture the temporal dynamics of the actions in a very compact and informative way. The dataset was extracted from the audio recordings which were captured from an existing video dataset, i.e., UCF101. Each data sample captured a duration of approximately 10 s long, and the overall dataset was split into 4893 training samples and 1944 testing samples. The resulting feature sequences were then converted into images, which can be used for human action recognition and other related tasks. These images can be used as a benchmark dataset for evaluating the performance of machine learning models for human action recognition and related tasks. These audio-image representations could be suitable for a wide range of applications, such as surveillance, healthcare monitoring, and robotics. The dataset can also be used for transfer learning, where pre-trained models can be fine-tuned on a specific task using specific audio images. Thus, this dataset can facilitate the development of new techniques and approaches for improving the accuracy of human action-related tasks and also serve as a standard benchmark for testing the performance of different machine learning models and algorithms.

Actions are characterized by ‘canonical moments’ in a sequence of movements

Understanding what others are doing is an essential aspect of social cognition that depends on our ability to quickly recognize and categorize their actions. To effectively study action recognition we need to understand how actions are bounded, where they start and where they end. Here we borrow a conceptual approach – the notion of ‘canonicality' – introduced by Palmer and colleagues in their study of object recognition and apply it to the study of action recognition. Using a set of 50 video clips sourced from stock photography sites, we show that many everyday actions – transitive and intransitive, social and non-social, communicative – are characterized by ‘canonical moments' in a sequence of movements that are agreed by participants to ‘best represent’ a named action, as indicated in a forced choice (Exp 1, n = 142) and a free choice (Exp 2, n = 125) paradigm. In Exp 3 (n = 102) we confirm that canonical moments from action sequences are more readily named as depicting specific actions and, mirroring research in object recognition, that such canonical moments are privileged in memory (Exp 4, n = 95). We suggest that ‘canonical moments', being those that convey maximal information about human actions, are integral to the representation of human action.11This research was first presented at the European Conference on Visual Perception (Brady et al., 2021).

Semantic representations of human actions across vision and language

Semantic representations of human actions across vision and language

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

Two‐stream spatiotemporal networks for skeleton action recognition

AbstractSkeleton‐based neural networks have been considered a focus for human action recognition (HAR). It is noteworthy that the existing skeleton‐based methods are not capable of combining the spatial and temporal features reasonably to derive more effective high‐level representations, and it continues to be a challenging task of learning and representing the skeleton action discriminatively. In this study, a novel two‐stream spatiotemporal network (TSTN) is proposed, which is capable of processing the spatial and temporal features respectively and collectively to achieve a better representation and understanding of human action. The temporal branch stacks three gate recurrent unit (GRU) blocks in a new architecture to encode the temporal correlations from different aspects of human action, achieving high‐level temporal semantic feature expressions. The spatial branch encodes the spatial features with multi‐stacked graph convolutional network (GCN) blocks. Self‐attention mechanisms incorporated with the graph structure of the skeleton are explored to add weight influence and structural hints to further enhance the performance. The experimental results verify the effectiveness and superiority of the proposed model in skeleton action recognition; the model reaches state‐of‐the‐art on specific datasets.

Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework.

Vision-based human activity recognition (HAR) has emerged as one of the essential research areas in video analytics. Over the last decade, numerous advanced deep learning algorithms have been introduced to recognize complex human actions from video streams. These deep learning algorithms have shown impressive performance for the video analytics task. However, these newly introduced methods either exclusively focus on model performance or the effectiveness of these models in terms of computational efficiency, resulting in a biased trade-off between robustness and computational efficiency in their proposed methods to deal with challenging HAR problem. To enhance both the accuracy and computational efficiency, this paper presents a computationally efficient yet generic spatial-temporal cascaded framework that exploits the deep discriminative spatial and temporal features for HAR. For efficient representation of human actions, we propose an efficient dual attentional convolutional neural network (DA-CNN) architecture that leverages a unified channel-spatial attention mechanism to extract human-centric salient features in video frames. The dual channel-spatial attention layers together with the convolutional layers learn to be more selective in the spatial receptive fields having objects within the feature maps. The extracted discriminative salient features are then forwarded to a stacked bi-directional gated recurrent unit (Bi-GRU) for long-term temporal modeling and recognition of human actions using both forward and backward pass gradient learning. Extensive experiments are conducted on three publicly available human action datasets, where the obtained results verify the effectiveness of our proposed framework (DA-CNN+Bi-GRU) over the state-of-the-art methods in terms of model accuracy and inference runtime across each dataset. Experimental results show that the DA-CNN+Bi-GRU framework attains an improvement in execution time up to 167× in terms of frames per second as compared to most of the contemporary action-recognition methods.

TFC-GCN: Lightweight Temporal Feature Cross-Extraction Graph Convolutional Network for Skeleton-Based Action Recognition

For skeleton-based action recognition, graph convolutional networks (GCN) have absolute advantages. Existing state-of-the-art (SOTA) methods tended to focus on extracting and identifying features from all bones and joints. However, they ignored many new input features which could be discovered. Moreover, many GCN-based action recognition models did not pay sufficient attention to the extraction of temporal features. In addition, most models had swollen structures due to too many parameters. In order to solve the problems mentioned above, a temporal feature cross-extraction graph convolutional network (TFC-GCN) is proposed, which has a small number of parameters. Firstly, we propose the feature extraction strategy of the relative displacements of joints, which is fitted for the relative displacement between its previous and subsequent frames. Then, TFC-GCN uses a temporal feature cross-extraction block with gated information filtering to excavate high-level representations for human actions. Finally, we propose a stitching spatial–temporal attention (SST-Att) block for different joints to be given different weights so as to obtain favorable results for classification. FLOPs and the number of parameters of TFC-GCN reach 1.90 G and 0.18 M, respectively. The superiority has been verified on three large-scale public datasets, namely NTU RGB + D60, NTU RGB + D120 and UAV-Human.

Recognition of human action for scene understanding using world cup optimization and transfer learning approach.

Understanding human activities is one of the vital steps in visual scene recognition. Human daily activities include diverse scenes with multiple objects having complex interrelationships with each other. Representation of human activities finds application in areas such as surveillance, health care systems, entertainment, automated patient monitoring systems, and so on. Our work focuses on classifying scenes into different classes of human activities like waving hands, gardening, walking, running, etc. The dataset classes were pre-processed using the fuzzy color stacking technique. We adopted the transfer learning concept of pretrained deep CNN models. Our proposed methodology employs pretrained AlexNet, SqueezeNet, ResNet, and DenseNet for feature extraction. The adaptive World Cup Optimization (WCO) algorithm is used halfway to select the superior dominant features. Then, these dominant features are classified by the fully connected classifier layer of DenseNet 201. Evaluation of the performance matrices showed an accuracy of 94.7% with DenseNet as the feature extractor and WCO for feature selection compared to other models. Also, our proposed methodology proved to be superior to its counterpart without feature selection. Thus, we could improve the quality of the classification model by providing double filtering using the WCO feature selection process.

Radar Human Activity Recognition with an Attention-Based Deep Learning Network

Radar-based human activity recognition (HAR) provides a non-contact method for many scenarios, such as human-computer interaction, smart security, and advanced surveillance with privacy protection. Feeding radar-preprocessed micro-Doppler signals into a deep learning (DL) network is a promising approach for HAR. Conventional DL algorithms can achieve high performance in terms of accuracy, but the complex network structure causes difficulty for their real-time embedded application. In this study, an efficient network with an attention mechanism is proposed. This network decouples the Doppler and temporal features of radar preprocessed signals according to the feature representation of human activity in the time-frequency domain. The Doppler feature representation is obtained in sequence using the one-dimensional convolutional neural network (1D CNN) following the sliding window. Then, HAR is realized by inputting the Doppler features into the attention-mechanism-based long short-term memory (LSTM) as a time sequence. Moreover, the activity features are effectively enhanced using the averaged cancellation method, which improves the clutter suppression effect under the micro-motion conditions. Compared with the traditional moving target indicator (MTI), the recognition accuracy is improved by about 3.7%. Experiments based on two human activity datasets confirm the superiority of our method compared to traditional methods in terms of expressiveness and computational efficiency. Specifically, our method achieves an accuracy close to 96.9% on both datasets and has a more lightweight network structure compared to algorithms with similar recognition accuracy. The method proposed in this article has great potential for real-time embedded applications of HAR.

SMAM: Self and Mutual Adaptive Matching for Skeleton-Based Few-Shot Action Recognition.

This paper focuses on skeleton-based few-shot action recognition. Since skeleton is essentially a sparse representation of human action, the feature maps extracted from it, through a standard encoder network in the few-shot condition, may not be sufficiently discriminative for some action sequences that look partially similar to each other. To address this issue, we propose a self and mutual adaptive matching (SMAM) module to convert such feature maps into more discriminative feature vectors. Our method, named as SMAM-Net, first leverages both the temporal information associated with each individual skeleton joint and the spatial relationship among them for feature extraction. Then, the SMAM module adaptively measures the similarity between labeled and query samples and further carries out feature matching within the query set to distinguish similar skeletons of various action categories. Experimental results show that the SMAM-Net outperforms other baselines on the large-scale NTU RGB+D 120 dataset in the tasks of one-shot and five-shot action recognition. We also report our results on smaller datasets including NTU RGB+D 60, SYSU and PKU-MMD to demonstrate that our method is reliable and generalises well on different datasets. Codes and the pretrained SMAM-Net will be made publicly available.

VW-SC3D: A Sparse 3D CNN-Based Spatial–Temporal Network with View Weighting for Skeleton-Based Action Recognition

In recent years, human action recognition has received increasing attention as a significant function of human–machine interaction. The human skeleton is one of the most effective representations of human actions because it is highly compact and informative. Many recent skeleton-based action recognition methods are based on graph convolutional networks (GCNs) as they preserve the topology of the human skeleton while extracting features. Although many of these methods give impressive results, there are some limitations in robustness, interoperability, and scalability. Furthermore, most of these methods ignore the underlying information of view direction and rely on the model to learn how to adjust the view from training data. In this work, we propose VW-SC3D, a spatial–temporal model with view weighting for skeleton-based action recognition. In brief, our model uses a sparse 3D CNN to extract spatial features for each frame and uses a transformer encoder to obtain temporal information within the frames. Compared to GCN-based methods, our method performs better in extracting spatial–temporal features and is more adaptive to different types of 3D skeleton data. The sparse 3D CNN makes our model more computationally efficient and more flexible. In addition, a learnable view weighting module enhances the robustness of the proposed model against viewpoint changes. A test on two different types of datasets shows a competitive result with SOTA methods, and the performance is even better in view-changing situations.

TRANSFORMATIONS OF MOTIVES OF RELIGIOUS DRAMA IN THE LITERATURE OF THE 20TH-21ST CENTURIES: THE PHILOSOPHIC ASPECT

The article studies the transformation of motives of religious drama in the literature of the 20th-21st centuries from the point of view of religious philosophy, namely the neo-Thomism paradigm. The concept of transformation is interpreted as a transition to the use of meaning-generating models close to drama as a kind of literature in the context of the visual turn in the culture of the 20th-21st centuries. The actualization of action as a form-creating means along with the explicit or implicit accentuation of the integrity of the world picture is interpreted as a search for metaphysical grounds for the aesthetic systems of modernism and postmodernism and the poetics of individual authors and their works. The article substantiates the philosophical aspect of the interpretation of religious drama accepted in scientific literature as a metagenre for the literature of the 20th-21st centuries. The study outlines the universal character of generic methods and means of expressiveness of religious drama for literature in the conditions of the visual turn.
 The aesthetics of literature between phenomenological and metaphysical approaches to human action may be inclined to the fluctuation (metaxis) between them, which conditions the stylistic and genre openness of religious drama. In this context, its elements are used both in traditional genres and to rethink the genre and stylistic boundaries of the representation of human action in art in general and in literature in particular. Motives of religious drama (journey, overcoming obstacles, battle, feat as a leap of faith, love, despair, hatred, boredom as existential aspects of the human condition, service, sacrifice) in the works of P. Claudel, H. Chubai and D. Barthelmy are presented with significant differences, but with the preservation of paradigmatic features – symbolism and metaphysics as an indication of the ultimate integrity, unattainable neither rationally nor sensually. The universalism of the motives of religious drama is interpreted as the present ambivalence, which brings together high and low, profane and sacred, parody and grotesque with pathos, etc.

ConvNet-based performers attention and supervised contrastive learning for activity recognition

Human activity recognition based on generated sensor data plays a major role in a large number of applications such as healthcare monitoring and surveillance system. Yet, accurately recognizing human activities is still challenging and active research due to people’s tendency to perform daily activities in a different and multitasking way. Existing approaches based on the recurrent setting for human activity recognition have some issues, such as the inability to process data parallelly, the requirement for more memory and high computational cost albeit they achieved reasonable results. Convolutional Neural Network processes data parallelly, but, it breaks the ordering of input data, which is significant to build an effective model for human activity recognition. To overcome these challenges, this study proposes causal convolution based on performers-attention and supervised contrastive learning to entirely forego recurrent architectures, efficiently maintain the ordering of human daily activities and focus more on important timesteps of the sensors’ data. Supervised contrastive learning is integrated to learn a discriminative representation of human activities and enhance predictive performance. The proposed network is extensively evaluated for human activities using multiple datasets including wearable sensor data and smart home environments data. The experiments on three wearable sensor datasets and five smart home public datasets of human activities reveal that our proposed network achieves better results and reduces the training time compared with the existing state-of-the-art methods and basic temporal models.

When proxy-driven learning is no better than random: The consequences of representational incompleteness.

Machine learning is widely used for personalisation, that is, to tune systems with the aim of adapting their behaviour to the responses of humans. This tuning relies on quantified features that capture the human actions, and also on objective functions—that is, proxies – that are intended to represent desirable outcomes. However, a learning system’s representation of the world can be incomplete or insufficiently rich, for example if users’ decisions are based on properties of which the system is unaware. Moreover, the incompleteness of proxies can be argued to be an intrinsic property of computational systems, as they are based on literal representations of human actions rather than on the human actions themselves; this problem is distinct from the usual aspects of bias that are examined in machine learning literature. We use mathematical analysis and simulations of a reinforcement-learning case study to demonstrate that incompleteness of representation can, first, lead to learning that is no better than random; and second, means that the learning system can be inherently unaware that it is failing. This result has implications for the limits and applications of machine learning systems in human domains.

Colloquy with Juan José García-Noblejas on understanding communication from the perspective of Aristotle's Poetics

In this interview we will explore García-Noblejas’ thought on public communication, which is deeply marked by his interpretation of Aristotle’s Poetics and influenced by some contemporary thinkers. For García-Noblejas, communication is a service and a form of practical knowledge with five dimensions: ethics, politics, rhetoric, aesthetics, and poetics. Poetics is the most important of these, since its symbolic perspective of the representation of human actions allows us to deeply understand the narrative and dramatic texts typical of journalism, advertising, and propaganda, as well as written and audiovisual fiction. Poetics is the art of creating “possible worlds” in which the reader/spectator recognizes himself (or herself) and with which he dialogues starting from his own identity. García-Noblejas defends public communication that is based on the gift of self and looks to community fruitfulness. He also discusses some frequent pathologies in the media. Finally, he proposes his theory of the Second Navigation to discern the meaning of a text and explains why it is necessary to give a transcendent meaning to the Aristotelian concept of catharsis.

When Measurement Misleads

The discipline of information retrieval (IR) has a long history of examination of how best to measure performance. In particular, there is an extensive literature on the practice of assessing retrieval systems using batch experiments based on collections and relevance judgements. However, this literature has only rarely considered an underlying principle: that measured scores are inherently incomplete as a representation of human activity, that is, there is an innate gap between measured scores and the desired goal of human satisfaction. There are separate challenges such as poor experimental practices or the shortcomings of specific measures, but the issue considered here is more fundamental - straightforwardly, in batch experiments the human-machine gap cannot be closed. In other disciplines, the issue of the gap is well recognised and has been the subject of observations that provide valuable perspectives on the behaviour and effects of measures and the ways in which they can lead to unintended consequences, notably Goodhart's law and the Lucas critique. Here I describe these observations and argue that there is evidence that they apply to IR, thus showing that blind pursuit of performance gains based on optimisation of scores, and analysis based solely on aggregated measurements, can lead to misleading and unreliable outcomes.

Two distinct networks containing position-tolerant representations of actions in the human brain.

Humans can recognize others' actions in the social environment. This action recognition ability is rarely hindered by the movement of people in the environment. The neural basis of this position tolerance for observed actions is not fully understood. Here, we aimed to identify brain regions capable of generalizing representations of actions across different positions and investigate the representational content of these regions. In a functional magnetic resonance imaging experiment, participants viewed point-light displays of different human actions. Stimuli were presented in either the upper or the lower visual field. Multivariate pattern analysis and a surface-based searchlight approach were employed to identify brain regions that contain position-tolerant action representation: Classifiers were trained with patterns in response to stimuli presented in one position and were tested with stimuli presented in another position. Results showed above-chance classification in the left and right lateral occipitotemporal cortices, right intraparietal sulcus, and right postcentral gyrus. Further analyses exploring the representational content of these regions showed that responses in the lateral occipitotemporal regions were more related to subjective judgments, while those in the parietal regions were more related to objective measures. These results provide evidence for two networks that contain abstract representations of human actions with distinct representational content.