Skeleton Representation Research Articles

Kinematic matrix: One-shot human action recognition using kinematic data structure

One-shot action recognition, which refers to recognizing human-performed actions using only a single training example, holds significant promise in advancing video analysis, particularly in domains requiring rapid adaptation to new actions. However, existing algorithms for one-shot action recognition face multiple challenges, including high computational complexity, limited accuracy, and difficulties in generalization to unseen actions. To address these issues, we propose a novel kinematic-based skeleton representation that effectively reduces computational demands while enhancing recognition performance. This representation leverages skeleton locations, velocities, and accelerations to formulate the one-shot action recognition task as a metric learning problem, where a model projects kinematic data into an embedding space. In this space, actions are distinguished based on Euclidean distances, facilitating efficient nearest-neighbour searches among activity reference samples. Our approach not only reduces computational complexity but also achieves higher accuracy and better generalization compared to existing methods. Specifically, our model achieved a validation accuracy of 78.5%, outperforming state-of-the-art methods by 8.66% under comparable training conditions. These findings underscore the potential of our method for practical applications in real-time action recognition systems.

Language-guided temporal primitive modeling for skeleton-based action recognition

Human actions describe the complex body dynamics, which are defined by an ordered set of sub-action primitives. Existing methods in skeleton-based action recognition primarily focus on designing various networks to learn the entire action representation, in alignment with the action label. However, training the heterogeneous action as a whole increases the burden on the network and is not conducive to learning the temporal structure of the action. In this paper, the traditional action label is extended into the ordered primitive language description by employing a large language model, which not only unveils the temporal structure of the action but also enriches its connotation. Based on this, we propose a language-guided skeleton representation learning network (LGS-Net) that leverages the language description to guide the skeleton feature learning from the ordered primitive and global action levels. Especially, the primitive guidance aligns the features of skeleton clips with those of the ordered primitive descriptions while preserving temporal order, which promotes the network to model the temporal primitive and capture the internal structure within the action from skeletons. To enhance the cross-modal alignment performance, we develop an innovative temporal alignment loss supplemented with diversity and sparsity regularization terms to generate the discriminative multi-modal representation. Evaluated on three benchmark datasets, NTU-60, NTU-120 and N-UCLA, the proposed LGS-Net achieves comparable results to state-of-the-art methods, which proves the effectiveness of the language-guided learning mechanism.

Versatile Graph Neural Networks Toward Intuitive Human Activity Understanding.

Benefiting from the advanced human visual system, humans naturally classify activities and predict motions in a short time. However, most existing computer vision studies consider those two tasks separately, resulting in an insufficient understanding of human actions. Moreover, the effects of view variations remain challenging for most existing skeleton-based methods, and the existing graph operators cannot fully explore multiscale relationship. In this article, a versatile graph-based model (Vers-GNN) is proposed to deal with those two tasks simultaneously. First, a skeleton representation self-regulated scheme is proposed. It is among the first trials that successfully integrate the idea of view adaptation into a graph-based human activity analysis system. Next, several novel graph operators are proposed to model the positional relationships and learn the abstract dynamics between different human joints and parts. Finally, a practical multitask learning framework and a multiobjective self-supervised learning scheme are proposed to promote both the tasks. The comparative experimental results show that Vers-GNN outperforms the recent state-of-the-art methods for both the tasks, with the to date highest recognition accuracies on the datasets of NTU RGB + D (CV: 97.2%), UWA3D (88.7%), and CMU (1000 ms: 1.13).

Early gesture detection in untrimmed streams: A controlled CTC approach for reliable decision-making

This paper focuses on the problem of online action detection for interactive systems, with a special emphasis on earliness. Online Action Detection (OAD) refers to the challenging task of recognizing gestures in untrimmed, streaming videos where the actions occur in unpredictable orders and durations. To address these challenges, we present a skeleton-based system for OAD incorporating a decision mechanism to accurately detect ongoing gestures. This allows us to provide instance-level output, achieving a high level of stream understanding. This mechanism relies on a novel Connectionist Temporal Classification (CTC) loss design that restricts the path possibilities according to the action boundaries. We also present a mechanism to tune the trade-off between accuracy and earliness according to the needs of the interactive system using a weighted label prior. This system includes a 3D CNN network, referred to as DOLT-C3D, exploiting the spatial–temporal information provided by the euclidean skeleton representation. We extensively evaluate our approach on eight publicly available datasets, demonstrating its superior performance compared to state-of-the-art methods in terms of both accuracy and earliness. We also successfully applied our approach to early 2D gestures detection. Furthermore, our system shows real-time performance, making it a suitable choice for interactive systems.

Object Knowledge Integration for Skeleton-Based Action Recognition

3D-Skeleton-based action recognition has been widely adopted due to its efficiency and robustness to complex backgrounds. While it is capable of conveying a significant amount of information regarding the dynamics of human poses, we argue that its performance is curtailed when confronted with actions involving interactions between humans and objects due to the absence of the study of the surrounding objects. It is of great importance to delve deeper into the study of human-object interactions for skeleton-based action recognition. This paper proposes a novel approach to represent the spatial-temporal skeleton features, along with the present nearby objects and their dynamics. To accomplish this, a new formulation named object knowledge is presented, which entails the categorization of object characteristics, based on whether or not the object necessitates a motion analysis. With a piece of prior knowledge, in cases where it is required, the motion is calculated, while for those where it is not necessary, only the category of object is considered. This object knowledge is then early-fusion along with the skeleton representation, in such a way that it fits into the self-attention model. The experimental results on different popular action recognition datasets (NTU RGB+D 60, NTU RGB+ D 120 illustrate that the proposed approach outperforms the current state-of-the-art methods.

Efficient shrub modelling based on terrestrial laser scanning (TLS) point clouds

ABSTRACT This paper presents a methodology for automatically generating 3D models of individual shrubs from point clouds acquired by Terrestrial Laser Scanning (TLS) devices. The creation of accurate 3D models for shrubs holds significant value for ecological studies. The shrub modelling is more challenging compared to other tree modelling tasks, due to the presence of substantial noise and fuzzy branch structures. To address these challenges, our approach first partitions the input point cloud into two categories: branches and others. This point segmentation process effectively eliminates interference of leaves and noise, thereby enhancing the visibility of branch skeletons. Then, we construct a triangulation network using the branch points and establish a minimum spanning tree (MST) based on this network. Serving as the initial skeleton representation of the plant, the MST preserves the essential topological structures of the branches. Within the extracted MST, we implement a recursive trimming technique to eliminate redundant branches by merging adjacent points and edges, ultimately consolidating the skeleton structure. Finally, we employ an adaptive cylinder fitting algorithm that relies on robust principal component analysis (RPCA) to generate the model. The effectiveness and robustness of the proposed method are demonstrated via experiments on different datasets, with an average fitting error of 1.13 cm, indicating that our approach can achieve high accuracy in modelling individual shrubs.

Vision based crop row navigation under varying field conditions in arable fields

Accurate crop row detection is often challenged by the varying field conditions present in real-world arable fields. Traditional colour based segmentation is unable to cater for all such variations. The lack of comprehensive datasets in agricultural environments limits the researchers from developing robust segmentation models to detect crop rows. We present a dataset for crop row detection with 11 field variations from sugar beet and maize crops. We also present a novel crop row detection algorithm for visual servoing in crop row fields. The proposed method uses deep learning based crop row skeleton segmentation method followed by a crop row scanning algorithm that identifies the central crop row which the robot then follows. The unique dataset we used with skeleton representations for crop row detection enables robust crop row detection in challenging real world field conditions. Our algorithm can detect crop rows against varying field conditions such as curved crop rows, weed presence, discontinuities, growth stages, tramlines, shadows and light levels. Dense weed presence within inter-row space and discontinuities in crop rows were the most challenging field conditions for our crop row detection algorithm. An End-of Row detector algorithm was developed to detect the end of the crop row and navigate the robot towards the headland area when it reaches the end of the crop row.

SSRL: Self-Supervised Spatial-Temporal Representation Learning for 3D Action Recognition

For 3D action recognition, the main challenge is to extract long-range semantic information in both temporal and spatial dimensions. In this paper, in order to better excavate long-range semantic information from large number of unlabelled skeleton sequences, we propose Self-supervised Spatial-temporal Representation Learning (SSRL), a contrastive learning framework to learn skeleton representation. SSRL consists of two novel inference tasks that enable the network to learn global semantic information in the temporal and spatial dimensions, respectively. The temporal inference task learns the temporal persistence of human actions through temporally incomplete skeleton sequences. And the spatial inference task learns the spatially coordinated nature of human action through spatially partially skeleton sequence. We design two transformation modules to efficiently realize these two tasks while fitting the encoder network. To avoid the difficulty of constructing and maintaining high-quality negative samples, our proposed framework learns by maintaining consistency among positive samples without the need of any negative sample. Experiments demonstrate that our proposed method can achieve better results in comparison with state-of-the-art methods under a variety of evaluation protocols on NTU RGB+D 60, PKU-MMD and NTU RGB+D 120 datasets.

Self-Supervised 3D Action Representation Learning With Skeleton Cloud Colorization.

3D Skeleton-based human action recognition has attracted increasing attention in recent years. Most of the existing work focuses on supervised learning which requires a large number of labeled action sequences that are often expensive and time-consuming to annotate. In this paper, we address self-supervised 3D action representation learning for skeleton-based action recognition. We investigate self-supervised representation learning and design a novel skeleton cloud colorization technique that is capable of learning spatial and temporal skeleton representations from unlabeled skeleton sequence data. We represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. Specifically, we design a two-steam pretraining network that leverages fine-grained and coarse-grained colorization to learn multi-scale spatial-temporal features. In addition, we design a Masked Skeleton Cloud Repainting task that can pretrain the designed auto-encoder framework to learn informative representations. We evaluate our skeleton cloud colorization approach with linear classifiers trained under different configurations, including unsupervised, semi-supervised, fully-supervised, and transfer learning settings. Extensive experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins and achieves competitive performance in supervised 3D action recognition as well.

A robust approach to 3D neuron shape representation for quantification and classification

We consider the problem of finding an accurate representation of neuron shapes, extracting sub-cellular features, and classifying neurons based on neuron shapes. In neuroscience research, the skeleton representation is often used as a compact and abstract representation of neuron shapes. However, existing methods are limited to getting and analyzing “curve” skeletons which can only be applied for tubular shapes. This paper presents a 3D neuron morphology analysis method for more general and complex neuron shapes. First, we introduce the concept of skeleton mesh to represent general neuron shapes and propose a novel method for computing mesh representations from 3D surface point clouds. A skeleton graph is then obtained from skeleton mesh and is used to extract sub-cellular features. Finally, an unsupervised learning method is used to embed the skeleton graph for neuron classification. Extensive experiment results are provided and demonstrate the robustness of our method to analyze neuron morphology.

Geometry-Aware Planar Embedding of Treelike Structures

The growing complexity of spatial and structural information in 3D data makes data inspection and visualization a challenging task. We describe a method to create a planar embedding of 3D treelike structures using their skeleton representations. Our method maintains the original geometry, without overlaps, to the best extent possible, allowing exploration of the topology within a single view. We present a novel camera view generation method which maximizes the visible geometric attributes (segment shape and relative placement between segments). Camera views are created for individual segments and are used to determine local bending angles at each node by projecting them to 2D. The final embedding is generated by minimizing an energy function (the weights of which are user adjustable) based on branch length and the 2D angles, while avoiding intersections. The user can also interactively modify segment placement within the 2D embedding, and the overall embedding will update accordingly. A global to local interactive exploration is provided using hierarchical camera views that are created for subtrees within the structure. We evaluate our method both qualitatively and quantitatively and demonstrate our results by constructing planar visualizations of line data (traced neurons) and volume data (CT vascular and bronchial data).

Generative design of space frames for additive manufacturing technology

A generative design methodology is presented that solves for minimum volume and compliance space-frame systems, with consideration of stress and buckling constraints. The solution space is explored using formal topology optimisation routines. A parameterisation method converts voxelised topology optimisation solutions into skeletonised connectivity representations. An inequality constrained gradient descent optimisation method optimises and defines cross-sectional geometry. This enables fast and automatic solution generation, providing designers with sets of high-performing problem solutions. Skeleton representations provide an inexpensive modelling tool for parallel analysis of physical problems difficult to model using topology optimisation. Geometry is represented using traditional engineering cross-sections with well understood behaviour. This improves certainty in the performance of solutions, simplifying certification processes. The generative design of a structural aerospace bracket for additive manufacture has been used as a case study within this research. A design of experiments produced 360 topology optimisation results, altering input variables and discretisation resolution to identify their effect on solution outcomes and the performance of parameterisation. The proposed method was found to robustly generate high-performing solutions utilising low-resolution topology optimisation. Additionally, 6 high-performing topologies were identified, providing designers with a set of solutions to select from. Limitations on the parameterisation process were identified, with topology optimisation solutions with volume fractions above 0.2 not parameterising successfully, and simulations with a resolution of 5 mm element size and below generating low performing skeletonised topologies.

Self-Supervised Learning for Multilevel Skeleton-Based Forgery Detection via Temporal-Causal Consistency of Actions

Skeleton-based human action recognition and analysis have become increasingly attainable in many areas, such as security surveillance and anomaly detection. Given the prevalence of skeleton-based applications, tampering attacks on human skeletal features have emerged very recently. In particular, checking the temporal inconsistency and/or incoherence (TII) in the skeletal sequence of human action is a principle of forgery detection. To this end, we propose an approach to self-supervised learning of the temporal causality behind human action, which can effectively check TII in skeletal sequences. Especially, we design a multilevel skeleton-based forgery detection framework to recognize the forgery on frame level, clip level, and action level in terms of learning the corresponding temporal-causal skeleton representations for each level. Specifically, a hierarchical graph convolution network architecture is designed to learn low-level skeleton representations based on physical skeleton connections and high-level action representations based on temporal-causal dependencies for specific actions. Extensive experiments consistently show state-of-the-art results on multilevel forgery detection tasks and superior performance of our framework compared to current competing methods.

Neural Network-Based Method for Early Diagnosis of Autism Spectral Disorder Head-Banging Behavior from Recorded Videos

Autism spectrum disorder (ASD) is a mental developmental disorder associated with social and communicational defects and Stereotypical Motor Movements (SMM). SMM is a set of repetitive motor activities associated with several mental developmental disorders like Autism. SMM has several forms like arm flapping, head banging, ear covering, and spinning with various degrees of severity that might lead to self-injury in severe cases. Developing a computer-vision-based technology to detect noticeable SMM behaviors can help in the early diagnosis of autism. In this paper, a computer vision-based neural network model was proposed to detect and recognize repetitive motor behaviors. The proposed model went through three main stages: First, data preparation. Second, human body features extraction using deep learning pose estimation and the skeleton representation model, and finally, multiclass classification to distinguish between several classes of headbanging. The proposed solution was evaluated using the Self Stimulatory Behavior Dataset (SSBD) which is a public dataset of three classes of repetitive motor behaviors associated with autism. We also collected a set of 40 videos of autistic children exhibiting headbanging from public domains like YouTube. In addition to that, we captured 25 videos of typically developing subjects mimicking headbanging. The collected and the videoed videos were used to evaluate the proposed model. This work proves the applicability of diagnosing mental developmental syndrome symptoms using vision-based techniques in cooperation with neural networks. The produced results prove that the used techniques can operate well in real-world challenging applications. The proposed model achieved 85.5% accuracy on SSBD and 93% on the collected and recorded videos.

Efficient dual-scale flow simulation for Resin Transfer Molding process based on domains skeletonization

In the Resin Transfer Molding (RTM) process, a polymeric resin is injected inside a dry preform to fill the gaps around and inside the fiber tows. Simulating this process at the scale of the tows is challenging because of the computational cost associated to solving a three-dimensional dual-scale flow problem. In this work, a novel Dual-Scale Skeleton model (DSS) is introduced, capable of solving a dual-scale flow problem at an affordable computational cost. The three-dimensional geometry of a multi-layer layup, consisting of inter-tow channels and permeable tows, is replaced by a skeletonized representation of the original subdomains. Dual-scale flow is modeled using a Reynolds-Darcy finite elements formulation. The model is validated numerically and its application is demonstrated over a few test cases. The adoption of the DSS model allows one to simulate complex dual-scale flow problems over large domains at a reduced computational cost when compared to full 3D solutions.

Skeleton-CutMix: Mixing Up Skeleton With Probabilistic Bone Exchange for Supervised Domain Adaptation.

We present Skeleton-CutMix, a simple and effective skeleton augmentation framework for supervised domain adaptation and show its advantage in skeleton-based action recognition tasks. Existing approaches usually perform domain adaptation for action recognition with elaborate loss functions that aim to achieve domain alignment. However, they fail to capture the intrinsic characteristics of skeleton representation. Benefiting from the well-defined correspondence between bones of a pair of skeletons, we instead mitigate domain shift by fabricating skeleton data in a mixed domain, which mixes up bones from the source domain and the target domain. The fabricated skeletons in the mixed domain can be used to augment training data and train a more general and robust model for action recognition. Specifically, we hallucinate new skeletons by using pairs of skeletons from the source and target domains; a new skeleton is generated by exchanging some bones from the skeleton in the source domain with corresponding bones from the skeleton in the target domain, which resembles a cut-and-mix operation. When exchanging bones from different domains, we introduce a class-specific bone sampling strategy so that bones that are more important for an action class are exchanged with higher probability when generating augmentation samples for that class. We show experimentally that the simple bone exchange strategy for augmentation is efficient and effective and that distinctive motion features are preserved while mixing both action and style across domains. We validate our method in cross-dataset and cross-age settings on NTU-60 and ETRI-Activity3D datasets with an average gain of over 3% in terms of action recognition accuracy, and demonstrate its superior performance over previous domain adaptation approaches as well as other skeleton augmentation strategies.

Human Skeleton Feature Optimizer and Adaptive Structure Enhancement Graph Convolution Network for Action Recognition

Human action recognition based on the graph convolution network (GCN) is a hot topic in computer vision. Existing GCN-based methods fail to capture internal implicit information when extracting action features, thereby leading to over-smoothing in the training stage. These issues result in poor performance and inaccurate extraction of action features. To address these problems, a new GCN is constructed. In this paper, a human skeleton feature optimizer (SFO) and adaptive structure enhancement graph convolution network (ASE-GCN) for action recognition are proposed in an end-to-end manner. To obtain discriminative features, the SFO is proposed to construct a new skeleton representation for action recognition through the connection criterion, which extracts the internal implicit information of action. The action feature of the joint coordinates is extracted by graph structure mask (GSM), directed graph mapping (DGM), and adaptive pooling operation (APO) in the proposed ASE-GCN network. The GSM acts as the regularizer of skeleton structure information to strengthen the representation of the graph structure. The DGM correlates the directed graph with human motion information through kinematic principle, and the APO strengthens the global high-frequency features to alleviate over-smoothing. The proposed method achieves comparable or superior results over state-of-the-art methods when used in experiments on two large public-scale datasets, NTU-RGB+D and Kinetics.

Adaptive Spatiotemporal Representation Learning for Skeleton-Based Human Action Recognition

How do humans recognize an action or an interaction in the real world? Due to the diversity of viewing perspectives, it is a challenge for humans to identify a regular activity when they observe it from an uncommon perspective. We argue that discriminative spatiotemporal information remains an essential cue for human action recognition. Most existing skeleton-based methods learn optimal representation based on the human-crafted criterion that requires many labeled data and much human effort. This article introduces adaptive skeleton-based neural networks to learn optimal spatiotemporal representation automatically through a data-driven manner. First, an adaptive skeleton representation transformation method (ASRT) is proposed to model view-variation data without hand-crafted criteria. Next, powered by a novel attentional LSTM (C3D-LSTM) encapsulated with 3-D-convolution, the proposed model could effectively enable memory blocks to learn short-term frame dependency and long-term relations. Hence, the proposed model can more accurately understand long-term or complex actions. Furthermore, a data enhancement-driven end-to-end training scheme is proposed to train key parameters under fewer training samples. Enhanced by learned high-performance spatiotemporal representation, the proposed model achieves state-of-the-art performance on five challenging benchmarks.

Deep-Learning-Guided Student Classroom Action Understanding for Preschool Education.

A deep architecture for enhancing students' action recognition is proposed to improve preschool education. This paper seamlessly combines the teaching objectives, teaching scope, teaching implementation, and breeding evaluation status of preschool breeding practice theory. We attempt to solve the problem of effective preschool teaching, based on which we propose the simple adaptation strategies. We further evaluate the practice of preschool breeding and its effectiveness. In this way, civilized and high-quality preschool talents will be cultivated, and preschool educational experiences will be promoted. In the method of promoting the preschool culture of weak-aged children, owing to the problem that the traditional action recognition algorithm can indicate the specific students' actions, an action recognition method based on the combination of deep integration and human skeleton representation is proposed. First, the connected spatial locations and constraints are fed into a long-short-specified recall (LSTM) mode with a spatially and temporally aware algorithm which is designed to obtain spatiotemporal feature and highly separable deep joint features. Afterward, a new mechanism is introduced to resolve keyframes as well as the joints. Finally, based on the two-stream deep architecture, the effective discrimination of similar actions is achieved by integrating the color and shape features into the skeleton features by designing the deep model. Extensive experiments have demonstrated that, compared with the mainstream algorithms, this method can effectively distinguish students' action types in the classroom of homogeneous preschool children. Thus, we can substantially improve the efficiency of preschool teaching.

Sequential Gesture Learning for Continuous Labanotation Generation Based on the Fusion of Graph Neural Networks

Labanotation is a symbolic recording system for human movements, and also a powerful tool for protecting and spreading folk dances and other performing arts. State-of-the-art automatic Labanotation uses end-to-end methods with sequence-based skeleton representation, which cannot capture the relationship between joints and bones in the skeleton for accurate descriptions of continuous lower limb movements such as dance steps. In this paper, we propose a novel double-stream fusion method of directed graph neural networks (DGNN), combined with connectionist temporal classification (CTC), namely DFGNN-CTC, for sequential fine-grained motion recognition, such as the Labanotation generation of unsegmented dance movement. First, we extract double-stream directed graph feature, employing an orientation-normalized directed acyclic graph (ON-DAG) and an orientation-normalized temporal directed acyclic graph (ON-TDAG), to jointly model spatiotemporal properties of movement recorded in motion capture data. Then, we design a CTC-based fusion-pooling module to fuse the spatial and temporal streams encoded by two DGNNs. It concatenates and fuses the two streams to generate discriminative descriptions of each time step, and concentrates them to make per-time-step predictions of Laban gesture type, from which the CTC searches the optimal Laban symbol sequence, corresponding to elemental motions composing the movement. In this way, the new method enables much finer discrimination for similar Laban gestures with subtle differences in spatial and temporal properties through joint contextual spatiotemporal modeling so that it achieves much superior performance in continuous Labanotation generation to existing methods, which only have single-stream analysis either spatially or temporally. The experiments on two Labanotation-labelled motion capture datasets demonstrate the effectiveness of the components in the proposed method and its superiority comparing with the state-of-the-art methods, especially for lower limb movements.