Skeleton-based Model Research Articles

Spatiotemporal analysis using deep learning and fuzzy inference for evaluating broiler activities

Observing poultry activity is crucial for assessing their health status; however, the inspection process is often time-consuming and labor-intensive, particularly in cases involving large numbers of chickens. Inexperienced breeders may also misjudge their activity levels, potentially missing opportunities for prevention and treatment. This study integrates traditional video surveillance with an advanced monitoring system to identify various broiler behaviors in a breeding environment. A two-stage deep learning approach is employed: in the first stage, the broilers are detected, and in the second stage, five key body points (head, abdomen, two legs, and tail) are identified. A skeleton-based model is then developed centered around the abdomen, with six angles calculated using trigonometric methods. These angles are analyzed by a long short-term memory network to estimate behaviors such as “Standing”, “Walking”, “Resting”, “Eating”, “Preening”, and “Flapping”, selecting the behavior with the highest probability. Dual-layer fuzzy logic inference systems were used to evaluate the proportion of time broilers spent in static versus dynamic states, providing a robust determination of their activity levels. Validated in a mixed-sex breeding environment, the proposed system achieved accuracies of at least 85.2% for identifying broiler type, 79.2% for identifying body parts, and 50.8% for identifying behaviors. The activity level evaluation results were consistent with those conducted by experienced poultry experts.

Enhanced HAR using Dynamic STGAT

Action recognition has seen significant advancements with the integration of spatio-temporal representations, particularly leveraging skeleton-based models and cross-modal data fusion techniques. However, existing approaches face challenges in capturing long- range dependencies within the human body skeleton and effectively balancing features from diverse modalities. To address these limitations, a novel framework, the Dynamic Spatio-Temporal Graph Attention Transformer (D-STGAT), is proposed, which seamlessly integrates the strengths of dynamic graph attention mechanisms and transformer architectures for enhanced action recognition. The framework builds upon recent innovations in graph attention networks (GAT) and transformer models. First, the Spatial-Temporal Dynamic Graph Attention Network (ST-DGAT) is introduced, extending traditional GAT by incorporating a dynamic attention mechanism to capture spatial- temporal patterns within skeleton sequences. By reordering the weighted vector operations in GAT, the approach achieves a global approximate attention function, significantly enhancing its expressivity and capturing long-distance dependencies more effectively than static attention mechanisms. Furthermore, to address the challenges of cross-modal feature representation and fusion, the spatio-temporal Cross Attention Transformer (ST-CAT) is introduced. This model efficiently integrates spatio-temporal information from both video frames and skeleton sequences by employing a combination of full spatio-temporal attention (FAttn), zigzag spatio-temporal attention (ZAttn), and binary spatio-temporal attention (BAttn) modules. Through the proper arrangement of these modules within the transformer encoder and decoder, ST-CAT learns a multi-feature representation that effectively captures the intricate spatiotemporal dynamics inherent in action recognition tasks. Experimental results on the Penn- Action, NTU-RGB+D 60, and 120 datasets showcase the efficacy of the approach, yielding promising performance improvements over previous state-of-the-art methods. In summary, the proposed D-STGAT and ST-CAT frameworks offer novel solutions for action recognition tasks by leveraging dynamic graph attention mechanisms and transformer architectures to effectively capture and fuse spatiotemporal features from diverse modalities, leading to superior performance compared to existing approaches.

A robust multimodal detection system: physical exercise monitoring in long-term care environments.

Falls are a major cause of accidents that can lead to serious injuries, especially among geriatric populations worldwide. Ensuring constant supervision in hospitals or smart environments while maintaining comfort and privacy is practically impossible. Therefore, fall detection has become a significant area of research, particularly with the use of multimodal sensors. The lack of efficient techniques for automatic fall detection hampers the creation of effective preventative tools capable of identifying falls during physical exercise in long-term care environments. The primary goal of this article is to examine the benefits of using multimodal sensors to enhance the precision of fall detection systems. The proposed paper combines time-frequency features of inertial sensors with skeleton-based modeling of depth sensors to extract features. These multimodal sensors are then integrated using a fusion technique. Optimization and a modified K-Ary classifier are subsequently applied to the resultant fused data. The suggested model achieved an accuracy of 97.97% on the UP-Fall Detection dataset and 97.89% on the UR-Fall Detection dataset. This indicates that the proposed model outperforms state-of-the-art classification results. Additionally, the proposed model can be utilized as an IoT-based solution, effectively promoting the development of tools to prevent fall-related injuries.

Estimation of skeletal kinematics in freely moving rodents

Forming a complete picture of the relationship between neural activity and skeletal kinematics requires quantification of skeletal joint biomechanics during free behavior; however, without detailed knowledge of the underlying skeletal motion, inferring limb kinematics using surface-tracking approaches is difficult, especially for animals where the relationship between the surface and underlying skeleton changes during motion. Here we developed a videography-based method enabling detailed three-dimensional kinematic quantification of an anatomically defined skeleton in untethered freely behaving rats and mice. This skeleton-based model was constrained using anatomical principles and joint motion limits and provided skeletal pose estimates for a range of body sizes, even when limbs were occluded. Model-inferred limb positions and joint kinematics during gait and gap-crossing behaviors were verified by direct measurement of either limb placement or limb kinematics using inertial measurement units. Together we show that complex decision-making behaviors can be accurately reconstructed at the level of skeletal kinematics using our anatomically constrained model.

Fusing Skeleton Recognition With Face-TLD for Human Following of Mobile Service Robots

Target recognition is a challenging task for human following of mobile service robots. In this paper, we combine the principal-component-analysis (PCA)-based face recognition with the tracking-learning-detection applied to the human face (Face-TLD) to obtain an improvement, named as IFace-TLD. The proposed IFace-TLD can significantly improve the discrimination ability of the Face-TLD for ambiguous facial appearances. To further deal with motion uncertainties of the human head, especially the sudden motion change, which makes face-based target recognition methods unstable or even loses the target, a skeleton-based model is introduced to improve the accuracy and robustness of the target recognition. Specifically, within a walk half-cycle, the skeleton features are extracted from the upper-body three-dimensional skeleton coordinates. Then, the extracted skeleton features are fed into the support vector data description (SVDD) to identify the target person when the IFace-TLD becomes invalid. The seamless fusion of the skeleton recognition and the IFace-TLD, named as the SIFace-TLD, significantly enhances the robustness in complex scenarios, especially for people tracking from both front and behind. To achieve a complete human following system, the particle filter (PF) is adopted for estimating the state of the human motion. And then, a controller is designed to maintain the relative position between the robot and the target. Experimental results demonstrate that the proposed IFace-TLD is more accurate and flexible than the original Face-TLD. And the SIFace-TLD shows a robust performance to human motion uncertainties. Moreover, the developed controller can achieve a satisfactory human following performance.

Siamese denoising autoencoders for joints trajectories reconstruction and robust gait recognition

Dynamics of body skeletons convey significant information for human gait recognition. However, it is inevitable that missing points, overlapping error, or confusion of left and right error will frequently occur during the process of skeleton estimation. Existing skeleton-based methods have difficulty in achieving satisfactory performance in gait recognition since they treat the noisy data and the normal data equally to the recognition process. In this paper, we propose a novel skeleton-based model called Siamese Denoising Autoencoder networks (Siamese DAE), which can automatically learn to remove position noise, recover missing skeleton points and correct outliers in joint trajectories. More precisely, we construct an encoder that compresses the characteristics of input trajectories into a latent space and a decoder that attempts to reconstruct more accurate skeleton trajectories from the latent feature. The corrected joint trajectories not only lead to higher discriminative power but also stronger generalization capability. Moreover, we design a Siamese structure to reduce intra-class variations and increase inter-class variations of the encoded features. Experiments demonstrate that our method enhances the robustness against inaccurate skeleton estimation and achieves substantial improvements over mainstream skeleton-based methods for gait recognition.

A Mobility Model for Wearable Antennas on Dynamic Users

This paper presents a mobility model for the variations in position and orientation of wearable antennas on dynamic users, considering walking and running motions. Motion is represented as a composition of a linear forward movement plus a periodic component, modeled by a Fourier series with up to two harmonics. The model is simple, yet realistic, as Motion Capture (MoCap) data are used to calculate its parameters. It is suitable for use with a variety of propagation channel models, including deterministic ray-tracing and stochastic geometry-based ones, but can also allow for analytical inference in simplified scenarios. Considering an off-body communication scenario, simulations show that the proposed mobility model provides similar received power as the skeleton-based model with MoCap data, the maximum difference in the considered scenario being below 1 dB. A significant influence of user’s motion on the channel is observed for both free-space and multipath propagation, yielding received power variations up to 28 dB in the considered scenarios.

N-ary implicit blends with topology control

Constructive implicit surfaces are attractive for modeling and animation because they seamlessly handle shapes with complex and dynamic topology. However, the way they merge shapes is difficult to control. This paper introduces a solution: an improved blend operator that provides control over how topology changes are handled. It is based on a correction applied to the standard blending operator: the sum. Building on summation preserves the n-ary nature of the blend, providing the simplicity of arbitrary (e.g. flat) construction trees and segmentation invariance. The correction is based on projection to a reference case in the variation-space defined by the field and the norm of its gradient. It provides a single parameter, allowing for tuning behavior to achieve effects ranging from avoiding topological combination, through merging only during overlap, to merging at a distance. Dynamic adjustment of the parameter allows for context-dependent effects. Applications range from skeleton-based modeling, where shapes keep the topology of their skeleton, to objects that change topology during animation, with controllable merging. We illustrate the latter with Manga-style hair, where merging depends on the angle between hair wisps.

Fully automatic brain segmentation using model-guided level sets and skeleton-based models

A fully automatic brain segmentation method is presented. First the skull is stripped using a model-based level set on T1-weighted inversion recovery images, then the brain ventricles and basal ganglia are segmented using the same method on T1-weighted images. The central white matter is segmented using a regular level set method but with high curvature regulation. To segment the cortical gray matter, a skeleton-based model is created by extracting the mid-surface of the gray matter from a preliminary segmentation using a threshold-based level set. An implicit model is then built by defining the thickness of the gray matter to be 2.7 mm. This model is incorporated into the level set framework and used to guide a second round more precise segmentation. Preliminary experiments show that the proposed method can provide relatively accurate results compared with the segmentation done by human observers. The processing time is considerably shorter than most conventional automatic brain segmentation methods.

Detecting shapes in noise: the role of contour-based and region-based representations

We investigated the detection of shapes (closed contours) in noise as a function of their complexity. As in previous work (VSS2012), we quantified complexity in several ways reflecting alternate shape-generating models. In a contour-based model, complexity is defined as the integrated surprisal along the contour, that is, the summed negative log probability of the sequence of turning angles defining it. Alternatively, in a skeleton-based model of shape, complexity is defined as the surprisal of the shape given its generating skeleton, that is, the negative log probability of the shape's boundary conditioned on an estimated skeleton. The two models have the same basic probabilistic conception but differ in assumed generating model. In VSS2012 we applied these measures to natural shapes (animals and leaves), but these shapes necessarily result in an irregular and sparse sampling of the shape space. Here we present a series of more controlled experiments in which we manipulated the structure of the target shape in order to distinguish the potentially separate effects of contour complexity and different components of skeleton-based shape complexity, such as the log prior and log likelihood. Subjects detected closed contours embedded in background noise (2IFC task). We parametrically varied several shape factors, including the number of parts in the shape, and the variability of the contour around its internal skeleton (hence the amplitude of contour noise). The number of parts influences the prior probability of the skeleton, while the variability influences the complexity under the contour model as well as the likelihood under the skeletal model. Both factors show significant influences on shape detectability, providing evidence for both contour-based and region-based representational formats. The results shed light on basic processes of perceptual organization, including object segmentation (extracting whole shapes from cluttered scenes) and shape representation. Meeting abstract presented at VSS 2013

A tract-specific framework for white matter morphometry combining macroscopic and microscopic tract features.

Diffusion tensor imaging plays a key role in our understanding of white matter both in normal populations and in populations with brain disorders. Existing techniques focus primarily on using diffusivity-based quantities derived from diffusion tensor as surrogate measures of microstructural tissue properties of white matter. In this paper, we describe a novel tract-specific framework that enables the examination of white matter morphometry at both the macroscopic and microscopic scales. The framework leverages the skeleton-based modeling of sheet-like white matter fasciculi using the continuous medial representation, which gives a natural definition of thickness and supports its comparison across subjects. The thickness measure provides a macroscopic characterization of white matter fasciculi that complements existing analysis of microstructural features. The utility of the framework is demonstrated in quantifying white matter atrophy in Amyotrophic Lateral Sclerosis, a severe neurodegenerative disease of motor neurons. We show that, compared to using microscopic features alone, combining the macroscopic and microscopic features gives a more complete characterization of the disease.

Game Technology for Training and Education

Serious games are being more and more deployed in such diverse areas as public awareness, military training, and higher education. One of the driving forces behind this stems from the rapidly growing availability of game technologies, providing not only better, faster, and more realistic graphics, physics, and animations, but above all making “the language” of game development accessible to increasingly more people. The articles in this special issue are a rich sampling of how game technology can be creatively put to service within games for training and education purposes. The first two articles are focused on simulation applications. In “SIDH—a game-based architecture for a training simulator,” Backlund et al. extend the notion of gamebased simulation proposing an architecture for a professional firefighter training simulator that incorporates novel visualization and interaction modes. The serious game, developed in cooperation with the government agency responsible for the training of fire and rescue personnel, is a good example of how game technology helps making the delicate combination of engaging level design and carefully tuned learning objectives. In “Experiential learning in vehicle dynamics education via motion simulation and interactive gaming,” Hulme et al. present an original methodology developed within their vehicle dynamics curriculum, incorporating a combination of driving simulation, motion simulation, and educational practices into a game-based simulation framework. The paper also includes a comprehensive evaluation of their approach in two practical, educational scenarios. The next two articles describe specific applications of game technology to serious games aimed at computer science education. In “An application of game development framework in higher education,” Wang et al. present their experiences with an XNA-based game project for teaching software architecture. The paper also includes a brief survey and evaluation of game frameworks currently available for these purposes. On the other hand, in “Towards a serious game to help students learn computer programming,” Muratet et al. describe the design and development of a strategy game aimed at strengthening novice programming skills, including an interesting discussion on the mapping of learning objectives into the game play. Finally, in “Venation skeleton-based modeling plant leaf wilting,” Lu et al. present an original method for simulating and visualizing leaf wilting, useful in the domain of biology. Interestingly, their technique can be applied both in a didactic setting andwithin any graphics framework requiring realistic visualization of natural phenomena. We believe that serious games will continue to foster the development of new game technologies, and, conversely, combining and deploying existing game technology in novel ways will likely extend and amplify the impact and innovation achieved by these games in our society.

A tract-specific framework for white matter morphometry combining macroscopic and microscopic tract features.

Diffusion tensor imaging plays a key role in our understanding of white matter (WM) both in normal populations and in populations with brain disorders. Existing techniques focus primarily on using diffusivity-based quantities derived from diffusion tensor as surrogate measures of microstructural tissue properties of WM. In this paper, we describe a novel tract-specific framework that enables the examination of WM morphometry at both the macroscopic and microscopic scales. The framework leverages the skeleton-based modeling of sheet-like WM fasciculi using the continuous medial representation, which gives a natural definition of thickness and supports its comparison across subjects. The thickness measure provides a macroscopic characterization of WM fasciculi that complements existing analysis of microstructural features. The utility of the framework is demonstrated in quantifying WM atrophy in Amyotrophic Lateral Sclerosis, a severe neurodegenerative disease of motor neurons. We show that, compared to using microscopic features alone, combining the macroscopic and microscopic features gives a more holistic characterization of the disease.

BONE STIFFNESS STRONGLY DEPENDS ON TRABECULAR THICKNESS. A PARAMETRIC SKELETON-BASED FE STUDY

Metabolic bone diseases affect, among others, the structure and thickness of trabecular bone. To study the potential effects of trabecular bone thickness variations, the use of commonly used micro-finite element ( FE) models is limited; adding just one element to a trabecula may result in a 25% increase in trabecular thickness. Skeleton-based FE models, in which individual trabeculae are represented by single elements circumvent this problem. It was the aim of this study to use such models to assess the effect of trabecular thickness on bone stiffness.