Pose Features Research Articles

TLCSFI: A Pose-Guided Person Re-Identification Method with Two-Level Channel–Spatial Feature Integration

Person re-identification methods currently encounter challenges in feature learning, primarily due to difficulties in expressing the correlation between local features and integrating global and local features effectively. To address these issues, a pose-guided person re-identification method with Two-Level Channel–Spatial Feature Integration (TLCSFI) is proposed. In TLCSFI, a two-level integration mechanism is implemented. At the first level, TLCSFI integrates the spatial information from local features to generate fine-grained spatial features. At the second level, the fine-grained spatial feature and the coarse-grained channel feature are integrated together to complete channel–spatial feature integration. In the method, a Pose-based Spatial Feature Integration (PSFI) module is introduced to generate the pose union feature, which calculates intra-body affinity to guide the integration of spatial information among local pose feature maps. Then, a Channel and Spatial Union Feature Integration (CSUFI) module is proposed to efficiently integrate the channel information of the global feature and the spatial information of the pose union feature. Two individual networks are designed to extract channel and spatial information, respectively, in CSUFI, which are then weighted and integrated. Experiments are conducted on three publicly available datasets to evaluate TLCSFI, and the experimental results demonstrate its competitive performance.

Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose Prior

AbstractCreating plausible motions for a diverse range of characters is a long‐standing goal in computer graphics. Current learning‐based motion synthesis methods rely on large‐scale motion datasets, which are often difficult if not impossible to acquire. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we tap into this alternative data source and introduce a neural motion synthesis approach through retargeting, which generates plausible motion of various characters that only have pose data by transferring motion from one single existing motion capture dataset of another drastically different characters. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist‐created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Our code and dataset can be accessed here.

Human–object interaction detection algorithm based on graph structure and improved cascade pyramid network

Aiming at the problem of insufficient use of human–object interaction (HOI) information and spatial location information in images, we propose a human–object​ interaction detection network based on graph structure and improved cascade pyramid. This network is composed of three branches, namely, graph branch, human–object branch and human pose branch. In graph branch, we propose a Graph-based Interactive Feature Generation Algorithm (GIFGA) to address the inadequate utilization of interaction information. GIFGA constructs an initial dense graph model by taking humans and objects as nodes and their interaction relationships as edges. Then, by traversing each node, the graph model is updated to generate the final interaction features. In human pose branch, we propose an Improved Cascade Pyramid Network (ICPN) to tackle the underutilization of spatial location information. ICPN extracts human pose features and maps both the object bounding boxes and extracted human pose maps onto the global feature map to capture the most discriminative interaction-related region features within the global context. Finally, the features from the three branches are fed into a Multi-Layer Perceptron (MLP) for fusion and then classified for recognition. Experimental results demonstrate that our network achieves mAP of 54.93% and 28.69% on the V-COCO and HICO-DET datasets, respectively.

ESMformer: Error-aware self-supervised transformer for multi-view 3D human pose estimation

Multi-view 3D human pose estimation (HPE) currently faces several key challenges. Information from different viewpoints exhibits high variability due to complex environmental factors, posing difficulties in cross-view feature extraction and fusion. Additionally, multi-view 3D labeled pose data is rather scarce, and the impact of input 2D poses on 3D HPE accuracy has received little attention. To address these issues, we propose an Error-aware Self-supervised transformer framework for Multi-view 3D HPE (ESMformer). Firstly, we introduce a single-view multi-level feature extraction module to enhance pose features in individual viewpoints, which incorporates a novel relative attention mechanism for representative feature extraction at different levels. Subsequently, we develop multi-view intra-level and cross-level fusion modules to exploit spatio-temporal feature dependencies among human joints, and progressively fuse pose information from all views and levels. Furthermore, we explore an error-aware self-supervised learning strategy to reduce the model’s reliance on 3D pose annotations and mitigate the impact of incorrect 2D poses. This strategy adaptively selects reliable input 2D poses based on 3D pose prediction errors. Experiments on three popular benchmarks show that ESMformer achieves state-of-the-art results and maintains cost-effective computational complexity. Notably, ESMformer does not rely on any 3D pose annotations or prior human body knowledge, making it highly versatile and adaptable in practical applications.11The code and models are available at https://github.com/z0911k/ESMformer.

Explore human parsing modality for action recognition

AbstractMultimodal‐based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, the authors introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts while filtering out most irrelevant noise. The authors propose a new dual‐branch framework called ensemble human parsing and pose network (EPP‐Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in the graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing features via convolutional backbones. The two high‐level features will be effectively combined through a late fusion strategy for better action recognition. Extensive experiments on NTU RGB + D and NTU RGB + D 120 benchmarks consistently verify the effectiveness of our proposed EPP‐Net, which outperforms the existing action recognition methods. Our code is available at https://github.com/liujf69/EPP‐Net‐Action.

A novel lameness detection method for dairy cows based on temporal gait and spatial post features

Lameness is one of the common diseases in dairy farms, which seriously affects the health and welfare of dairy cows. The existing lameness detection methods based on computer vision have poor ability to extract spatiotemporal features, which reduces the lameness detection accuracy in dairy cows. In this paper, a spatiotemporal energy network (SEN) is proposed to detect lameness in dairy cows. It compresses the cow walking video into history energy image (HEI) and gait energy image (GEI), and then accurately extracts the temporal gait and spatial pose features of dairy cows. First, the YOLOv8n algorithm is used to remove background information from the video and segment the cow’s body. Second, the body contour sequence of the dairy cow is stacked based on the time series, and then different body parts of dairy cows are stacked on the channel to form the HEI. The HEI preserves the movement trajectory of the dairy cows. The first lameness classification model is obtained by training the MobileNetv2 network with HEI dataset. Third, the body contour sequence of the dairy cow is cropped according to the center of mass and body proportion and then stacked into an image, which is named GEI. The second lameness classification model is obtained by training the MobileNetv2 network with GEI dataset. Finally, the prediction results of the two models are fused according to the weight ratio to detect lameness in dairy cows. To verify the lameness detection effect of this method, 264 samples were randomly selected from 330 samples as the training set and the remaining 66 samples as the test set. The proposed method obtained competitive lameness detection results with classification accuracy of 96.22 %, sensitivity of 96.35 % and specificity of 98.14 %. By combining HEI and GEI, the spatiotemporal features of the cow walking are extracted more comprehensively. This method provides guidance for the lameness detection of dairy cows and reference for the gait analysis of animals.

HR-YOLO: A Multi-Branch Network Model for Helmet Detection Combined with High-Resolution Network and YOLOv5

Automatic detection of safety helmet wearing is significant in ensuring safe production. However, the accuracy of safety helmet detection can be challenged by various factors, such as complex environments, poor lighting conditions and small-sized targets. This paper presents a novel and efficient deep learning framework named High-Resolution You Only Look Once (HR-YOLO) for safety helmet wearing detection. The proposed framework synthesizes safety helmet wearing information from the features of helmet objects and human pose. HR-YOLO can use features from two branches to make the bounding box of suppression predictions more accurate for small targets. Then, to further improve the iterative efficiency and accuracy of the model, we design an optimized residual network structure by using Optimized Powered Stochastic Gradient Descent (OP-SGD). Moreover, a Laplace-Aware Attention Model (LAAM) is designed to make the YOLOv5 decoder pay more attention to the feature information from human pose and suppress interference from irrelevant features, which enhances network representation. Finally, non-maximum suppression voting (PA-NMS voting) is proposed to improve detection accuracy for occluded targets, using pose information to constrain the confidence of bounding boxes and select optimal bounding boxes through a modified voting process. Experimental results demonstrate that the presented safety helmet detection network outperforms other approaches and has practical value in application scenarios. Compared with the other algorithms, the proposed algorithm improves the precision, recall and mAP by 7.27%, 5.46% and 7.3%, on average, respectively.

Semantic-driven diffusion for sign language production with gloss-pose latent spaces alignment

Sign Language Production (SLP) aims to translate spoken language into visual sign language sequences. The most challenging process in SLP is the transformation of a sequence of sign glosses into corresponding sign poses (G2P). Existing approaches on G2P mainly focus on constructing mappings of sign language glosses to frame-level sign pose representations, while neglecting gloss is just a weak annotation of the sequence of sign poses. To address this problem, this paper proposes the semantic-driven diffusion model with gloss-pose latent spaces alignment (SDD-GPLA) for G2P. G2P is divided into two phases. In the first phase, we design the gloss-pose latent spaces alignment (GPLA) to model the sign pose latent representations with glosses dependency. In the second phase, we propose semantic-driven diffusion (SDD) with supervised pose reconstruction guidance as a mapping between the gloss and sign poses latent features. In addition, we propose the sign pose decoder (Decoderp) to progressively generate high-resolution sign poses from latent sign pose features and to guide the SDD training process. We evaluated SDD-GPLA on a self-collected dataset of Daily Chinese Sign Language (DCSL) and a public dataset called RWTH-Phoenix-Weather-2014T. Compared with the state-of-the-art G2P methods, we obtain at least 22.9% and 2.3% improvement in WER scores on the above two datasets, respectively.

Offistretch: camera-based real-time feedback for daily stretching exercises

AbstractIn this paper, we present OffiStretch, a camera-based system for optimal stretching guidance at home or in the workplace. It consists of a vision-based method for real-time assessment of the user’s body pose to provide visual feedback as interactive guidance during stretching exercises. Our method compares the users’ actual pose with a pre-trained target pose to assess the quality of stretching for a number of different exercises. We utilize angular and spatial pose features to perform this comparison for each individual exercise. The result of this pose assessment is presented to the user as real-time visual feedback on an "augmented mirror" display. As our method relies simply on a single RGB camera, it can be easily utilized in everyday training scenarios. We validate our method in a user study, comparing users’ performance and motivation in stretching when receiving audio-visual guidance on a TV screen both with and without our live feedback. While participants performed equally well in both conditions, feedback boosted their motivation to perform the exercises, highlighting its potential for increasing users’ well-being. Moreover, our results suggest that participants preferred stretching exercises with our live feedback over the condition without the feedback. Finally, an expert evaluation with professional physiotherapists reveals that further work must target improvements of the feedback to ensure correct guidance during stretching.

Cartoon copyright recognition method based on character personality action

Aiming at the problem of cartoon piracy and plagiarism, this paper proposes a method of cartoon copyright recognition based on character personality actions. This method can be used to compare the original cartoon actions with the action characteristics of pirated or copied cartoons to identify whether there is piracy or plagiarism. Firstly, an image preprocessing scheme for character extraction is designed. GrabCut interactive image segmentation algorithm was used to obtain cartoon characters, and then binarization and morphological processing were performed on the results. Secondly, a feature extraction scheme based on character profile, moving character and character pose is designed. By extracting the perimeter and area of the character contour, the length-to-width ratio of the smallest rectangle and the inclination angle of the contour, the character contour features are obtained. The three-dimensional coordinates are established by the central point position of the cartoon character in the two-dimensional image and the change of the character's zoom in and out, and the character's motion angle characteristics are calculated. By skeletonizing a character to obtain the pose characteristics, and using deburring operation to remove redundant branches, then extract the skeleton joint angle information. Finally, feature fusion is performed on the extracted features. The experimental results show that the proposed method breaks the limitation of the conventional single feature based recognition, and can better extract the character features including contour feature, motion feature and pose feature through multi-feature based extraction, so as to protect the cartoon copyright.

A self-supervised spatio-temporal attention network for video-based 3D infant pose estimation

General movement and pose assessment of infants is crucial for the early detection of cerebral palsy (CP). Nevertheless, most human pose estimation methods, in 2D or 3D, focus on adults due to the lack of large datasets and pose annotations on infants. To solve these problems, here we present a model known as YOLO-infantPose, which has been fine-tuned, for infant pose estimation in 2D. We further propose a self-supervised model called STAPose3D for 3D infant pose estimation based on videos. We employ multi-view video data during the training process as a strategy to address the challenge posed by the absence of 3D pose annotations. STAPose3D combines temporal convolution, temporal attention, and graph attention to jointly learn spatio-temporal features of infant pose. Our methods are summarized into two stages: applying YOLO-infantPose on input videos, followed by lifting these 2D poses along with respective confidences for every joint to 3D. The employment of the best-performing 2D detector in the first stage significantly improves the precision of 3D pose estimation. We reveal that fine-tuned YOLO-infantPose outperforms other models tested on our clinical dataset as well as two public datasets MINI-RGBD and YouTube-Infant dataset. Results from our infant movement video dataset demonstrate that STAPose3D effectively comprehends the spatio-temporal features among different views and significantly improves the performance of 3D infant pose estimation in videos. Finally, we explore the clinical application of our method for general movement assessment (GMA) in a clinical dataset annotated as normal writhing movements or abnormal monotonic movements according to the GMA standards. We show that the 3D pose estimation results produced by our STAPose3D model significantly boost the GMA prediction performance than 2D pose estimation. Our code is available at github.com/wwYinYin/STAPose3D.

An Algorithm for Distracted Driving Recognition Based on Pose Features and an Improved KNN

To reduce safety accidents caused by distracted driving and address issues such as low recognition accuracy and deployment difficulties in current algorithms for distracted behavior detection, this paper proposes an algorithm that utilizes an improved KNN for classifying driver posture features to predict distracted driving behavior. Firstly, the number of channels in the Lightweight OpenPose network is pruned to predict and output the coordinates of key points in the upper body of the driver. Secondly, based on the principles of ergonomics, driving behavior features are modeled, and a set of five-dimensional feature values are obtained through geometric calculations. Finally, considering the relationship between the distance between samples and the number of samples, this paper proposes an adjustable distance-weighted KNN algorithm (ADW-KNN), which is used for classification and prediction. The experimental results show that the proposed algorithm achieved a recognition rate of 94.04% for distracted driving behavior on the public dataset SFD3, with a speed of up to 50FPS, superior to mainstream deep learning algorithms in terms of accuracy and speed. The superiority of ADW-KNN was further verified through experiments on other public datasets.

Improving diversity of speech‐driven gesture generation with memory networks as dynamic dictionaries

AbstractGenerating co‐speech gestures for interactive digital humans remains challenging because of the indeterministic nature of the problem. The authors observe that gestures generated from speech audio or text by existing neural methods often contain less movement shift than expected, which can be viewed as slow or dull. Thus, a new generative model coupled with memory networks as dynamic dictionaries for speech‐driven gesture generation with improved diversity is proposed. More specifically, the dictionary network dynamically stores connections between text and pose features in a list of key‐value pairs as the memory for the pose generation network to look up; the pose generation network then merges the matching pose features and input audio features for generating the final pose sequences. To make the improvements more accurately measurable, a new objective evaluation metric for gesture diversity that can remove the influence of low‐quality motions is also proposed and tested. Quantitative and qualitative experiments demonstrate that the proposed architecture succeeds in generating gestures with improved diversity.

Spatio‐temporal attention modules in orientation‐magnitude‐response guided multi‐stream CNNs for human action recognition

AbstractThis paper introduces a new descriptor called orientation‐magnitude response maps as a single 2D image to effectively explore motion patterns. Moreover, boosted multi‐stream CNN‐based model with various attention modules is designed for human action recognition. The model incorporates a convolutional self‐attention autoencoder to represent compressed and high‐level motion features. Sequential convolutional self‐attention modules are used to exploit the implicit relationships within motion patterns. Furthermore, 2D discrete wavelet transform is employed to decompose RGB frames into discriminative coefficients, providing supplementary spatial information related to the actors actions. A spatial attention block, implemented through the weighted inception module in a CNN‐based structure, is designed to weigh the multi‐scale neighbours of various image patches. Moreover, local and global body pose features are combined by extracting informative joints based on geometry features and joint trajectories in 3D space. To provide the importance of specific channels in pose descriptors, a multi‐scale channel attention module is proposed. For each data modality, a boosted CNN‐based model is designed, and the action predictions from different streams are seamlessly integrated. The effectiveness of the proposed model is evaluated across multiple datasets, including HMDB51, UTD‐MHAD, and MSR‐daily activity, showcasing its potential in the field of action recognition.

Occlusion robust sign language recognition system for indian sign language using CNN and pose features

Occlusion robust sign language recognition system for indian sign language using CNN and pose features

An optimization method of human skeleton keyframes selection for action recognition

In the action recognition field based on the characteristics of human skeleton joint points, the selection of keyframes in the skeleton sequence is a significant issue, which directly affects the action recognition accuracy. In order to improve the effectiveness of keyframes selection, this paper proposes inflection point frames, and transforms keyframes selection into a multi-objective optimization problem based on it. First, the pose features are extracted from the input skeleton joint point data, which used to construct the pose feature vector of each frame in time sequence; then, the inflection point frames in the sequence are determined according to the flow of momentum of each body part. Next, the pose feature vectors are input into the keyframes multi-objective optimization model, with the fusion of domain information and the number of keyframes; finally, the output keyframes are input to the action classifier. To verify the effectiveness of the method, the MSR-Action3D, the UTKinect-Action and Florence3D-Action, and the 3 public datasets, are chosen for simulation experiments and the results show that the keyframes sequence obtained by this method can significantly improve the accuracy of multiple action classifiers, and the average recognition accuracy of the three data sets can reach 94.6%, 97.6% and 94.2% respectively. Besides, combining the optimized keyframes with deep learning classifier on the NTU RGB + D dataset can make the accuracies reaching 83.2% and 93.7%.

Cross-Modal Supervised Human Body Pose Recognition Techniques for Through-Wall Radar.

Through-wall radar human body pose recognition technology has broad applications in both military and civilian sectors. Identifying the current pose of targets behind walls and predicting subsequent pose changes are significant challenges. Conventional methods typically utilize radar information along with machine learning algorithms such as SVM and random forests to aid in recognition. However, these approaches have limitations, particularly in complex scenarios. In response to this challenge, this paper proposes a cross-modal supervised through-wall radar human body pose recognition method. By integrating information from both cameras and radar, a cross-modal dataset was constructed, and a corresponding deep learning network architecture was designed. During training, the network effectively learned the pose features of targets obscured by walls, enabling accurate pose recognition (e.g., standing, crouching) in scenarios with unknown wall obstructions. The experimental results demonstrated the superiority of the proposed method over traditional approaches, offering an effective and innovative solution for practical through-wall radar applications. The contribution of this study lies in the integration of deep learning with cross-modal supervision, providing new perspectives for enhancing the robustness and accuracy of target pose recognition.

Deep Semantic Graph Transformer for Multi-View 3D Human Pose Estimation

Most Graph Convolutional Networks based 3D human pose estimation (HPE) methods were involved in single-view 3D HPE and utilized certain spatial graphs, existing key problems such as depth ambiguity, insufficient feature representation, or limited receptive fields. To address these issues, we propose a multi-view 3D HPE framework based on deep semantic graph transformer, which adaptively learns and fuses multi-view significant semantic features of human nodes to improve 3D HPE performance. First, we propose a deep semantic graph transformer encoder to enrich spatial feature information. It deeply mines the position, spatial structure, and skeletal edge knowledge of joints and dynamically learns their correlations. Then, we build a progressive multi-view spatial-temporal feature fusion framework to mitigate joint depth uncertainty. To enhance the pose spatial representation, deep spatial semantic feature are interacted and fused across different viewpoints during monocular feature extraction. Furthermore, long-time relevant temporal dependencies are modeled and spatial-temporal information from all viewpoints is fused to intermediately supervise the depth. Extensive experiments on three 3D HPE benchmarks show that our method achieves state-of-the-art results. It can effectively enhance pose features, mitigate depth ambiguity in single-view 3D HPE, and improve 3D HPE performance without providing camera parameters. Codes and models are available at https://github.com/z0911k/SGraFormer.

Pose-Aware Facial Expression Recognition Assisted by Expression Descriptions

Although expression descriptions provide additional information about facial behaviors despite of different poses, and pose features are beneficial to adapt to pose variety, neither has been fully leveraged in facial expression recognition. This paper proposes a pose-aware text-assisted facial expression recognition method using cross-modality attention. Specifically, the method contains three components. The pose feature extractor extracts pose-related features from facial images, and then cooperates with a fully-connected layer for pose classification. When poses can be clearly discriminated and classified, features obtained from the extractor can represent the corresponding poses. To eliminate bias due to appearance and illumination, cluster centers are taken as the final pose features. The text feature extractor obtains embeddings from expression descriptions. These descriptions are first passed through Intra-Exp attention to obtain preliminary embeddings. To leverage the correlations among expressions, all expression embeddings are then concatenated and passed through Inter-Exp attention. The cross-modality module attempts to learn attention maps that distinguish the importance of facial regions by using prior knowledge about poses and expression descriptions. The image features weighted by the attention maps are utilized to recognize pose and expression jointly. Experiments on three benchmark datasets demonstrate the superiority of the proposed method.

Research on the Design and Practical Effectiveness of Intelligent Teaching Mode Design for the Integration of Wushu and Ethnic Traditional Sports Culture

Abstract As an important part of Chinese culture, national sports culture not only has profound cultural connotations but also has rich cultural value. This paper proposes a path that integrates national sports culture and martial arts into teaching, focusing on three key aspects: diversification, scientification, and digitization. The neural network operator Involution improves the recognition accuracy of the human pose estimation model after addressing network degradation issues with the deep residual network. We preprocess the collected pose images to extract human wushu pose features from both static and dynamic directions. The feature data is inputted to the encoder to extract martial arts poses. Simulation experiments verify the error correlation between two-dimensional martial arts movements and three-dimensional angles. In the backside horse stance punch movement, the 7th frame of the left knee and right knee, the 3rd frame of the left hip, and the 1st, 2nd, 5th, and 7th frames of the right hip exhibit significant errors, while the remaining keyframes fall within the acceptable range. Through empirical investigation, we analyze the teaching effect of integrating Wushu and national traditional culture. Following the experiment, we observed an average score difference of 8.91 points in the degree of sports participation between the experimental group and the control group, with a P value of less than 0.01. There is a significant difference between the two integrations. The integration of the teaching effect is good.