Human Pose Research Articles

LFSimCC: Spatial fusion lightweight network for human pose estimation

To address the limitations of existing 2D human pose estimation methods in terms of speed and lightweight, we propose a method called Lightweight Fusion SimCC (LFSimCC). LFSimCC incorporates two modules: LiteFNet, which enhances multi-scale spatial information fusion, and LKC-GAU, which improves the modeling capability of spatial information. Specifically, LiteFNet utilizes a combination of self-attention mechanism and novel spatial convolution to enable feature maps to capture richer multi-level global feature representations within the network. On the other hand, LKC-GAU enhances SimCC’s ability to capture spatial relationships between joints by incorporating a large kernel of convolution and a self-attention mechanism. Furthermore, we design a keypoint information fusion loss (IFL) that enhances the model’s sensitivity to information between keypoints in the human body. Experimental results demonstrate that our method is capable of extracting more decisive information and suppressing redundant feature representations, leading to high recognition accuracy and low inference latency.

Enhancing 3D human pose estimation with NIR single-pixel imaging and time-of-flight technology: a deep learning approach.

The extraction of 3D human pose and body shape details from a single monocular image is a significant challenge in computer vision. Traditional methods use RGB images, but these are constrained by varying lighting and occlusions. However, cutting-edge developments in imaging technologies have introduced new techniques such as single-pixel imaging (SPI) that can surmount these hurdles. In the near-infrared spectrum, SPI demonstrates impressive capabilities in capturing a 3D human pose. This wavelength can penetrate clothing and is less influenced by lighting variations than visible light, thus providing a reliable means to accurately capture body shape and pose data, even in difficult settings. In this work, we explore the use of an SPI camera operating in the NIR with time-of-flight (TOF) at bands 850-1550nm as a solution to detect humans in nighttime environments. The proposed system uses the vision transformers (ViT) model to detect and extract the characteristic features of humans for integration over a 3D body model SMPL-X through 3D body shape regression using deep learning. To evaluate the efficacy of NIR-SPI 3D image reconstruction, we constructed a laboratory scenario that simulates nighttime conditions, enabling us to test the feasibility of employing NIR-SPI as a vision sensor in outdoor environments. By assessing the results obtained from this setup, we aim to demonstrate the potential of NIR-SPI as an effective tool to detect humans in nighttime scenarios and capture their accurate 3D body pose and shape.

A Lightweight Context-Aware Feature Transformer Network for Human Pose Estimation

We propose a Context-aware Feature Transformer Network (CaFTNet), a novel network for human pose estimation. To address the issue of limited modeling of global dependencies in convolutional neural networks, we design the Transformerneck to strengthen the expressive power of features. Transformerneck directly substitutes 3×3 convolution in the bottleneck of HRNet with a Contextual Transformer (CoT) block while reducing the complexity of the network. Specifically, the CoT first produces keys with static contextual information through 3×3 convolution. Then, relying on query and contextualization keys, dynamic contexts are generated through two concatenated 1×1 convolutions. Static and dynamic contexts are eventually fused as an output. Additionally, for multi-scale networks, in order to further refine the features of the fusion output, we propose an Attention Feature Aggregation Module (AFAM). Technically, given an intermediate input, the AFAM successively deduces attention maps along the channel and spatial dimensions. Then, an adaptive refinement module (ARM) is exploited to activate the obtained attention maps. Finally, the input undergoes adaptive feature refinement through multiplication with the activated attention maps. Through the above procedures, our lightweight network provides powerful clues for the detection of keypoints. Experiments are performed on the COCO and MPII datasets. The model achieves a 76.2 AP on the COCO val2017 dataset. Compared to other methods with a CNN as the backbone, CaFTNet has a 72.9% reduced number of parameters. On the MPII dataset, our method uses only 60.7% of the number of parameters, acquiring similar results to other methods with a CNN as the backbone.

Dense-connected Stacked Hourglass Networks for Human Pose Estimation

The main idea of this project is to try to improve the accuracy of human pose estimation in previous models. The new model proposed is based on the Stacked Hourglass Network with new structures added. The new structures ensured that the preservation of features of the original data by adding connections across the network, which we refer to as a Dense-connected Stacked Hourglass network, and we expected the new structure and the feature preserved could be helpful in the later stages because the Stacked Hourglass network pools down to very low resolution, during which important information may be lost. The data sets used in the project are MPII Human Pose and FLIC (Frames Labelled in Cinema). The final results show that the proposed architecture is able to improve the estimation accuracy to certain extend in identifying head, wrist and hip, while further studies on the architecture and improvements are still required.

AI Fitness Model using Deep Learning

The project “AI Fitness Model Using Deep Learning (YOLOv5)” aims inorder to transform the fitness business by leveraging state-of-the-art deep learning techniques, specifically YOLOv5 (You Only Look Once version 5), to develop an advanced and efficient fitness tracking system. The model is designed to accurately detect and analyze human poses, movements, and exercise routines in real-time using computer vision. By employing YOLOv5’s object detection capabilities, the AI fitness model can identify key body points, track exercise execution, and provide personalized feedback to users, enhancing their workout experience. This creative strategy not only facilitates automated performance monitoring but also enables the creation of adaptive and dynamic fitness routines tailored to individual needs, fostering improved engagement and results in the realm of health and wellness

Lightweight Whole-Body Human Pose Estimation With Two-Stage Refinement Training Strategy

Lightweight Whole-Body Human Pose Estimation With Two-Stage Refinement Training Strategy

Binarizing by Classification: Is Soft Function Really Necessary?

Binary neural networks leverage Sign function to binarize weights and activations, which require gradient estimators to overcome its non-differentiability and will inevitably bring gradient errors during backpropagation. Although many hand-designed soft functions have been proposed as gradient estimators to better approximate gradients, their mechanism is not clear and there are still huge performance gaps between binary models and their full-precision counterparts. To address these issues and reduce gradient error, we propose to tackle network binarization as a binary classification problem and use a multi-layer perceptron (MLP) as the classifier in the forward pass and gradient estimator in the backward pass. Benefiting from the MLP’s theoretical capability to fit any continuous function, it can be adaptively learned to binarize networks and backpropagate gradients without any prior knowledge of soft functions. From this perspective, we further empirically justify that even a simple linear function can outperform previous complex soft functions. Extensive experiments demonstrate that the proposed method yields surprising performance both in image classification and human pose estimation tasks. Specifically, we achieve 65.7% top-1 accuracy of ResNet-34 on ImageNet dataset, with an absolute improvement of 2.6%. Moreover, we take binarization as a lightweighting approach for pose estimation models and propose well-designed binary pose estimation networks SBPN and BHRNet. When evaluating on the challenging Microsoft COCO keypoint dataset, the proposed method enables binary networks to achieve a mAP of up to 60.6 for the first time. Experiments conducted on real platforms demonstrate that BNN achieves a better balance between performance and computational complexity, especially when computational resources are extremely low.

Parallel Self-Attention and Spatial-Attention Fusion for Human Pose Estimation and Running Movement Recognition

Human pose estimation (HPE) is a fundamental yet promising visual recognition problem. Existing popular methods (e.g., Hourglass and its variants) either attempt to directly add local features element-wisely, or (e.g., vision transformers) try to learn the global relationships among different human parts. However, it remains an open problem to effectively integrate the local-global representations for accurate HPE. In this work, we design four feature fusion strategies on the hierarchical ResNet structure, including direct channel concatenation, element-wise addition and two parallel structures. Both two parallel structures adopt the naive self-attention encoder to model global dependencies. The difference between them is that one adopts the original ResNet BottleNeck while the other employs a spatial-attention module (named SSF) to learn the local patterns. Experiments on COCO Keypoint 2017 show that our SSF for HPE (named SSPose) achieves the best average precision with acceptable computational cost among the compared state-of-the-art methods. In addition, we build a lightweight running dataset to verify the effectiveness of SSPose. Based solely on the keypoints estimated by our SSPose, we propose a regression model to identify valid running movements without training any other classifiers. Our source codes and running dataset are publicly available.

Intelligent System Detection of Dead Victims at Natural Disaster Areas Using Deep Learning

Disaster is the occurrence or sequence of occurrences that endangers and disrupts people’s lives and livelihoods due to natural and/or non-natural as well as human elements, including fatalities, property loss, environmental harm, and psychological effects. In addition to concentrating on the victims’ safety and their own safety, the search and rescue (SAR) team plays a significant part in this evacuation operation. Based on these issues, this study examined how to use a drone equipped with electronic equipment to search for victims on the ground to speed up the evacuation process at natural disaster sites, assisting the evacuation process and enhancing the safety of the SAR team. The drone carries a near-infrared camera and GPS. The images captured by the camera provide the parameters for classifying victims using deep learning. The system has been implemented by sampling data from human poses resembling the position of the victims’ bodies from natural disasters. From the experimental results, the system can detect objects with high accuracy, that is, 99% in both static and dynamic conditions. The best model results were obtained at a height of 2 meters with a low error percentage.

A study of human pose estimation in low-light environments using YOLOv8 model

Human pose estimation is a formidable task in the field of computer vision., often constrained by limited training samples and various complexities encountered during target detection, including complex backgrounds, object occlusion, crowded scenes, and varying perspectives. The primary objective of this research paper is to explore the performance disparities of the recently introduced YOLOv8 model in the context of human pose estimation. We conduct a comprehensive evaluation of six different models with varying complexities on the same low-light photograph to assess their precision and speed. The objective is to determine the suitability of each model for specific environmental contexts. The experimental results reveal that our findings demonstrate a partial regression in accuracy for the yolov8s-pose and yolov8m-pose models when tested on our sampled images. The increase in model layers indicates enhanced complexity and expressive power, while additional parameters signify improved learning capabilities at the expense of increased computational resource requirements.

YOLO-Rlepose: Improved YOLO Based on Swin Transformer and Rle-Oks Loss for Multi-Person Pose Estimation

In recent years, there has been significant progress in human pose estimation, fueled by the widespread adoption of deep convolutional neural networks. However, despite these advancements, multi-person 2D pose estimation still remains highly challenging due to factors such as occlusion, noise, and non-rigid body movements. Currently, most multi-person pose estimation approaches handle joint localization and association separately. This study proposes a direct regression-based method to estimate the 2D human pose from a single image. The authors name this network YOLO-Rlepose. Compared to traditional methods, YOLO-Rlepose leverages Transformer models to better capture global dependencies between image feature blocks and preserves sufficient spatial information for keypoint detection through a multi-head self-attention mechanism. To further improve the accuracy of the YOLO-Rlepose model, this paper proposes the following enhancements. Firstly, this study introduces the C3 Module with Swin Transformer (C3STR). This module builds upon the C3 module in You Only Look Once (YOLO) by incorporating a Swin Transformer branch, enhancing the YOLO-Rlepose model’s ability to capture global information and rich contextual information. Next, a novel loss function named Rle-Oks loss is proposed. The loss function facilitates the training process by learning the distributional changes through Residual Log-likelihood Estimation. To assign different weights based on the importance of different keypoints in the human body, this study introduces a weight coefficient into the loss function. The experiments proved the efficiency of the proposed YOLO-Rlepose model. On the COCO dataset, the model outperforms the previous SOTA method by 2.11% in AP.

Kinematics modeling network for video-based human pose estimation

Estimating human poses from videos is critical in human–computer interaction. Joints cooperate rather than move independently during human movement. There are both spatial and temporal correlations between joints. Despite the positive results of previous approaches, most of them focus on modeling the spatial correlation between joints while only straightforwardly integrating features along the temporal dimension, which ignores the temporal correlation between joints. In this work, we propose a plug-and-play kinematics modeling module (KMM) to explicitly model temporal correlations between joints across different frames by calculating their temporal similarity. In this way, KMM can capture motion cues of the current joint relative to all joints in different time. Besides, we formulate video-based human pose estimation as a Markov Decision Process and design a novel kinematics modeling network (KIMNet) to simulate the Markov Chain, allowing KIMNet to locate joints recursively. Our approach achieves state-of-the-art results on two challenging benchmarks. In particular, KIMNet shows robustness to the occlusion. Code will be released at https://github.com/YHDang/KIMNet.

3D Human pose estimation from video via multi-scale multi-level spatial temporal features

3D Human pose estimation from video via multi-scale multi-level spatial temporal features

Bidirectional Temporal Pose Matching for Tracking

Multi-person pose tracking is a challenging task. It requires identifying the human poses in each frame and matching them across time. This task still faces two main challenges. Firstly, sudden camera zooming and drastic pose changes between adjacent frames may result in mismatched poses between them. Secondly, the time relationships modeled by most existing methods provide insufficient information in scenarios with long-term occlusion. In this paper, to address the first challenge, we propagate the bounding boxes of the current frame to the previous frame for pose estimation, and match the estimated results with the previous ones, which we call the Backward Temporal Pose-Matching (BTPM) module. To solve the second challenge, we design an Association Across Multiple Frames (AAMF) module that utilizes long-term temporal relationships to supplement tracking information lost in the previous frames as a Re-identification (Re-id) technique. Specifically, we select keyframes with a fixed step size in the videos and label other frames as general frames. In the keyframes, we use the BTPM module and the AAMF module to perform tracking. In the general frames, we propagate poses in the previous frame to the current frame for pose estimation and association, which we call the Forward Temporal Pose-Matching (FTPM) module. If the pose association fails, the current frame will be set as a keyframe, and tracking will be re-performed. In the PoseTrack 2018 benchmark tests, our method shows significant improvements over the baseline methods, with improvements of 2.1 and 1.1 in mean Average Precision (mAP) and Multi-Object Tracking Accuracy (MOTA), respectively.

Image-free single-pixel keypoint detection for privacy preserving human pose estimation.

Computer vision technology has been applied in various fields such as identification, surveillance, and robot vision. However, computer vision algorithms used for human-related tasks operate on human images, which raises data security and privacy concerns. In this Letter, we propose an image-free human keypoint detection technique using a few coded illuminations and a single-pixel detector. Our proposed method can complete the keypoint detection task at an ultralow sampling rate on a measured one-dimensional sequence without image reconstruction, thus protecting privacy from the data collection stage and preventing the acquisition of detailed visual information from the source. The network is designed to optimize both the illumination patterns and the human keypoint predictor with an encoder-decoder framework. For model training and validation, we used 2000 images from Leeds Sport Dataset and COCO Dataset. By incorporating EfficientNet backbone, the inference time is reduced from 4 s to 0.10 s. In the simulation, the proposed network achieves 91.7% average precision. Our experimental results show an average precision of 88.4% at a remarkably low sampling rate of 0.015. In summary, our proposed method has the advantages of privacy protection and resource efficiency, which can be applied to many monitoring and healthcare tasks, such as clinical monitoring, construction site monitoring, and home service robots.

Fall Detection System Based on Point Cloud Enhancement Model for 24 GHz FMCW Radar.

Automatic fall detection plays a significant role in monitoring the health of senior citizens. In particular, millimeter-wave radar sensors are relevant for human pose recognition in an indoor environment due to their advantages of privacy protection, low hardware cost, and wide range of working conditions. However, low-quality point clouds from 4D radar diminish the reliability of fall detection. To improve the detection accuracy, conventional methods utilize more costly hardware. In this study, we propose a model that can provide high-quality three-dimensional point cloud images of the human body at a low cost. To improve the accuracy and effectiveness of fall detection, a system that extracts distribution features through small radar antenna arrays is developed. The proposed system achieved 99.1% and 98.9% accuracy on test datasets pertaining to new subjects and new environments, respectively.

On the Evaluation of Diverse Vision Systems towards Detecting Human Pose in Collaborative Robot Applications.

Tracking human operators working in the vicinity of collaborative robots can improve the design of safety architecture, ergonomics, and the execution of assembly tasks in a human-robot collaboration scenario. Three commercial spatial computation kits were used along with their Software Development Kits that provide various real-time functionalities to track human poses. The paper explored the possibility of combining the capabilities of different hardware systems and software frameworks that may lead to better performance and accuracy in detecting the human pose in collaborative robotic applications. This study assessed their performance in two different human poses at six depth levels, comparing the raw data and noise-reducing filtered data. In addition, a laser measurement device was employed as a ground truth indicator, together with the average Root Mean Square Error as an error metric. The obtained results were analysed and compared in terms of positional accuracy and repeatability, indicating the dependence of the sensors' performance on the tracking distance. A Kalman-based filter was applied to fuse the human skeleton data and then to reconstruct the operator's poses considering their performance in different distance zones. The results indicated that at a distance less than 3 m, Microsoft Azure Kinect demonstrated better tracking performance, followed by Intel RealSense D455 and Stereolabs ZED2, while at ranges higher than 3 m, ZED2 had superior tracking performance.

Learning scale-aware relationships via Laplacian decomposition-based transformer for 3D human pose estimation

Learning scale-aware relationships via Laplacian decomposition-based transformer for 3D human pose estimation

Comparison of Shoulder Range of Motion Quantified with Mobile Phone Video-Based Skeletal Tracking and 3D Motion Capture-Preliminary Study.

The accuracy of human pose tracking using smartphone camera (2D-pose) to quantify shoulder range of motion (RoM) is not determined. Twenty healthy individuals were recruited and performed shoulder abduction, adduction, flexion, or extension, captured simultaneously using a smartphone-based human pose estimation algorithm (Apple's vision framework) and using a skin marker-based 3D motion capture system. Validity was assessed by comparing the 2D-pose outcomes against a well-established 3D motion capture protocol. In addition, the impact of iPhone positioning was investigated using three smartphones in multiple vertical and horizontal positions. The relationship and validity were analysed using linear mixed models and Bland-Altman analysis. We found that 2D-pose-based shoulder RoM was consistent with 3D motion capture (linear mixed model: R2 > 0.93) but was somewhat overestimated by the smartphone. Differences were dependent on shoulder movement type and RoM amplitude, with adduction the worst performer among all tested movements. All motion types were described using linear equations. Correction methods are provided to correct potential out-of-plane shoulder movements. Shoulder RoM estimated using a smartphone camera is consistent with 3D motion-capture-derived RoM; however, differences between the systems were observed and are likely explained by differences in thoracic frame definitions.

Through-Wall Human Pose Estimation by Mutual Information Maximizing Deeply Supervised Nets

Through-Wall Human Pose Estimation by Mutual Information Maximizing Deeply Supervised Nets