Human Keypoint Detection Research Articles

A Multi-Scale Graph Attention-Based Transformer for Occluded Person Re-Identification

The objective of person re-identification (ReID) tasks is to match a specific individual across different times, locations, or camera viewpoints. The prevalent issue of occlusion in real-world scenarios affects image information, rendering the affected features unreliable. The difficulty and core challenge lie in how to effectively discern and extract visual features from human images under various complex conditions, including cluttered backgrounds, diverse postures, and the presence of occlusions. Some works have employed pose estimation or human key point detection to construct graph-structured information to counteract the effects of occlusions. However, this approach introduces new noise due to issues such as the invisibility of key points. Our proposed module, in contrast, does not require the use of additional feature extractors. Our module employs multi-scale graph attention for the reweighting of feature importance. This allows features to concentrate on areas genuinely pertinent to the re-identification task, thereby enhancing the model’s robustness against occlusions. To address these problems, a model that employs multi-scale graph attention to reweight the importance of features is proposed in this study, significantly enhancing the model’s robustness against occlusions. Our experimental results demonstrate that, compared to baseline models, the method proposed herein achieves a notable improvement in mAP on occluded datasets, with increases of 0.5%, 31.5%, and 12.3% in mAP scores.

Intelligent Surveillance of Airport Apron: Detection and Location of Abnormal Behavior in Typical Non-Cooperative Human Objects

Most airport surface surveillance systems focus on monitoring and commanding cooperative objects (vehicles) while neglecting the location and detection of non-cooperative objects (humans). Abnormal behavior by non-cooperative objects poses a potential threat to airport security. This study collects surveillance video data from civil aviation airports in several regions of China, and a non-cooperative abnormal behavior localization and detection framework (NC-ABLD) is established. As the focus of this paper, the proposed framework seamlessly integrates a multi-scale non-cooperative object localization module, a human keypoint detection module, and a behavioral classification module. The framework uses a serial structure, with multiple modules working in concert to achieve precise position, human keypoints, and behavioral classification of non-cooperative objects in the airport field. In addition, since there is no publicly available rich dataset of airport aprons, we propose a dataset called IIAR-30, which consists of 1736 images of airport surfaces and 506 video clips in six frequently occurring behavioral categories. The results of experiments conducted on the IIAR-30 dataset show that the framework performs well compared to mainstream behavior recognition methods and achieves fine-grained localization and refined class detection of typical non-cooperative human abnormal behavior on airport apron surfaces.

Convolutional Block Attention Module-Multimodal Feature-Fusion Action Recognition: Enabling Miner Unsafe Action Recognition.

The unsafe action of miners is one of the main causes of mine accidents. Research on underground miner unsafe action recognition based on computer vision enables relatively accurate real-time recognition of unsafe action among underground miners. A dataset called unsafe actions of underground miners (UAUM) was constructed and included ten categories of such actions. Underground images were enhanced using spatial- and frequency-domain enhancement algorithms. A combination of the YOLOX object detection algorithm and the Lite-HRNet human key-point detection algorithm was utilized to obtain skeleton modal data. The CBAM-PoseC3D model, a skeleton modal action-recognition model incorporating the CBAM attention module, was proposed and combined with the RGB modal feature-extraction model CBAM-SlowOnly. Ultimately, this formed the Convolutional Block Attention Module-Multimodal Feature-Fusion Action Recognition (CBAM-MFFAR) model for recognizing unsafe actions of underground miners. The improved CBAM-MFFAR model achieved a recognition accuracy of 95.8% on the NTU60 RGB+D public dataset under the X-Sub benchmark. Compared to the CBAM-PoseC3D, PoseC3D, 2S-AGCN, and ST-GCN models, the recognition accuracy was improved by 2%, 2.7%, 7.3%, and 14.3%, respectively. On the UAUM dataset, the CBAM-MFFAR model achieved a recognition accuracy of 94.6%, with improvements of 2.6%, 4%, 12%, and 17.3% compared to the CBAM-PoseC3D, PoseC3D, 2S-AGCN, and ST-GCN models, respectively. In field validation at mining sites, the CBAM-MFFAR model accurately recognized similar and multiple unsafe actions among underground miners.

Light-Adaptive Human Body Key Point Detection Algorithm Based on Multi-Source Information Fusion.

The identification of key points in the human body is vital for sports rehabilitation, medical diagnosis, human-computer interaction, and related fields. Currently, depth cameras provide more precise depth information on these crucial points. However, human motion can lead to variations in the positions of these key points. While the Mediapipe algorithm demonstrates effective anti-shake capabilities for these points, its accuracy can be easily affected by changes in lighting conditions. To address these challenges, this study proposes an illumination-adaptive algorithm for detecting human key points through the fusion of multi-source information. By integrating key point data from the depth camera and Mediapipe, an illumination change model is established to simulate environmental lighting variations. Subsequently, the fitting function of the relationship between lighting conditions and adaptive weights is solved to achieve lighting adaptation for human key point detection. Experimental verification and similarity analysis with benchmark data yielded R2 results of 0.96 and 0.93, and cosine similarity results of 0.92 and 0.90. With a threshold range of 8, the joint accuracy rates for the two rehabilitation actions were found to be 89% and 88%. The experimental results demonstrate the stability of the proposed method in detecting key points in the human body under changing illumination conditions, its anti-shake ability for human movement, and its high detection accuracy. This method shows promise for applications in human-computer interaction, sports rehabilitation, and virtual reality.

Form Perfector

Home-based workouts are known to have significant health and fitness benefits, but it could be dangerous if not performed with proper posture. This could be due to a lack of sufficient training and bad habits, or the inability to afford a personal trainer to fix the individual's posture. With the current Artificial Intelligence technology, we are now able to achieve workout analysis by using computer vision approach without any involvement of personal trainer. This report discussed the approach of keypoint detection, human pose estimation and pose analysis that will be utilized to produce the proposed project through Rapid Application Development (RAD) methodology, which will monitor these procedures phase by phase. Pretrained model from MediaPipe was chosen as the main human key point detection, and various machine learning algorithms such as Linear Regression and Random Forest were studied to find the suitable algorithm to be implemented in detecting and analyze home workout postures. Furthermore, this research also studies on geometrical approach to determining proper and improper posture for analysis. These approaches were then compared to see which will produce the best results and user experience. Since machine learning approach does not produce the best result due to lack of training data and variety of other factors, the geometrical approach was chosen as the final approach before implementing it into desktop application. Keyword: Personal Trainer, Keypoint Detection, MediaPipe, Rapid Application Development.

Two-Stage Method for Clothing Feature Detection

The rapid expansion of e-commerce, particularly in the clothing sector, has led to a significant demand for an effective clothing industry. This study presents a novel two-stage image recognition method. Our approach distinctively combines human keypoint detection, object detection, and classification methods into a two-stage structure. Initially, we utilize open-source libraries, namely OpenPose and Dlib, for accurate human keypoint detection, followed by a custom cropping logic for extracting body part boxes. In the second stage, we employ a blend of Harris Corner, Canny Edge, and skin pixel detection integrated with VGG16 and support vector machine (SVM) models. This configuration allows the bounding boxes to identify ten unique attributes, encompassing facial features and detailed aspects of clothing. Conclusively, the experiment yielded an overall recognition accuracy of 81.4% for tops and 85.72% for bottoms, highlighting the efficacy of the applied methodologies in garment categorization.

Image-free single-pixel keypoint detection for privacy preserving human pose estimation.

Computer vision technology has been applied in various fields such as identification, surveillance, and robot vision. However, computer vision algorithms used for human-related tasks operate on human images, which raises data security and privacy concerns. In this Letter, we propose an image-free human keypoint detection technique using a few coded illuminations and a single-pixel detector. Our proposed method can complete the keypoint detection task at an ultralow sampling rate on a measured one-dimensional sequence without image reconstruction, thus protecting privacy from the data collection stage and preventing the acquisition of detailed visual information from the source. The network is designed to optimize both the illumination patterns and the human keypoint predictor with an encoder-decoder framework. For model training and validation, we used 2000 images from Leeds Sport Dataset and COCO Dataset. By incorporating EfficientNet backbone, the inference time is reduced from 4 s to 0.10 s. In the simulation, the proposed network achieves 91.7% average precision. Our experimental results show an average precision of 88.4% at a remarkably low sampling rate of 0.015. In summary, our proposed method has the advantages of privacy protection and resource efficiency, which can be applied to many monitoring and healthcare tasks, such as clinical monitoring, construction site monitoring, and home service robots.

AIP-Net: An anchor-free instance-level human part detection network

Human part detection has significant research and application in computer vision fields such as human–robot interaction, motion capture, facial recognition, and human key point detection. However, the current human body part detection method encounters challenges when detecting multi-scale objects and capturing the correlation relationship between human instances and human parts. To address these problems, a new anchor-free instance-level human part detection network (AIP-Net) is proposed. AIP-Net is a “two-level” structure that consists of two lightweight anchor-free detectors: a body detector and a parts detector. AIP-Net gradually focuses the human body on the human part from top to down, effectively avoiding the interference of extraneous background and enhancing the correlation relationship between human instances and body parts. Additionally, we design a body-part multidimensional context (BPMC) model in the parts detector branch to enhance the capability of the network. We trained the AIP-Ne end-to-end and achieved a state-of-the-art (SOTA) performance of 36.2 mean average precision (mAP) on COCO Human Parts Dataset. Moreover, we successfully utilized the AIP-Net in the human–robot interaction(HRI) platform and validated its practicality.

Recognition of starting movement correction for long distance runners based on human key point detection

Recognition of starting movement correction for long distance runners based on human key point detection

Recognition of Starting Movement Correction for Long Distance Runners Based on Human Key Point Detection

Recognition of Starting Movement Correction for Long Distance Runners Based on Human Key Point Detection

Multi-User Mobile Augmented Reality with ID-Aware Visual Interaction

Most existing multi-user Augmented Reality (AR) systems only support multiple co-located users to view a common set of virtual objects but lack the ability to enable each user to directly interact with other users appearing in his/her view. Such multi-user AR systems should be able to detect the human keypoints and estimate device poses (for identifying different users) in the meantime. However, due to the stringent low latency requirements and the intensive computation of the preceding two capabilities, previous research only enables either of the two capabilities for mobile devices even with the aid of the edge server. Integrating the two capabilities is promising but non-trivial in terms of latency, accuracy, and matching. To fill this gap, we propose DiTing to achieve real-time ID-aware multi-device visual interaction for multi-user AR applications, which contains three key innovations: Shared On-device Tracking to merge the similar computation for optimized latency, Tightly Coupled Dual Pipeline to enhance the accuracy of each task through mutual assistance, and Body Affinity Particle Filter to precisely match device poses with human bodies. We implement DiTing on four types of mobile AR devices and develop a multi-user AR game as a case study. Extensive experiments show that DiTing can provide high-quality human keypoint detection and pose estimation in real time (30fps) for ID-aware multi-device interaction and outperform the state-of-the-art baseline approaches.

Home-based rehabilitation training with human key point detection for chronic low back pain patients: a randomized controlled trial protocol

BackgroundCore stability exercise (CSE) is a globally acknowledged intervention for managing chronic low back pain. However, the sustained adherence of patients with chronic low back pain to CSE can be challenging, mainly due to the absence of supervision and guidance from physical therapists during home-based exercise sessions. Consequently, exercise compliance tends to decline, resulting in suboptimal long-term effectiveness of the intervention. In this trial, our primary aim is to evaluate the potential therapeutic equivalence between home-based rehabilitation training employing key point identification technology and exercise guidance administered in a hospital setting.MethodsIn this trial, we will randomly assign 104 adults with chronic low back pain (CLBP) to either an intervention or control group, with 52 participants in each group. Both interventions will consist of three weekly 0.5-h sessions of core stability exercise (CSE). The intervention group will engage in home rehabilitation training utilizing key identification technology for movement, while the control group will perform supervised exercises in a hospital setting. Outcome assessments will be conducted at 4 weeks and 16 weeks after randomization. The primary outcome measure will be the change in pain intensity based on numeric rating scale (NRS scores) from baseline to 4 weeks. Secondary outcomes will include changes in physical function (measured by the Oswestry Disability Index (ODI)) and lumbar spine mobility as well as activity participation and treatment satisfaction.DiscussionIf home-based rehabilitation method is demonstrated to be non-inferior or even superior to traditional face-to-face exercise guidance, it could significantly advance the adoption of digital medical care and contribute to improving the overall health of the population.Trial registrationNCT05998434. Registered on 16 August 2023.

Human pose estimation and action recognition for fitness movements

Human pose estimation and action recognition have important applications in the areas of security, medicine and sports. In this work, we propose an improved YOLOv7-Pose algorithm to solve the problem of human posture and action recognition for fitness movements in various scenarios. For the algorithm based on YOLOv7-Pose, we add the function of classification to the original network. And we introduce Coordinate Attention to the network to improve the model’s ability to identify important features of human skeletal joints and action classifications. The improved ConvNeXt network structure is introduced to replace the CBS convolution kernel of the original model to improve the accuracy of human key point detection and action classification of the model. We optimise the spatial pyramid pool structure, which can reduce the loss function and accelerate the convergence rate of the model. We adopt EIOU as the regression function of the target detection frame to improve the accuracy of the coordinate regression. Experiments show that the improved YOLOv7-Pose has an mAP of 95.9% on a homemade test set of fitness actions, which is 5.4% higher than HRNet, with a 4.2% improvement over the original YOLOv7 algorithm, suggesting that the accuracy of action recognition and key point estimation has significantly improved.

Multi-Person Pose Estimation in the Wild: Using Adversarial Method to Train a Top-Down Pose Estimation Network

Recent studies estimate human anatomical key points through the single monocular image, in which multichannel heatmaps are the key factor in determining the quality of human pose estimation. Multichannel heatmaps can efficiently handle the image-to-coordinate mapping task and the processing of semantic features. Most methods ignore physical constraints and internal relationships of human body parts, which easily misclassify left and right symmetrical parts as similar features. Some studies use RNNs on the top to incorporate priors about the structure of pose components and body configuration. Therefore, a novel top-down convolutional network is proposed to consider these priors during training, which can improve the robustness under complex field conditions in the wild. In order to learn the prior knowledge of human pose configuration, the hierarchy of fully convolutional networks (discriminator) is used to distinguish real poses from fake ones. Consequently, the pose network is inclined to make a pose estimation that the discriminator misjudges as true, which is reasonable in complex situations. The performance of the method is experimentally validated by pose estimation on the MS COCO human key point detection task. The proposed approach outperforms the original method and generates robust pose predictions, demonstrating efficiency by using adversarial learning.

VaBUS: Edge-Cloud Real-Time Video Analytics via Background Understanding and Subtraction

Edge-cloud collaborative video analytics is transforming the way data is being handled, processed, and transmitted from the ever-growing number of surveillance cameras around the world. To avoid wasting limited bandwidth on unrelated content transmission, existing video analytics solutions usually perform temporal or spatial filtering to realize aggressive compression of irrelevant pixels. However, most of them work in a context-agnostic way while being oblivious to the circumstances where the video content is happening and the context-dependent characteristics under the hood. In this work, we propose VaBUS, a real-time video analytics system that leverages the rich contextual information of surveillance cameras to reduce bandwidth consumption for semantic compression. As a task-oriented communication system, VaBUS dynamically maintains the background image of the video on the edge with minimal system overhead and sends only highly confident Region of Interests (RoIs) to the cloud through adaptive weighting and encoding. With a lightweight experience-driven learning module, VaBUS is able to achieve high offline inference accuracy even when network congestion occurs. Experimental results show that VaBUS reduces bandwidth consumption by 25.0%-76.9% while achieving 90.7% accuracy for both the object detection and human keypoint detection tasks.

Lightweight human pose estimation algorithm based on polarized self-attention

In recent years, human pose estimation has been widely used in human-computer interaction, augmented reality, video surveillance, and many other fields, but the task of pose estimation still faces many challenges. To address the large number of parameters and complicated calculation in the current mainstream human pose estimation network, this paper proposes a lightweight pose estimation network (Lightweight Polarized Network, referred to as LPNet) based on a polarized self-attention mechanism. First, ghost convolution is used to reduce the number of parameters of the feature extraction network; second, by introducing the polarized self-attention module, the pixel-level regression task can be better solved, the lack of extracted features due to the decrease in the number of parameters can be reduced, and the accuracy of the regression of human keypoints can be improved; finally, a new coordinate decoding method is designed to reduce the error in the heatmap decoding process and improve the accuracy of keypoint regression. The method proposed in this paper was evaluated on the human keypoint detection datasets COCO and MPII, and compared with the current mainstream methods. The experimental results show that the proposed method greatly reduces the number of parameters of the model while ensuring a small loss in accuracy.

Towards High Performance Human Keypoint Detection

Human keypoint detection from a single image is very challenging due to occlusion, blur, illumination, and scale variance. In this paper, we address this problem from three aspects by devising an efficient network structure, proposing three effective training strategies, and exploiting four useful postprocessing techniques. First, we find that context information plays an important role in reasoning human body configuration and invisible keypoints. Inspired by this, we propose a cascaded context mixer (CCM), which efficiently integrates spatial and channel context information and progressively refines them. Then, to maximize CCM’s representation capability, we develop a hard-negative person detection mining strategy and a joint-training strategy by exploiting abundant unlabeled data. It enables CCM to learn discriminative features from massive diverse poses. Third, we present several sub-pixel refinement techniques for postprocessing keypoint predictions to improve detection accuracy. Extensive experiments on the MS COCO keypoint detection benchmark demonstrate the superiority of the proposed method over representative state-of-the-art (SOTA) methods. Our single model achieves comparable performance with the winner of the 2018 COCO Keypoint Detection Challenge. The final ensemble model sets a new SOTA on this benchmark. The source code will be released at https://github.com/chaimi2013/CCM .

PGNet: Pipeline Guidance for Human Key-Point Detection.

Human key-point detection is a challenging research field in computer vision. Convolutional neural models limit the number of parameters and mine the local structure, and have made great progress in significant target detection and key-point detection. However, the features extracted by shallow layers mainly contain a lack of semantic information, while the features extracted by deep layers contain rich semantic information but a lack of spatial information that results in information imbalance and feature extraction imbalance. With the complexity of the network structure and the increasing amount of computation, the balance between the time of communication and the time of calculation highlights the importance. Based on the improvement of hardware equipment, network operation time is greatly improved by optimizing the network structure and data operation methods. However, as the network structure becomes deeper and deeper, the communication consumption between networks also increases, and network computing capacity is optimized. In addition, communication overhead is also the focus of recent attention. We propose a novel network structure PGNet, which contains three parts: pipeline guidance strategy (PGS); Cross-Distance-IoU Loss (CIoU); and Cascaded Fusion Feature Model (CFFM).

Multi person pose estimation based on improved openpose model

In order to solve the problem of multi person pose estimation, a human body pose estimation model framework based on openpose from American Carnegie Mellon University is presented, which uses the COCO (Microsoft common objects in context) data set to augment the data, optimize the network structure and retrain, adjust the learning rate properly and the convergence speed in the early stage of the model, finally using the model to carry out research on human key point detection and visualizing analysis in a multi-person complex scene. It is found that the improved openpose model adopts a bottom to up detection strategy to avoid the influence of the number of human bodies on real-time performance, and has higher key point detection efficiency than the top to down method. The study provides a real-time position feedback scheme for human pose estimation in complex scenes.

Multiple Object Tracking for Dense Pedestrians by Markov Random Field Model with Improvement on Potentials

Pedestrian tracking in dense crowds is a challenging task, even when using a multi-camera system. In this paper, a new Markov random field (MRF) model is proposed for the association of tracklet couplings. Equipped with a new potential function improvement method, this model can associate the small tracklet coupling segments caused by dense pedestrian crowds. The tracklet couplings in this paper are obtained through a data fusion method based on image mutual information. This method calculates the spatial relationships of tracklet pairs by integrating position and motion information, and adopts the human key point detection method for correction of the position data of incomplete and deviated detections in dense crowds. The MRF potential function improvement method for dense pedestrian scenes includes assimilation and extension processing, as well as a message selective belief propagation algorithm. The former enhances the information of the fragmented tracklets by means of a soft link with longer tracklets and expands through sharing to improve the potentials of the adjacent nodes, whereas the latter uses a message selection rule to prevent unreliable messages of fragmented tracklet couplings from being spread throughout the MRF network. With the help of the iterative belief propagation algorithm, the potentials of the model are improved to achieve valid association of the tracklet coupling fragments, such that dense pedestrians can be tracked more robustly. Modular experiments and system-level experiments are conducted using the PETS2009 experimental data set, where the experimental results reveal that the proposed method has superior tracking performance.