Pose Estimation System Research Articles

A NEURAL NETWORK MODEL FOR RECOGNIZING HUMAN POSES BASED ON INFORMATION OBTAINED FROM VIDEO IMAGES

Nowadays, there are several ways of recognizing and identifying a person, and these keep enhancing day by day. However, cases of falsification of these methods can also be observed. Modern video surveillance systems keep on developing, and satellite systems enable us to photograph everything that happens in a certain area and analyze effectively the data obtained using neural networks. Video images help to monitor motions of people, detect illegal entries into prohibited areas, as well as to identify remaining criminals by means of photos taken from cameras and control exit and/or entry of wanted criminals to foreign countries by acquiring other biometric data. This system serves to catch criminals at airports, railway stations, seaports; it automatically counts numbers of people in a line or in a crowd and analyzes the nature of their movements, which lessens the amount of subjective human intervention and reduces time required for data processing. Moreover, neural network-based pose estimation system is now being used widely in sports, and this can distinguish between athletes and change their behavior based on their personal characteristics.

Towards Simple and Accurate Human Pose Estimation With Stair Network

In this paper, we focus on tackling the precise keypoint coordinates regression task. Most existing approaches adopt complicated networks with a large number of parameters, leading to a heavy model with poor cost-effectiveness in practice. To overcome this limitation, we develop a small yet discrimicative model called STair Network, which can be simply stacked towards an accurate multi-stage pose estimation system. Specifically, to reduce computational cost, STair Network is composed of novel basic feature extraction blocks which focus on promoting feature diversity and obtaining rich local representations with fewer parameters, enabling a satisfactory balance on efficiency and performance. To further improve the performance, we introduce two mechanisms with negligible computational cost, focusing on feature fusion and replenish. We demonstrate the effectiveness of the STair Network on two standard datasets, e.g., 1-stage STair Network achieves a higher accuracy than HRNet by 5.5% on COCO test dataset with 80% fewer parameters and 68% fewer GFLOPs.

Neuromorphic high-frequency 3D dancing pose estimation in dynamic environment

Technology-mediated dance experiences, as a medium of entertainment, are a key element in both traditional and virtual reality-based gaming platforms. These platforms predominantly depend on unobtrusive and continuous human pose estimation as a means of capturing input. Current solutions primarily employ RGB or RGB-Depth cameras for dance gaming applications; however, the former is hindered by low-light conditions due to motion blur and reduced sensitivity, while the latter exhibits excessive power consumption, diminished frame rates, and restricted operational distance. Boasting ultra-low latency, energy efficiency, and a wide dynamic range, neuromorphic cameras present a viable solution to surmount these limitations. Here, we introduce YeLan, a neuromorphic camera-driven, three-dimensional, high-frequency human pose estimation (HPE) system capable of withstanding low-light environments and dynamic backgrounds. We have compiled the first-ever neuromorphic camera dance HPE dataset and devised a fully adaptable motion-to-event, physics-conscious simulator. YeLan surpasses baseline models under strenuous conditions and exhibits resilience against varying clothing types, background motion, viewing angles, occlusions, and lighting fluctuations.

Large-Scale Aircraft Pose Estimation System Based on Depth Cameras

In the fields of wind tunnel measurement and aerospace, the real-time pose information of aircraft is an important index. In this paper, we propose a large-scale aircraft pose estimation system, in which depth cameras are used to scan the entire aircraft model in multiple directions. Using a principal component analysis (PCA) featuring vectors as the target coordinate system through a coordinate transformation matrix for the point cloud calibration of aircraft, we merge the complete aircraft model with the point cloud. An intrinsic shape signature (ISS) key point extraction and a signature of histograms of orientations (SHOT) feature description are used to form feature descriptors. The scale of the point clouds is reduced, and coarse registration of the point clouds is performed by feature matching and random sample consensus (RANSAC) mismatching. The robustness of the algorithm is improved, and the initial pose estimation is achieved for the precise registration of point clouds. The experimental results demonstrate that the proposed system can achieve an angle measurement accuracy of 0.05°.

Evaluation of mouse behavioral responses to nutritive versus nonnutritive sugar using a deep learning-based 3D real-time pose estimation system

Animals are able to detect the nutritional content of sugar independently of taste. When given a choice between nutritive sugar and nonnutritive sugar, animals develop a preference for nutritive sugar over nonnutritive sugar during a period of food deprivation (Buchanan et al., 2022; Dus et al., 2011; 2015; Tan et al., 2020; Tellez et al., 2016). To quantify behavioral features during an episode of licking nutritive versus nonnutritive sugar, we implemented a multi-vision, deep learning-based 3D pose estimation system, termed the AI Vision Analysis for Three-dimensional Action in Real-Time (AVATAR)(Kim et al., 2022). Using this method, we found that mice exhibit significantly different approach behavioral responses toward nutritive sugar versus nonnutritive sugar even before licking a sugar solution. Notably, the behavioral sequences during the approach toward nutritive versus nonnutritive sugar became significantly different over time. These results suggest that the nutritional value of sugar not only promotes its consumption but also elicits distinct repertoires of feeding behavior in deprived mice.

The IMU Augment SLAM on Unmanned Vehicle for Detection of Protective Measures in COVID-19

Detecting protective measures (e.g., masks, goggles and protective clothing) is a momentous step in the fight against COVID-19. The detection mode of unmanned devices based on Simultaneous localization and mapping (SLAM) and fusion technology is more efficient, economical and safe than the traditional manual detection. In this paper, a tightly-coupled nonlinear optimization approach is used to augment the visual feature extraction of SLAM by the gyroscope of the IMU to obtain a high-precision visual inertial system for joint position and pose estimation. Based on the VINS-Mono frame, first, an LSD algorithm based on a conditional selection strategy is proposed to extract line features efficiently. Then, we propose recovering missing point features from line features. Moreover, we propose a strategy to recover vanishing point features from line features, and add residuals to the SLAM cost function based on optimization, which optimizes point-line features in real time to promote the tracking and matching accuracy. Second, the wavelet threshold denoising method based on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\sigma $ </tex-math></inline-formula> criterion is used to carry out real-time online denoising for gyroscope to improve the output precision. Our WD-PL-VINS was measured on publicly available EuRoC datasets, TUM VI datasets and evaluated and validated in lab testing with a unmanned vehicle (UV) based on the NVIDIA Jetson-TX2 development board. The results show that our method’s APE and RPE on MH_03_easy sequences are improved by 69.28% and 97.66%, respectively, compared with VINS-Mono.

A Powered Floor System with Integrated Robot Localization

One of the most pressing issues in the field of mobile robotics is power delivery. In the past, we have proposed a powered floor based solution. In this article, we propose a system which combines a powered floor with a robot pose estimation system. The floor on which the mobile robots stand is composed of an array of interdigitated conductors that provide DC power supply; the stripes of conductors are interwoven similarly to what happens in a carpet, thus creating a sort of checkerboard of positive and negative pads. The robots are powered through sliding contacts. The power supply voltage is modulated with a binary encoding that uniquely identifies each conductor stripe power line; in this way, each robot is able to self-localize, by exploiting the information coming from the contacting pins. We describe the theoretical framework that allows concurrent power delivery and localization (with error boundaries). Then, we present the experimental evaluation that we performed using a prototype realization of the proposed powered floor system.

A Robust Learned Feature-Based Visual Odometry System for UAV Pose Estimation in Challenging Indoor Environments

Unmanned Aerial Vehicles (UAVs) are becoming popular nowadays due to their versatility and flexibility for indoor applications, such as the autonomous visual inspection for the inner surface of a pressure vessel. Nevertheless, robust and reliable position estimation is critical for completing these tasks. Visual Odometry (VO) and Visual Simultaneous Localisation and Mapping (VSLAM) allow the UAV to estimate its position in unknown environments. However, traditional feature-based VO/VSLAM systems struggle to deal with complex scenes such as low illumination and textureless environments. Replacing the traditional features with deep learning-based features provides the advantage of handling the challenging environment, but the efficiency is ignored. In this work, an efficient VO system based on a novel lightweight feature extraction network for UAV onboard platforms has been developed. The Deformable Convolution (DFConv) is utilised to improve the feature extraction capability. Owing to the limited onboard computing capability, the Depth-wise Separable Convolution (DWConv) is adopted to calculate the offsets for the deformable convolution and construct the backbone network to improve the feature extraction efficiency. Experiments on public datasets indicate that the efficiency of the VO system is improved by 30.03% while preserving the accuracy on embedded platforms with the feature points and descriptors detected by the proposed Convolutional Neural Network (CNN). Moreover, the proposed VO system is verified through UAV flying tests in a real-world scenario. The results prove that the proposed VO system is able to handle the challenging environments where both the latest traditional and deep learning feature-based VO/VSLAM systems fail, and it is feasible for UAV self-localisation and autonomous navigation in the confined, low illumination and textureless indoor environment.

Identity-Preserved Human Posture Detection in Infrared Thermal Images: A Benchmark

Human pose estimation has a variety of real-life applications, including human action recognition, AI-powered personal trainers, robotics, motion capture and augmented reality, gaming, and video surveillance. However, most current human pose estimation systems are based on RGB images, which do not seriously take into account personal privacy. Although identity-preserved algorithms are very desirable when human pose estimation is applied to scenarios where personal privacy does matter, developing human pose estimation algorithms based on identity-preserved modalities, such as thermal images concerned here, is very challenging due to the limited amount of training data currently available and the fact that infrared thermal images, unlike RGB images, lack rich texture cues which makes annotating training data itself impractical. In this paper, we formulate a new task with privacy protection that lies between human detection and human pose estimation by introducing a benchmark for IPHPDT (i.e., Identity-Preserved Human Posture Detection in Thermal images). This task has a threefold novel purpose: the first is to establish an identity-preserved task with thermal images; the second is to achieve more information other than the location of persons as provided by human detection for more advanced computer vision applications; the third is to avoid difficulties in collecting well-annotated data for human pose estimation in thermal images. The presented IPHPDT dataset contains four types of human postures, consisting of 75,000 images well-annotated with axis-aligned bounding boxes and postures of the persons. Based on this well-annotated IPHPDT dataset and three state-of-the-art algorithms, i.e., YOLOF (short for You Only Look One-level Feature), YOLOX (short for Exceeding YOLO Series in 2021) and TOOD (short for Task-aligned One-stage Object Detection), we establish three baseline detectors, called IPH-YOLOF, IPH-YOLOX, and IPH-TOOD. In the experiments, three baseline detectors are used to recognize four infrared human postures, and the mean average precision can reach 70.4%. The results show that the three baseline detectors can effectively perform accurate posture detection on the IPHPDT dataset. By releasing IPHPDT, we expect to encourage more future studies into human posture detection in infrared thermal images and draw more attention to this challenging task.

Vision-Guided Object Recognition and 6D Pose Estimation System Based on Deep Neural Network for Unmanned Aerial Vehicles towards Intelligent Logistics

Unmanned aerial vehicle (UAV) express delivery is facing a period of rapid development and continues to promote the aviation logistics industry due to its advantages of elevated delivery efficiency and low labor costs. Automatic detection, localization, and estimation of 6D poses of targets in dynamic environments are key prerequisites for UAV intelligent logistics. In this study, we proposed a novel vision system based on deep neural networks to locate targets and estimate their 6D pose parameters from 2D color images and 3D point clouds captured by an RGB-D sensor mounted on a UAV. The workflow of this system can be summarized as follows: detect the targets and locate them, separate the object region from the background using a segmentation network, and estimate the 6D pose parameters from a regression network. The proposed system provides a solid foundation for various complex operations for UAVs. To better verify the performance of the proposed system, we built a small dataset called SIAT comprising some household staff. Comparative experiments with several state-of-the-art networks on the YCB-Video dataset and SIAT dataset verified the effectiveness, robustness, and superior performance of the proposed method, indicating its promising applications in UAV-based delivery tasks.

Full-body pose estimation for excavators based on data fusion of multiple onboard sensors

To reduce machine-related accidents on sites, automatically monitoring the full-body poses of operating heavy machines is crucial. Conventional pose estimation systems relying on homogeneous sensors are vulnerable to negative environmental impacts, leading to inaccurate and unstable estimation of machine states. Hence, a full-body pose estimation framework is proposed for excavators, with a data fusion strategy to utilize different types of onboard sensors for enhanced accuracy and robustness. Specifically, a non-invasive onboard visual-inertial sensor system is designed for data fusion. Then, through competitive and complementary data fusion, the keypoints describing the full-body poses of the excavator are tracked in 3D space. Especially, an EKF-based localization algorithm is developed for optimized multi-keypoint tracking, which is verified to improve the accuracy and robustness of pose estimation by a real-world excavator case study. The proposed sensor-fusion method can effectively improve operational safety, by accurately monitoring the motion of heavy machines operating on construction sites.

Relative Pose Estimation of Non-Cooperative Space Targets Using a TOF Camera

It is difficult to determine the accurate pose of non-cooperative space targets in on-orbit servicing (OOS). The visual camera is easily affected by the extreme light environment in space, and the scanning lidar will have motion distortion when the target moves at high speed. Therefore, we proposed a non-cooperative target pose-estimation system combining a registration and a mapping algorithm using a TOF camera. We first introduce the projection model of the TOF camera and proposed a new calibration method. Then, we introduce the three modules of the proposed method: the TOF data preprocessing module, the registration module and the model mapping module. We assembled the experimental platform to conduct semi-physical experiments; the results showed that the proposed method has the smallest translation error 8 mm and Euler angle error 1° compared with other classical methods. The total time consumption is about 100 ms, and the pose tracking frequency can reach 10 Hz. We can conclude that the proposed pose-estimation scheme can achieve the high-precision pose estimation of non-cooperative targets and meet the requirements necessary for aerospace applications.

Adaptive heading correction for an industrial heavy-duty omnidirectional robot

The paper deals with the design and testing of a robot for industrial applications featuring omnidirectionality thanks to the use of mecanum wheels. While this architecture provides remarkable manoeuvrability in narrow or cluttered spaces, it has some drawbacks that limit its widespread deployment in practice, especially for heavy-duty and long-duration tasks. As an example, the variability in the mecanum wheel rolling radius leads to undesired dynamic ill-effects, such as slippage and vibrations that affect the accuracy of pose estimation and tracking control systems. Drawing on the modeling of the kinematic and dynamic behaviour of the robot, these effects have been tackled within an adaptive estimation framework that adjusts the robot control system based on the properties of the surface being traversed. The proposed approach has been validated in experimental tests using a physical prototype operating in real industrial settings.

An effectual classical dance pose estimation and classification system employing Convolution Neural Network –Long ShortTerm Memory (CNN-LSTM) network for video sequences

Automatic human action recognition remains an intricate issue that includes extractions as well as classifying spatial features of individual pose on video remains as difficult problem for computer vision specialists. Human activity is described as temporal variation of the human body. Capturing and assessing multimedia content of dance remains beneficial for the conservation of cultural heritage, for creating video recommendation systems, for aiding learners to employ tutoring systems, and so on. Indian classical dance (ICD) classification remains a fascinating discipline due to its intricate body posture. This gives a stage for examining diverse computer vision alongside deep learning notions. Through modification in learning styles, automated teaching solutions became unavoidable over all disciplines, from classical to online forums. Furthermore, ICD becomes a necessary portion of a prosperous cultural and heritage that at any means should be updated and conserved. Although several classification approaches prevail for bidimensional dance images, in this study, we define the issue of dance style classification for classifying Indian dance. The dance consists of intricate poses like full-body rotation and self-occlusion, and we proffer three-phase deep learning techniques. Initially, we excerpted necessary joint positions of the dance out of every video frame by pre-trained paradigm, TensorFlow Mobile Net Architecture, that assists us in assessing whatsoever body position within the frame. Next, correlative or identification of the determined action factors by Cosine Similarity was employed. Lastly, we classified the dance pose by excerpted salient details and trained this employing Convolution Neural Network – Long Short Term Memory (CNN-LSTM) Network structure utilizing training dataset of system for classification. We incorporate a feature vector of flow, angles linking anchor joint, and regularised remoteness vectors that catch proximity configuration in skeletal design rather than providing an informal chain of joints in sequence learner that ordinarily obstructs network execution. Therefore, kinematic association amidst body joints through frames employing pose assessment aids in finer showing of the Spatio-temporal reliance. The proffered CNN-LSTM structure for dance classification is compared with Multilayer Perceptron (MLP), Convolutional LSTM (LRCN), Multilayer Perceptron (MLP), CONB3D and LSTM concerning criteria like F1-score, accuracy, recall, AUC curve and precision. Therefore, determined that the proffered CNN-LSTM classification structure attains 98.53% of accuracy, 99.04% of precision, 98.47% of recall, 99.12% of AUC score, and 98.72% of F1-score during the examining procedure. During the training procedure, this attains 94.01% of accuracy, 93.11% of precision, 94.76% of recall, 96.06% of AUC score, and 93.51% of F1-score.

The reliability and validity of gait analysis system using 3D markerless pose estimation algorithms.

Quantifying kinematic gait for elderly people is a key factor for consideration in evaluating their overall health. However, gait analysis is often performed in the laboratory using optical sensors combined with reflective markers, which may delay the detection of health problems. This study aims to develop a 3D markerless pose estimation system using OpenPose and 3DPoseNet algorithms. Moreover, 30 participants performed a walking task. Sample entropy was adopted to study dynamic signal irregularity degree for gait parameters. Paired-sample t-test and intra-class correlation coefficients were used to assess validity and reliability. Furthermore, the agreement between the data obtained by markerless and marker-based measurements was assessed by Bland–Altman analysis. ICC (C, 1) indicated the test–retest reliability within systems was in almost complete agreement. There were no significant differences between the sample entropy of knee angle and joint angles of the sagittal plane by the comparisons of joint angle results extracted from different systems (p > 0.05). ICC (A, 1) indicated the validity was substantial. This is supported by the Bland–Altman plot of the joint angles at maximum flexion. Optical motion capture and single-camera sensors were collected simultaneously, making it feasible to capture stride-to-stride variability. In addition, the sample entropy of angles was close to the ground_truth in the sagittal plane, indicating that our video analysis could be used as a quantitative assessment of gait, making outdoor applications feasible.

GoPose

This paper presents GoPose, a 3D skeleton-based human pose estimation system that uses WiFi devices at home. Our system leverages the WiFi signals reflected off the human body for 3D pose estimation. In contrast to prior systems that need specialized hardware or dedicated sensors, our system does not require a user to wear or carry any sensors and can reuse the WiFi devices that already exist in a home environment for mass adoption. To realize such a system, we leverage the 2D AoA spectrum of the signals reflected from the human body and the deep learning techniques. In particular, the 2D AoA spectrum is proposed to locate different parts of the human body as well as to enable environment-independent pose estimation. Deep learning is incorporated to model the complex relationship between the 2D AoA spectrums and the 3D skeletons of the human body for pose tracking. Our evaluation results show GoPose achieves around 4.7cm of accuracy under various scenarios including tracking unseen activities and under NLoS scenarios.

Affordable Motion Tracking System for Intuitive Programming of Industrial Robots.

The paper deals with a lead-through method of programming for industrial robots. The goal is to automatically reproduce 6DoF trajectories of a tool wielded by a human operator demonstrating a motion task. We present a novel motion-tracking system built around the HTC Vive pose estimation system. Our solution allows complete automation of the robot teaching process. Specific algorithmic issues of system calibration and motion data post-processing are also discussed, constituting the paper’s theoretical contribution. The motion tracking system is successfully deployed in a pilot application of robot-assisted spray painting.

A Tensor‐based Regression Approach for Human Motion Prediction

AbstractCollaborative robotic systems will be a key enabling technology for current and future industrial applications. The main aspect of such applications is to guarantee safety for humans. To detect hazardous situations, current commercially available robotic systems rely on direct physical contact to the co‐working person. To further advance this technology, there are multiple efforts to develop predictive capabilities for such systems. Using motion tracking sensors and pose estimation systems combined with adequate predictive models, potential episodes of hazardous collisions between humans and robots can be predicted. Based on the provided predictive information, the robotic system can avoid physical contact by adjusting speed or position. A potential approach for such systems is to perform human motion prediction with machine learning methods like artificial neural networks (NNs). In our approach, the motion patterns of past seconds are used to predict future ones by applying a linear Tensor‐on‐Tensor Regression model, selected according to a similarity measure between motion sequences obtained by dynamic time warping (DTW). For test and validation of our proposed approach, industrial pseudo assembly tasks were recorded with a motion capture system, providing unique traceable Cartesian coordinates for each human joint. The prediction of repetitive human motions associated with assembly tasks, whose data vary significantly in length and have highly correlated variables, has been achieved in real time.

Gesture controlled interaction using hand pose model

Direct use of the hand as an input device is a smart method for providing natural human-computer interaction (HCI). Hand pose estimation is an attractive topic for research in recent years. It has been widely used in virtual reality. The domain of computer vision-based human hand three-dimensional shape and hand pose estimation has fascinated momentous attention recently due to its key role in various applications, such as natural human-computer interactions. Hand pose estimation is difficult due to some challenges. First, we need to detect the human hand which is very changeable. Second, the high degree of freedom leads to difficulties in pose estimation. In this paper, we aim to build a hand pose estimation system that can correctly detect a human hand and estimate its pose which can be useful in the areas like industrial automation, sign language recognition etc. We integrated a hand pose model with a game named U3D with 6 different gestures. As the object detection methods perform poorly in the palm detection tasks we used 21 hand joints and increased accuracy to approximately 95.63%. We used the Blender 3D computer graphics software toolset for creating animation, visual effects, motion graphics, and interactive unity 3D games.

A multi-sensor architecture combining human pose estimation and real-time location systems for workflow monitoring on hybrid operating suites

Despite the advancements to improve patient safety, a significant number of errors still occur in Operating Suites (OS). To improve medical decision-making and the resulting quality of care, it is essential to monitor and understand medical activities’ workflow and interactions. Although some strategies employ different sensor devices, their focus is not on generating complete workflow information. They only combine different data sources to generate their final output, lacking at least one piece of information from a workflow. To tackle this challenge, this paper presents ▪ , a distributed architecture model for sensor data acquisition and processing. ▪ ’s main contribution lies in its multi-sensor data fusion algorithms to extract a computational representation of activities in surgical procedures. In addition, ▪ is flexible to accommodate different deployment configurations,combining depth cameras, Ultra-Wideband positioning systems,and deep learning-based human pose estimation (HPE) mechanisms. The workflow monitoring mechanism was deployed in an actual hybrid OS for an extensive evaluation of the proposal. Our experiments demonstrate that the architecture can capture the information required to monitor the surgical workflow. In particular, the proposed HPE methodology accurately detects the poses of medical staff members, with a maximum error of 5cm.