Joints Of Human Body Research Articles

Hierarchical Contextual Refinement Networks for Human Pose Estimation.

Predicting human pose in the wild is a challenging problem due to high flexibility of joints and possible occlusion. Existing approaches generally tackle the difficulties either by holistic prediction or multi-stage processing, which suffer from poor performance for locating challenging joints or high computational cost. In this paper, we propose a new Hierarchical Contextual Refinement Network (HCRN) to robustly predict human poses in an efficient manner, where human body joints of different complexities are processed at different layers in a context hierarchy. Different from existing approaches, our proposed model predicts positions of joints from easy to difficult in a single stage through effectively exploiting informative contexts provided in the previous layer. Such approach offers two appealing advantages over state-of-the-arts: (1) more accurate than predicting all the joints together and (2) more efficient than multi-stage processing methods. We design a Contextual Refinement Unit (CRU) to implement the proposed model, which enables auto-diffusion of joint detection results to effectively transfer informative context from easy joints to difficult ones. In this way, difficult joints can be reliably detected even in presence of occlusion or severe distracting factors. Multiple CRUs are organized into a tree-structured hierarchy which is end-to-end trainable and does not require processing joints for multiple iterations. Comprehensive experiments evaluate the efficacy and efficiency of the proposed HCRN model to improve well-established baselines and achieve new state-of-the-art on multiple human pose estimation benchmarks.

Multi-Person Pose Estimation via Multi-Layer Fractal Network and Joints Kinship Pattern.

We propose an effective method to boost the accuracy of multi-person pose estimation in images. Initially, the three-layer fractal network was constructed to regress multi-person joints location heatmap that can help to enhance an image region with receptive field and capture more joints local-contextual feature information, thereby producing keypoints heatmap intermediate prediction to optimize human body joints regression results. Subsequently, the hierarchical bi-directional inference algorithm was proposed to calculate the degree of relatedness (call it Kinship) for adjacent joints, and it combines the Kinship between adjacent joints with the spatial constraints, which we refer to as joints kinship pattern matching mechanism, to determine the best matched joints pair. We iterate the above-mentioned joints matching process layer by layer until all joints are assigned to a corresponding individual. Comprehensive experiments demonstrate that the proposed approach outperforms the state-of-the-art schemes and achieves about 1% and 0.6% increase in mAP on MPII multi-person subset and MSCOCO 2016 keypoints challenge.

Combining fractal hourglass network and skeleton joints pairwise affinity for multi-person pose estimation

Human pose estimation, especially multi-person pose estimation, is vital for understanding human abnormal behavior. In this paper, we develop a fractal hourglass model to automatically regress human body joints, and propose a layered double-way inference algorithm to calculate the affinity between neighboring skeleton joints. Firstly, the original hourglass resident unit was replaced and the candidate skeleton joints location heatmap regression process was described. And then, we determine the specific body joints location and optimize the regression results. Next, the double-way conditional probabilities between adjacent joints is defined as joints pairwise affinity, and is applied to match adjacent human body part. What’s more, we adopt the spatial distance constraint to refine body joints matching result. Finally, we connect the best matching joints-pair, and iterate the process until all candidate joints are assigned into individual. Extensive experiments on the MPII multi-person subset and the COCO 2016 keypoints challenge show the effectiveness of our method, outperforming the second best method (Associative Embedding) by 0.45 and 1.20%.

Human action recognition in videos with articulated pose information by deep networks

Action recognition is of great importance in understanding human motion from video. It is an important topic in computer vision due to its many applications such as video surveillance, human–machine interaction and video retrieval. One key problem is to automatically recognize low-level actions and high-level activities of interest. This paper proposes a way to cope with low-level actions by combining information of human body joints to aid action recognition. This is achieved by using high-level features computed by a convolutional neural network which was pre-trained on Imagenet, with articulated body joints as low-level features. These features are then used to feed a Long Short-Term Memory network to learn the temporal dependencies of an action. For pose prediction, we focus on articulated relations between body joints. We employ a series of residual auto-encoders to produce multiple predictions which are then combined to provide a likelihood map of body joints. In the network topology, features are processed across all scales which capture the various spatial relationships associated with the body. Repeated bottom-up and top-down processing with intermediate supervision of each auto-encoder network is applied. We demonstrate state-of-the-art results on the popular FLIC, LSP and UCF Sports datasets.

3D human pose estimation from depth maps using a deep combination of poses

Many real-world applications require the estimation of human body joints for higher-level tasks as, for example, human behaviour understanding. In recent years, depth sensors have become a popular approach to obtain three-dimensional information. The depth maps generated by these sensors provide information that can be employed to disambiguate the poses observed in two-dimensional images. This work addresses the problem of 3D human pose estimation from depth maps employing a Deep Learning approach. We propose a model, named Deep Depth Pose (DDP), which receives a depth map containing a person and a set of predefined 3D prototype poses and returns the 3D position of the body joints of the person. In particular, DDP is defined as a ConvNet that computes the specific weights needed to linearly combine the prototypes for the given input. We have thoroughly evaluated DDP on the challenging ‘ITOP’ and ‘UBC3V’ datasets, which respectively depict realistic and synthetic samples, defining a new state-of-the-art on them.

Recognizing Involuntary Actions from 3D Skeleton Data Using Body States

Human action recognition has been one of the most active fields of research in computer vision for last years. Two dimensional action recognition methods are facing serious challenges such as occlusion and missing the third dimension of data. Development of depth sensors has made it feasible to track positions of human body joints over time. This paper proposes a novel method of action recognition which uses temporal 3D skeletal Kinect data. This method introduces the definition of body states and then every action is modeled as a sequence of these states. The learning stage uses Fisher Linear Discriminant Analysis (LDA) to construct discriminant feature space for discriminating the body states. Moreover, this paper suggests the use of the Mahalonobis distance as an appropriate distance metric for the classification of the states of involuntary actions. Hidden Markov Model (HMM) is then used to model the temporal transition between the body states in each action. According to the results, this method significantly outperforms other popular methods, with recognition rate of 88.64% for eight different actions and up to 96.18% for classifying fall actions.

Cross-View Person Identification by Matching Human Poses Estimated With Confidence on Each Body Joint

Cross-view person identification (CVPI) from multiple temporally synchronized videos taken by multiple wearable cameras from different, varying views is a very challenging but important problem, which has attracted more interests recently. Current state-of-the-art performance of CVPI is achieved by matching appearance and motion features across videos, while the matching of pose features does not work effectively given the high inaccuracy of the 3D human pose estimation on videos/images collected in the wild. In this paper, we introduce a new metric of confidence to the 3D human pose estimation and show that the combination of the inaccurately estimated human pose and the inferred confidence metric can be used to boost the CVPI performance---the estimated pose information can be integrated to the appearance and motion features to achieve the new state-of-the-art CVPI performance. More specifically, the estimated confidence metric is measured at each human-body joint and the joints with higher confidence are weighted more in the pose matching for CVPI. In the experiments, we validate the proposed method on three wearable-camera video datasets and compare the performance against several other existing CVPI methods.

A Real-Time Lift Detection Strategy for a Hip Exoskeleton.

Repetitive lifting of heavy loads increases the risk of back pain and even lumbar vertebral injuries to workers. Active exoskeletons can help workers lift loads by providing power assistance, and therefore reduce the moment and force applied on L5/S1 joint of human body when performing lifting tasks. However, most existing active exoskeletons for lifting assistance are unable to automatically detect user's lift movement, which limits the wide application of active exoskeletons in factories. In this paper, we propose a simple but effective lift detection strategy for exoskeleton control. This strategy uses only exoskeleton integrated sensors, without any extra sensors to capture human motion intentions. This makes the lift detection system more practical for applications in manufacturing environments. Seven healthy subjects participated in this research. Three different sessions were carried out, two for training and one for testing the algorithm. In the two training sessions, subjects were asked to wear a hip exoskeleton, controlled in transparent mode, and perform repetitive lifting and a locomotion circuit; lifting was executed with different techniques. The collected data were used to train the lift detection model. In the testing session, the exoskeleton was controlled in order to deliver torque to assist the lifting action, based on the lift detection made by the trained algorithm. The across-subject average accuracy of lift detection during online test was 97.97 ± 1.39% with subject-dependent model. Offline, the algorithm was trained with data acquired from all subjects to verify its performance for subject-independent detection, and an accuracy of 97.48 ± 1.53% was achieved. In addition, timeliness of the algorithm was quantitatively evaluated and the time delay was <160 ms across different lifting speeds. Surface electromyography was also measured to assess the efficacy of the exoskeleton in assisting subjects in performing load lifting tasks. These results validate the promise of applying the proposed lift detection strategy for exoskeleton control aiming at lift assistance.

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Body pose is an important indicator of human actions. The existing pose-based action recognition approaches are typically designed for individual human bodies and require a fixed-size (e.g., $$13\times 2$$ ) input vector. This requirement is questionable and may degrade the recognition accuracy, particularly for real-world videos, in which scenes with multiple people or partially visible bodies are common. Inspired by the recent success of convolutional neural networks (CNNs) in various computer vision tasks, we propose an approach based on a deep neural network architecture for 2D pose-based action recognition tasks in this work. To this end, a human pose encoding scheme is designed to eliminate the above requirement and to provide a general representation of 2D human body joints, which can be used as the input for CNNs. We also propose a weighted fusion scheme to integrate RGB and optical flow with human pose features to perform action classification. We evaluate our approach on two real-world datasets and achieve better performances compared to state-of-the-art approaches.

Automated segmentation of knee thermal imaging and X-ray in evaluation of rheumatoid arthritis

Rheumatoid arthritis (RA) is a long lasting autoimmune disorder that affects the multiple joints of human body. The aim and objective of the study was i) to implement the automated segmentation of knee x-ray image and thermal image using fuzzy c means and canny edge detection algorithm. ii) To compare both the imaging modalities by means of feature extraction and correlate with the biochemical method as standard. Fifteen subjects with RA in knee region and 15 healthy controls were included in this study. The segmentation of thermal images was performed using fuzzy c-means algorithm and x-ray segmentation was implemented using canny edge detection algorithm. The skin surface temperature weremeasured in the thermal image of knee regionin both RA and control subjects. The features wereextracted from the segmented region of the knee x-ray image. The automated segmentation implemented in thermal imaging provided better results compared to x-ray image segmentation process. The thermal imaging feature and x-ray imaging features correlated significantly with the standard parameters.

Exploiting deep residual networks for human action recognition from skeletal data

The computer vision community is currently focusing on solving action recognition problems in real videos, which contain thousands of samples with many challenges. In this process, Deep Convolutional Neural Networks (D-CNNs) have played a significant role in advancing the state-of-the-art in various vision-based action recognition systems. Recently, the introduction of residual connections in conjunction with a more traditional CNN model in a single architecture called Residual Network (ResNet) has shown impressive performance and great potential for image recognition tasks. In this paper, we investigate and apply deep ResNets for human action recognition using skeletal data provided by depth sensors. Firstly, the 3D coordinates of the human body joints carried in skeleton sequences are transformed into image-based representations and stored as RGB images. These color images are able to capture the spatial-temporal evolutions of 3D motions from skeleton sequences and can be efficiently learned by D-CNNs. We then propose a novel deep learning architecture based on ResNets to learn features from obtained color-based representations and classify them into action classes. The proposed method is evaluated on three challenging benchmark datasets including MSR Action 3D, KARD, and NTU-RGB+D datasets. Experimental results demonstrate that our method achieves state-of-the-art performance for all these benchmarks whilst requiring less computation resource. In particular, the proposed method surpasses previous approaches by a significant margin of 3.4% on MSR Action 3D dataset, 0.67% on KARD dataset, and 2.5% on NTU-RGB+D dataset.

Improving joint position estimation of Kinect using anthropometric constraint based adaptive Kalman filter for rehabilitation

The role of Microsoft Kinect in rehabilitation and clinical assessments of elderly and stroke patients is growing due to its affordability and unobtrusiveness in assessing human body joint kinematics. However, Kinect data is highly contaminated by noise due to the nature of the structured light based system. The noise characteristics vary for static and dynamic joints effecting the anthropometric measurement of the body segments connecting any two joints. Here, we use Kalman filter to track the body joints and to denoise the unwanted joint vibrations. Initially, the process noise of Kalman filter is chosen based on minimization of standard deviation of the position of static joints and the latency (delay) in velocity adaptation for dynamic joints. Further, the measurement noise is dynamically adapted based on the velocity of the joints. Additionally, a differential evolutionary based constraint is applied to minimize the variation in the estimated body segment length. The proposed method outperforms the existing techniques like low pass filtering, exponential and double exponential filtering methods in both static and dynamic conditions. Experimental results for Kinect Xbox 360 show atleast 52% and 32% improvement in standard deviation of bone length, associated with joints under motion, for 26 healthy subjects for range of motion (ROM) and single limb standing respectively. We found 40% improvement in deviation of bone length, when tested on 10 stroke patient’s data. We also tested our algorithm on healthy population for ROM exercises in Kinect Xbox One and achieved an average improvement of 34% in variation in bone length.

A highly stretchable and stable strain sensor based on hybrid carbon nanofillers/polydimethylsiloxane conductive composites for large human motions monitoring

Stretchable strain sensors have promising potentials in wearable electronics for human motion detection, health monitoring and so on. A reliable strain sensor with high flexibility and good stability should be designed to detect human joints motions with a large deformation. Here, a simple and facile solution mixing-casting method was adopted to fabricate a highly stretchable strain sensor based on composites mixing polydimethylsiloxane (PDMS) with hybrid carbon nanotubes (CNTs) and carbon black (CB) conductive nanofillers (CNTs-CB). Bridged and overlapped hybrid CNTs-CB nanofillers structure was achieved in the composite on the basis of the morphology observation. In monotonic stretching test, the CNTs-CB/PDMS composites strain sensors exhibited high stretchability, strain-dependent sensitivity in a wide strain sensing range (ca. 300% strain) and an excellent linear current-voltage behavior. During stretching-releasing cycles, the strain sensors presented excellent repeatability, good stability and superior durability (2500 cycles at 200% strain). Combined with the above outstanding strain sensing performances, the fabricated stretchable strain sensors were attached onto different joints of human body to monitor the corresponding human motions, demonstrating their attractive perspective in large human motions detection.

Effect of body mass index and chair height on the torque developed at the knee joint during back-to-sit and sit-to-stand movements

Back-to-sit (BTS) and sit-to-stand (STS) are one among many vital functions of knee joint of human body. While designing bipedal robots, the knee joint should have the capacity to replicate the characteristics, such as that of human beings, especially the torque. The effect of weight and length as well as sitting chair height and speed of movement during BTS and STS has to be studied to limit the weight and height of robots or exoskeleton. Here we have used a camera and motion analysis software to acquire the data and studied the effect and interaction of body mass index and chair height on knee torque at various sitting speeds using two-way analysis of variance.

Review on Design and Control Aspects of Robotic Shoulder Rehabilitation Orthoses

Robotic rehabilitation devices are more frequently used for the physical therapy of people with upper limb weakness, which is the most common type of stroke-induced disability. Rehabilitation robots can provide customized, prolonged, intensive, and repetitive training sessions for patients with neurological impairments. In most cases, the robotic exoskeletons have to be aligned with the human joints and provide natural arm movements. This is a challenging task to achieve for one of the most biomechanically complex joints of human body, i.e., the shoulder. Therefore, specific considerations have been made in the development of various existing robotic shoulder rehabilitation orthoses. Different types of actuation, degrees of freedom (DOFs), and control strategies have been utilized for the development of these shoulder rehabilitation orthoses. This paper presents a comprehensive review of these shoulder rehabilitation orthoses. Recent advancements in the mechanism design, their advantages and disadvantages, overview of hardware, actuation system, and power transmission are discussed in detail with the emphasis on the assisted DOFs for shoulder motion. A brief overview of control techniques and clinical studies conducted with the developed robotic shoulder orthoses is also presented. Finally, current challenges and directions of future development for robotic shoulder rehabilitation orthoses are provided at the end of this paper.

Research on the Optimization Method of Arm Movement in the Assembly Workshop Based on Ergonomics

In order to improve the work efficiency and comfortability, Ergonomics is used to research the work of the operator in the assembly workshop. An optimization algorithm of arm movement in the assembly workshop is proposed. In the algorithm, a mathematical model of arm movement is established based on multi rigid body movement model and D-H method. The solution of inverse kinematics equation on arm movement is solved through kinematics theory. The evaluation functions of each joint movement and the whole arm movement are given based on the comfortability of human body joint. The solution method of the optimal arm movement posture based on the evaluation functions is described. The software CATIA is used to verify that the optimal arm movement posture is valid in an example and the experimental result show the effectiveness of the algorithm.

Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates.

Skeleton-based human action recognition has attracted a lot of research attention during the past few years. Recent works attempted to utilize recurrent neural networks to model the temporal dependencies between the 3D positional configurations of human body joints for better analysis of human activities in the skeletal data. The proposed work extends this idea to spatial domain as well as temporal domain to better analyze the hidden sources of action-related information within the human skeleton sequences in both of these domains simultaneously. Based on the pictorial structure of Kinect's skeletal data, an effective tree-structure based traversal framework is also proposed. In order to deal with the noise in the skeletal data, a new gating mechanism within LSTM module is introduced, with which the network can learn the reliability of the sequential data and accordingly adjust the effect of the input data on the updating procedure of the long-term context representation stored in the unit's memory cell. Moreover, we introduce a novel multi-modal feature fusion strategy within the LSTM unit in this paper. The comprehensive experimental results on seven challenging benchmark datasets for human action recognition demonstrate the effectiveness of the proposed method.

Development of an economic wireless human motion analysis device for quantitative assessment of human body joint

In recent years, the study of human body dynamics has been attracting a significant amount of attention. Currently there are many camera or active sensor based motion analysis systems available on the market. They have been extensively adopted and used by the film and animation or entertainment industries such as film and video game producers. More recently their potential in studying human dynamics/motion for medical purposes has been realised to the extent that they are now used to study full body human biomechanics in the form of gait analysis systems. Most orthopaedic surgeries are usually about joint repair or implants. According health line, revision surgery is usually due to infection, continued pain, joint stiffness, wear, instability, loosening. Apart from infection, the rest can be linked to the operation itself. Currently, surgical planning and placing implants is performed in a subjective manner, relying on the surgeon’s experience and instinct, current systems to help the surgeon to place implant are also bulky, expensive, slow and not user friendly. The aim of this project is to develop an economic and portable motion assessment system which involves a wireless inertial measurement unit (IMU) dedicated to study and assess body joints. Through the data collected from the IMU, the system is capable real time measurement of relative position and orientation of the human joint. Several tests were conducted to validate the data extracted from gyroscope and accelerometer of the IMU. The joint motion results analysed using the device was compared with the results analysed using commercial video motion analysis software and it shows good correlation. It is found that the gyroscope of the IMU under DMP sensor fusion algorithm and calibration capability is able to give the angular velocity with less than 5% error. This has led to a more accurate orientation data which gives 7% error in average bending angle.

Geometrical kinematic modeling on human motion using method of multi-sensor fusion

Human motion sensing based on wearable sensors could be viewed as a multi-objects tracking issue of human body joints. Sensor drift errors and distortion are the main challenges of the tracking accuracy of human motion. Traditional filtering and fusion methods, such as Kalman filter, can to some extent reduce the instantaneous error but cannot avoid sensor drift fundamentally. Physical characteristics of human body should be considered and human motion models should be exhibited to describe human motions. The existing models such as skeleton model and cylinder model are either too simple or too complicated for practical applications. In this study, we put forward a geometrical kinematic characteristics based human motion model. The whole human body is viewed as an articulated skeleton and Denavit–Hartenberg convention is adopted to describe the forward kinematics structure. Theoretical analysis is conducted with the derivation of Posterior Cramer–Rao Lower Bound (PCRLB) in human movement scenes based on proposed model. Significant superiority is shown in simulation results. An experiment on human lower limb motions is carried out to verify the validity of the proposed human motion model in practice applications, from the angles of both capturing accuracy and energy consumption. The capturing accuracy has an obvious increase in the testing results, with acceptable energy consumption. It is far more efficient than traditional methods.

Multiple human 3D pose estimation from multiview images

Multiple human 3D pose estimation is a challenging task. It is mainly because of large variations in the scale and pose of humans, fast motions, multiple persons in the scene, and arbitrary number of visible body parts due to occlusion or truncation. Some of these ambiguities can be resolved by using multiview images. This is due to the fact that more evidences of body parts would be available in multiple views. In this work, a novel method for multiple human 3D pose estimation using evidences in multiview images is proposed. The proposed method utilizes a fully connected pairwise conditional random field that contains two types of pairwise terms. The first pairwise term encodes the spatial dependencies among human body joints based on an articulated human body configuration. The second pairwise term is based on the output of a 2D deep part detector. An approximate inference is then performed using the loopy belief propagation algorithm. The proposed method is evaluated on the Campus, Shelf, Utrecht Multi-Person Motion benchmark, Human3.6M, KTH Football II, and MPII Cooking datasets. Experimental results indicate that the proposed method achieves substantial improvements over the existing state-of-the-art methods in terms of the probability of correct pose and the mean per joint position error performance measures.