Skeleton Tracking Research Articles

Intercity rail platform abnormal action recognition based on a skeleton tracking and recognition framework

Intercity rail platform abnormal action recognition based on a skeleton tracking and recognition framework

Walking representation and simulation based on multi-source image fusion and multi-agent reinforcement learning for gait rehabilitation

In the formulation of strategies for walking rehabilitation, achieving precise identification of the current state and making rational predictions about the future state are crucial but often unrealized. To tackle this challenge, our study introduces a unified framework that integrates a novel 3D walking motion capture method using multi-source image fusion and a walking rehabilitation simulation approach based on multi-agent reinforcement learning. We found that, (i) the proposal achieved an accurate 3D walking motion capture and outperforms other advanced methods. Experimental evidence indicates that, compared to similar visual skeleton tracking methods, the proposed approach yields results with higher Pearson correlation (r=0.93), intra-class correlation coefficient (ICC(2,1)=0.91), and narrower confidence intervals ([0.90,0.95] for r, [0.88,0.94] for ICC(2,1)) when compared to standard results. The outcomes of the proposed approach also exhibit commendable correlation and concurrence with those obtained through the IMU-based skeleton tracking method in the assessment of gait parameters ([0.85,0.89] for r, [0.75,0.81] for ICC(2,1)); (ii) multi-agent reinforcement learning has the potential to be used to solve the simulation task of gait rehabilitation. In mimicry experiment, our proposed simulation method for gait rehabilitation not only enables the intelligent agent to converge from the initial state to the target state, but also observes evolutionary patterns similar to those observed in clinical practice through motor state resolution. This study offers valuable contributions to walking rehabilitation, enabling precise assessment and simulation-based interventions, with potential implications for clinical practice and patient outcomes.

Experimental implementation of skeleton tracking for collision avoidance in collaborative robotics

Collaborative robotic manipulators can stop in case of a collision, according to the ISO/TS 15066 and ISO 10218-1 standards. However, in a human-robot collaboration scenario where the robot and the human share the workspace, a better solution with concerns to production and operator safety would be to perceive in advance the presence of an obstacle and be able to avoid it, thereby completing the task without halting the robot. In this paper, an obstacle avoidance algorithm is tested using a sensor system for real-time human detection; the operator represents a potential dynamic obstacle that can interfere with the robot motion. The sensor system consists of three RGB-D cameras. A custom software framework has been developed in Python exploiting machine learning tools for human skeleton detection. The coordinates of the human body joints relative to the manipulator base are used as input to the obstacle avoidance algorithm. The use of multiple sensors makes it possible to limit the occlusion problem; in addition, the choice of non-wearable sensors goes in the direction of better operator comfort. A series of experimental tests were performed to verify the accuracy of skeleton detection and the ability of the system to avoid obstacles in real time. The human motion caption system, in particular, was validated through a comparison with a commercial system based on wearable IMU sensors widely used and validated in motion capture. There is an NRMSE of 3.23%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$3.23\\%$$\\end{document} for the RGB-D camera-based skeleton detection system, against an NRMSE of 12.23%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$12.23\\%$$\\end{document} for the IMU wearable sensor system. Test results confirm that the system is able to avoid collisions with the human body under various conditions, static or dynamic, ensuring a minimum safety distance to any part of the manipulator. In 44 tests in which the operator moves around the robot with possible collisions (at a speed typical of manufacturing operations), the minimum operator-robot distance averaged 208mm\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$208\\,\ extrm{mm}$$\\end{document}, being 200mm\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$200\\,\ extrm{mm}$$\\end{document} the limit safety distance set by algorithm.

Skeleton-based Human Activity Classification in Sparse Image Sequences

Research results on human activity classification in video are described, based on initial human skeleton estimation in video frames. Both single person actions and two-person interactions are considered. The initial skeleton data is estimated in selected video frames by OpenPose, HRNet or other dedicated library. Important contributions of presented work are computational steps of skeleton tracking and -refinement, and relational feature extraction from pairs of skeleton joints. It is shown, that this feature engineering significantly increases the classification accuracy. Regarding the final neural network encoder-classifier, two different architectures are designed and tested. The first solution is a lightweight MLP network, implementing the idea of a "mixture of pose experts". Several pose classifiers (experts) are trained independently on different time periods (snapshots) of single-person visual actions (or 2-person interactions), while the final classification is a time-related pooling of weighted expert classifications. All pose experts use the same deep encoding network. The second (middle weight) solution is based on a LSTM network.Both solutions are trained and tested on the action set of the well-known NTU RGB+D dataset, although only 2D data are used.Our results show comparable performance with some of the best reported STM- and CNN-based classifiers for this dataset. We conclude that by reducing the noise of skeleton data, highly successful lightweight- and midweight-approaches to visual activity recognition in image sequences can be achieved.

Joint angle estimation during shoulder abduction exercise using contactless technology

BackgroundTele-rehabilitation, also known as tele-rehab, uses communication technologies to provide rehabilitation services from a distance. The COVID-19 pandemic has highlighted the importance of tele-rehab, where the in-person visits declined and the demand for remote healthcare rises. Tele-rehab offers enhanced accessibility, convenience, cost-effectiveness, flexibility, care quality, continuity, and communication. However, the current systems are often not able to perform a comprehensive movement analysis. To address this, we propose and validate a novel approach using depth technology and skeleton tracking algorithms.MethodsOur data involved 14 participants (8 females, 6 males) performing shoulder abduction exercises. We collected depth videos from an LiDAR camera and motion data from a Motion Capture (Mocap) system as our ground truth. The data were collected at distances of 2 m, 2.5 m, and 3.5 m from the LiDAR sensor for both arms. Our innovative approach integrates LiDAR with the Cubemos and Mediapipe skeleton tracking frameworks, enabling the assessment of 3D joint angles. We validated the system by comparing the estimated joint angles versus Mocap outputs. Personalized calibration was applied using various regression models to enhance the accuracy of the joint angle calculations.ResultsThe Cubemos skeleton tracking system outperformed Mediapipe in joint angle estimation with higher accuracy and fewer errors. The proposed system showed a strong correlation with Mocap results, although some deviations were present due to noise. Precision decreased as the distance from the camera increased. Calibration significantly improved performance. Linear regression models consistently outperformed nonlinear models, especially at shorter distances.ConclusionThis study showcases the potential of a marker-less system, to proficiently track body joints and upper-limb angles. Signals from the proposed system and the Mocap system exhibited robust correlation, with Mean Absolute Errors (MAEs) consistently below 10∘\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$10^\\circ$$\\end{document}. LiDAR’s depth feature enabled accurate computation of in-depth angles beyond the reach of traditional RGB cameras. Altogether, this emphasizes the depth-based system’s potential for precise joint tracking and angle calculation in tele-rehab applications.

MotionRender: A simple Python implementation of video motion visualization for 3D motion capture data

We describe a Python library project for generating animations of simple 3D motion capture data points. The motion capture method is agnostic with respect to this visualizaiton tool, expecting a time series of time stamped coordinates captured in a 3-dimensional space. One of the advantages of this tool is the simple data format design. This tool was developed originally for a Kinect system implementing skeleton tracking of 15 joint positions, where each data point consists of an accurate time of capture (time stamp), and 3 accurate coordinate (x, y, z) positions of each of the 15 joints at each time step. In addition to the motion capture time series, all that is needed is a joint graph file describing the relationship of the joint edges to one another. This library can be extended to visualize and create animations of motion capture data that can be reformulated using this basic structure of a time series of time stamped 3D coordinates and a joint graph description file. This library should be useful as is, or with easy modification, for many such visualization requirements of similar motion capture data.

Face orientation recognition based on Kinectv2 and an improved ELM model

Abstract Face orientation recognition is a crucial component of human behavior analysis. Challenges such as face occlusion and varying lighting conditions frequently degrade recognition accuracy. This study introduces an innovative approach using the Kinect sensor’s skeleton tracking technology to capture face orientation images across different lighting scenarios. To enhance image quality, techniques such as light attenuation and compensation are applied. Furthermore, multiple feature vectors based on face orientation are extracted and fused, followed by the optimization of the Extreme Learning Machine (ELM) model through the Archimedean Optimization Algorithm (AOA). The developed AOA-ELM model for face orientation recognition is compared with traditional models, including Back Propagation (BP), Radial Basis Function (RBF), standard ELM, and Genetic Algorithm enhanced ELM (GA-ELM). The AOA-ELM model exhibits superior performance in estimating face orientation amidst lighting disturbances, achieving an accuracy exceeding 95% in varied lighting conditions, thereby outperforming other classification models.

In-situ ergonomics assessment of industrial manual tasks through AR-headset integrated sensor-systems

This paper presents an approach that is based on the use of sensors integrated into Augmented Reality (AR) headsets to acquire data for in-situ ergonomics assessment of industrial tasks. In the context of human factors evaluation, it is typically required to collect data related to anthropometric characteristics of the operators e.g. height, length of arms, width of shoulders etc., as well as data related to the execution of manufacturing tasks including working postures, task repetition, task duration, and so forth. To this end, it is needed to develop a plug and play system to collect ergonomics data from the field. The proposed approach relies on head-mounted AR devices and their integrated RGB and Depth sensors, image and point cloud processing, and skeleton tracking techniques to calculate the required data, which are later visualized in a user-friendly interface. The proposed approach was tested in a case study where the objective was to identify the proper mechanical adjustments of an exoskeleton on the spot for comfortable fitting for each operator. Five operators participated in the testing that was held under varying lighting conditions. The solution showcased improved efficiency in decision-making compared to the commonplace procedures for skeleton fitting, faster setup time compared to typical camera-based approaches, and satisfactory performance.

Pick the Right Co-Worker: Online Assessment of Cognitive Ergonomics in Human–Robot Collaborative Assembly

Human-robot collaborative assembly systems enhance the efficiency and productivity of the workplace but may increase the workers’ cognitive demand. This paper proposes an online and quantitative framework to assess the cognitive workload induced by the interaction with a co-worker, either a human operator or an industrial collaborative robot with different control strategies. The approach monitors the operator’s attention distribution and upper-body kinematics benefiting from the input images of a low-cost stereo camera and cutting-edge artificial intelligence algorithms (i.e. head pose estimation and skeleton tracking). Three experimental scenarios with variations in workstation features and interaction modalities were designed to test the performance of our online method against state-of-the-art offline measurements. Results proved that our vision-based cognitive load assessment has the potential to be integrated into the new generation of collaborative robotic technologies. The latter would enable human cognitive state monitoring and robot control strategy adaptation for improving human comfort, ergonomics, and trust in automation.

Research on the Application of Dance Talent Cultivation Mode in Colleges and Universities in the Context of Multimedia Era

Abstract This paper combines Kinect and convolutional neural networks to construct a dance movement recognition technology based on 3D CNNs. Applying dance movement recognition technology to dance teaching builds a new mode of training dance talents in colleges and universities. The role of Kinect in recognizing dance movements is explored from three aspects: real-time motion capture, human skeleton tracking, and information input. The dance movement image is computerized by calculating the depth of the points using stereo analysis. To analyze dance movements and classify them by features, a convolutional neural network is combined. Based on two-dimensional convolutional neural networks, three-dimensional convolutional neural networks have been constructed, which improve the comprehensiveness of dance movement information. By combining dance movement recognition technology with dance talent cultivation, we analyze the students’ professional dance ability and the teaching effect under the new talent cultivation mode. The results show that the teaching effect of the dance talent cultivation mode combined with the movement recognition technology is better, and the percentage of students dance movements reaching the standard in a movement completion is 0.95. The professional ability of dance talent is 0.8 percent.

Skeleton Tracking Solutions for a Low-Cost Stroke Rehabilitation Support System.

Computer systems based on motion assessment are promising solutions to support stroke survivors' autonomous rehabilitation exercises. In this regard, researchers keep trying to achieve engaging and low-cost solutions suitable mainly for home use. Aiming to achieve a system with a minimal technical setup, we compare Microsoft Kinect, OpenPose, and MediaPipe skeleton tracking approaches for upper extremity quality of movement assessment after stroke. We determine if classification models assess accurately exercise performance with OpenPose and MediaPipe data against Kinect, using a dataset of 15 stroke survivors. We compute Root Mean Squared Error to determine the alignment of trajectories and kinematic variables. MediaPipe World Landmarks revealed high alignment with Kinect, revealing to be a potential alternative method.

Lights! Dance! Freeze!

Conventionally, spectators enjoy films passively. This paper describes an interactive film installation that invites participants to use their entire body as a query to search and explore a large corpus of musical films in a novel way. Using an RGB camera, ML-based skeleton tracking technology and a unique pose and film indexing system, this installation tracks a participant's movements and mirrors them in real-time by finding matching poses among hundreds of thousands from well-known musicals. When the participant freezes in a pose, the installation instantly plays back the short film clip that begins with that pose, immersing them in the music and dance from musicals of different eras. This approach explores themes of tangible interfaces and the new possibilities that emerge from employing embodied interaction to traverse the dance pose space, which is traditionally difficult to index and interact with in real time. The pose indexing system and whole-body interaction we propose in this paper open new pathways for cultural participation, as they lend themselves to different datasets and require no technical skills from participants.

Human Pose Estimation with Multi-Camera Localization Using Multi-Objective Optimization Based on Topological Structured Learning

This paper presents a method of pose estimation with camera localization based on a genetic algorithm to derive the joint angle using an inverse kinematics model. In recent years, some open-source libraries for human skeleton tracking that can be implemented on smartphones have been released. Such software can detect joint positions of a skeleton from a 2D camera image; however, the 3D posture cannot be obtained. We therefore propose a method to estimate joint angles by using skeleton data. In the proposed approach, we use a multi-island genetic algorithm to maintain the diversity of the population and design a multi-objective function to improve the robustness. Moreover, structured learning based on topological mapping is implemented in the proposed method to enhance searching efficiency. In an experiment, the proposed method reduced the effect of outliers caused by misdetection. In addition, the structured learning was effective in decreasing the difference between skeleton data and the estimated poses.

A kinematic, imaging and electromyography dataset for human muscular manipulability index prediction

Human Muscular Manipulability is a metric that measures the comfort of an specific pose and it can be used for a variety of applications related to healthcare. For this reason, we introduce KIMHu: a Kinematic, Imaging and electroMyography dataset for Human muscular manipulability index prediction. The dataset is comprised of images, depth maps, skeleton tracking data, electromyography recordings and 3 different Human Muscular Manipulability indexes of 20 participants performing different physical exercises with their arm. The methodology followed to acquire and process the data is also presented for future replication. A specific analysis framework for Human Muscular Manipulability is proposed in order to provide benchmarking tools based on this dataset.

Human-Centered Design for Productivity and Safety in Collaborative Robots Cells: A New Methodological Approach

Nowadays, the current market trend is oriented toward increasing mass customization, meaning that modern production systems have to be able to be flexible but also highly productive. This is due to the fact that we are still living in the so-called Industry 4.0, with its cornerstone of high-productivity systems. However, there is also a migration toward Industry 5.0 that includes the human-centered design of the workplace as one of its principles. This means that the operators have to be put in the center of the design techniques in order to maximize their wellness. Among the wide set of new technologies, collaborative robots (cobots) represent one such technology that modern production systems are trying to integrate, because of their characteristic of working directly with the human operators, allowing for a mix of the flexibility of the manual systems with the productivity of the automated ones. This paper focuses on the impact that these technologies have on different levels within a production plant and on the improvement of the collaborative experience. At the workstation level, the control methodologies are investigated and developed: technologies such as computer vision and augmented reality can be applied to aid and guide the activities of the cobot, in order to obtain the following results. The first is an increase in the overall productivity generated by the reduction of idle times and safety stops and the minimization of the effort required to the operator during the work. This can be achieved through a multiobjective task allocation which aims to simultaneoulsy minimize the makespan, for productivity requirements, and the operator’s energy expenditure and mental workload, for wellness requirements. The second is a safe, human-centered, workspace in which collisions can be avoided in real time. This can be achieved by using real-time multicamera systems and skeleton tracking to constantly know where the operator is in the work cell. The system will offer the possibility of directing feedback based on the discrepancies between the physical world and the virtual models in order to dynamically reallocate the tasks to the resources if the requirements are not satisfied anymore. This allows the application of the technology to sectors that require constant process control, improving also the human–robot interaction: the human operator and the cobot are not merely two single resources working in the same cell, but they can achieve a real human–robot collaboration. In this paper, a framework is preented that allows us to reach the different aforementioned goals.

An Online Framework for Cognitive Load Assessment in Industrial Tasks

The ongoing trend towards Industry 4.0 has revolutionised ordinary workplaces, profoundly changing the role played by humans in the production chain. Research on ergonomics in industrial settings mainly focuses on reducing the operator's physical fatigue and discomfort to improve throughput and avoid safety hazards. However, as the production complexity increases, the cognitive resources demand and mental workload could compromise the operator's performance and the efficiency of the shop floor workplace. State-of-the-art methods in cognitive science work offline and/or involve bulky equipment hardly deployable in industrial settings. This paper presents a novel method for online assessment of cognitive load in manufacturing, primarily assembly, by detecting patterns in human motion directly from the input images of a stereo camera. Head pose estimation and skeleton tracking are exploited to investigate the workers' attention and assess hyperactivity and unforeseen movements. Pilot experiments suggest that our factor assessment tool provides significant insights into workers' mental workload, even confirmed by correlations with physiological and performance measurements. According to data gathered in this study, a vision-based cognitive load assessment has the potential to be integrated into the development of mechatronic systems for improving cognitive ergonomics in manufacturing.

Pattern Recognition of Wushu Routine Action Decomposition Process Based on Kinect

Human action recognition is a hotspot in the fields of computer vision and pattern recognition. Human action recognition technology has created huge social value and considerable economic value for the society. Meeting people’s needs and understanding people’s expressions are the current research focus. Aiming at the problem that the movement cannot be continuously identified and due to a lack of detailed features in the action decomposition pattern recognition in the traditional Wushu routine decomposition process, it is proposed to use Kinect technology to identify the Wushu routine movement decomposition process in the Wushu routine movement decomposition process. This paper analyzes the principle of skeleton tracking and skeleton extraction performed by the Kinect human sensor and uses the Kinect sensor with the Visual Studio 2015 development platform to collect and process the skeleton data of limb movements and defines eight static limb motion samples and four dynamic limbs. The study uses a deep learning neural network algorithm to train and identify the established database of static body movements and uses the same template matching algorithm and K-NN. The recognition effects of the algorithms were compared and analyzed, and it was concluded that the static body motion recognition rates of the three algorithms were all above 90%. In this paper, recognition experiments are carried out on the MSR action 3D database. The influence of different integrated decision-making methods on the recognition results is further discussed and analyzed, and the average method integrated decision-making, which is most suitable for the algorithm model in this paper, is proposed. The results show that the recognition accuracy of the algorithm reaches 98.1%, which proves the feasibility of the preprocessing algorithm.

Planning Collision-Free Robot Motions in a Human–Robot Shared Workspace via Mixed Reality and Sensor-Fusion Skeleton Tracking

The paper describes a method for planning collision-free motions of an industrial manipulator that shares the workspace with human operators during a human–robot collaborative application with strict safety requirements. The proposed workflow exploits the advantages of mixed reality to insert real entities into a virtual scene, wherein the robot control command is computed and validated by simulating robot motions without risks for the human. The proposed motion planner relies on a sensor-fusion algorithm that improves the 3D perception of the humans inside the robot workspace. Such an algorithm merges the estimations of the pose of the human bones reconstructed by means of a pointcloud-based skeleton tracking algorithm with the orientation data acquired from wearable inertial measurement units (IMUs) supposed to be fixed to the human bones. The algorithm provides a final reconstruction of the position and of the orientation of the human bones that can be used to include the human in the virtual simulation of the robotic workcell. A dynamic motion-planning algorithm can be processed within such a mixed-reality environment, allowing the computation of a collision-free joint velocity command for the real robot.

Systematic Motion Integration with Multiple Depth Cameras Allowing Sensor Movement for Stable Skeleton Tracking.

In recent years, markerless motion capture using a depth camera or RGB camera without any restriction on the subject has been attracting attention. Especially, depth cameras such as Kinect and RealSense allow instantaneous motion capture even at home outside lab environment, which is attractive for rehabilitation usage. However, single depth camera can capture steadily skeleton only when the subject stands facing to camera for the limited range, thus it is hard to apply to track skeletons while walking. Multiple depth cameras setting may allow to expand the range, but it can involve non-practical calibration process and can affect instantaneous capture advantage of depth camera. In this study, we propose a systematic method to integrate the motion information of skeletal models obtained from multiple depth cameras. The proposed method can perform a quick calibration using skeletal models instead of external reference objects, and estimate the spatial relationship of the sensors that allows the depth camera to move. The result demonstrates stable skeleton tracking free from occlusion problem keeping instantaneous capture capability of depth cameras.

Dual-channel cascade pose estimation network trained on infrared thermal image and groundtruth annotation for real-time gait measurement.

Real-time spatiotemporal parameter measurement for gait analysis is challenging. Previous techniques for 3D motion analysis, such as inertial measurement units, marker based motion analysis or the use of depth cameras, require expensive equipment, highly skilled staff and limits feasibility for sustainable applications. In this paper a dual-channel cascaded network to perform contactless real-time 3D human pose estimation using a single infrared thermal video as an input is proposed. An algorithm to calculate gait spatiotemporal parameters is presented by tracking estimated joint locations. Additionally, a training dataset composed of infrared thermal images and groundtruth annotations has been developed. The annotation represents a set of 3D joint locations from infrared optical trackers, which is considered to be the gold standard in clinical applications. On the proposed dataset, our pose estimation framework achieved a 3D human pose mean error of below 21mm and outperforms state-of-the-art methods. The results reveal that the proposed system achieves competitive skeleton tracking performance on par with the other motion capture devices and exhibited good agreement with a marker-based three-dimensional motion analysis system (3DMA) over a range of spatiotemporal parameters. Moreover, the process is shown to distinguish differences in over-ground gait parameters of older adults with and without Hemiplegia's disease. We believe that the proposed approaches can measure selected spatiotemporal gait parameters and could be effectively used in clinical or home settings.