Real-time Motion Capture System Research Articles

Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications.

With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user's facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user's face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users' facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user's facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user's eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user's facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.

Super-resolution in brain positron emission tomography using a real-time motion capture system

Super-resolution (SR) is a methodology that seeks to improve image resolution by exploiting the increased spatial sampling information obtained from multiple acquisitions of the same target with accurately known sub-resolution shifts. This work aims to develop and evaluate an SR estimation framework for brain positron emission tomography (PET), taking advantage of a high-resolution infra-red tracking camera to measure shifts precisely and continuously. Moving phantoms and non-human primate (NHP) experiments were performed on a GE Discovery MI PET/CT scanner (GE Healthcare) using an NDI Polaris Vega (Northern Digital Inc), an external optical motion tracking device. To enable SR, a robust temporal and spatial calibration of the two devices was developed as well as a list-mode Ordered Subset Expectation Maximization PET reconstruction algorithm, incorporating the high-resolution tracking data from the Polaris Vega to correct motion for measured line of responses on an event-by-event basis. For both phantoms and NHP studies, the SR reconstruction method yielded PET images with visibly increased spatial resolution compared to standard static acquisitions, allowing improved visualization of small structures. Quantitative analysis in terms of SSIM, CNR and line profiles were conducted and validated our observations. The results demonstrate that SR can be achieved in brain PET by measuring target motion in real-time using a high-resolution infrared tracking camera.

Scenario-Based Sensed Human Motion Editing and Validation Through the Motion-Sphere

Synthesizing realistic human motion data using a real-time motion capture system in a controlled environment is a critical challenge. In addition, effectively manipulating the existing motion data is another primary concern and using such modified data in human motion analysis and activity recognition systems are prone to errors. This paper presents a simplistic and comprehensive system to effortlessly author, edit, and validate human motion data. The system enables a naive user to edit the existing motion data interactively using a humanoid model in a three-dimensional space, based on user-defined scenarios and synthesize numerous variations of the motion sequences. A modular concept of scenario-based sensed unit motion editing has been adopted to demonstrate the proposed system. We employed an efficient analytical kinematic and constraint solver to enforce the inherent body joint limits and external constraints while editing to synthesize complete and meaningful motion sequences. Furthermore, we substantiated the proposed sensed unit motion editing framework through a visual validation study using an open-source intuitive visualization tool called the Motion-Sphere. Finally, we compared the resultant synthesized motion against the real-time motion capture system data to verify the body segments’ orientation and position accuracy deviations.

Sensitivity of cycling Motion Performance Indicators (MPIs) to incremented load and their relationship with performance in professional cyclists

There is a lack of research assessing Motion Performance Indicators (MPIs), which have been recently made commercially available. Therefore, this study explored: (1) the influence of incremented exercise on MPIs and; (2) the relationships between MPIs and cycling performance at different intensities during a graded exercise test (GXT) in professional cyclists. Thirty-six professional cyclists performed GXT until exhaustion with their own bikes attached to a cycle ergometer. MPIs were collected using a real-time motion capture system based on inertial measurement units at 100 Hz of sample rate. Data were extracted from intensities of the GXT when lactate thresholds (LT1, LT2) and peak power (POpeak) were determined. Results showed that only Pelvic Angle ( p < 0.01, d > 1.15) and Pelvic Rotation ( p < 0.01, d > 1.37) were sensitive to increases in exercise intensity (i.e. greater inclination and increased rotation at greater power). Multivariate liner regression analyses showed that a reduced range of movement (ROM) for the upper legs at sub-maximum intensities (LT1 and LT2) was associated with greater power production ( r2 > 0.21), whilst a reduced ROM for the right foot was associated with greater POpeak ( r2 = 0.20). In conclusion, changes in movement patterns were limited to a greater inclination and rotation of the pelvis at maximum power without changes in other MPIs throughout the GXT. Cyclists who produced greater power presented less ROM for their upper legs at LT1 and LT2 whilst at POpeak and greater power production was moderately associated with less ROM for the right foot. Coaches may be able to use MPI to analyze for excess ROM, particularly at higher exercise intensities, as this seems to increase inefficiencies and limit power production.

Strength Training Characteristics of Different Loads Based on Acceleration Sensor and Finite Element Simulation.

Deep squat, bench press and hard pull are important ways for people to improve their strength. The use of sensors to measure force is rare. Measuring strength with sensors is extremely valuable for people to master the intensity of exercise to scientifically effective exercise. To this end, in this paper, we used a real-time wireless motion capture and mechanical evaluation system of the wearable sensor to measure the dynamic characteristics of 30 young men performing deep squat, bench press and hard pull maneuvers. The data of tibia were simulated with AnyBody 5.2 and ANSYS 19.2 to verify the authenticity. The result demonstrated that the appropriate force of the deep squat elbow joint, the hip joint and the knee joint is 40% 1RM, the appropriate force of the bench press is 40% 1RM and the appropriate force of the hard pull is 80% 1RM. The external force is the main factor of bone change. The mechanical characteristics of knee joint can be simulated after the Finite Element Analysis and the simulation of AnyBody model are verified.

Characteristics of Aerobic Resistance Strength Training With Different Load Intensity Based on Acceleration Sensor

Objective: To compare the changes of kinematics and dynamics indexes of deep squat, bench press and hard pull in order to provide basis for the application of these three kinds of resistance strength training methods. Methods: The kinematic and dynamic characteristics of 30 young men in deep squat, bench press and hard pull were measured by the real-time wireless sensor motion capture and mechanical evaluation system of the wearable FAB. Results: In terms of kinematic parameters under 40%1RM and 80%1RM load, there were significant differences(P < 0.05) in flexion angle of deep squat hip joint, knee joint, hard pull hip joint and knee joint. There were significant differences in the angular velocity of trunk forward flexion and trunk extension, shoulder extension angle velocity of horizontal push, and flexion angular velocity of hard pull knee joint (P < 0.05), and significant differences were found in hip joint flexion angle acceleration, elbow extension angular acceleration in horizontal push, elbow flexion and knee joint flexion angle acceleration (P < 0.05). Conclusion: Hip joint and knee joint are the most important in deep squat and hard pull angle, angular velocity and angular acceleration, elbow joint is the most important one in bench press angle, angular velocity and angular acceleration.

Real-Time Multi-person Motion Capture from Multi-view Video and IMUs

A real-time motion capture system is presented which uses input from multiple standard video cameras and inertial measurement units (IMUs). The system is able to track multiple people simultaneously and requires no optical markers, specialized infra-red cameras or foreground/background segmentation, making it applicable to general indoor and outdoor scenarios with dynamic backgrounds and lighting. To overcome limitations of prior video or IMU-only approaches, we propose to use flexible combinations of multiple-view, calibrated video and IMU input along with a pose prior in an online optimization-based framework, which allows the full 6-DoF motion to be recovered including axial rotation of limbs and drift-free global position. A method for sorting and assigning raw input 2D keypoint detections into corresponding subjects is presented which facilitates multi-person tracking and rejection of any bystanders in the scene. The approach is evaluated on data from several indoor and outdoor capture environments with one or more subjects and the trade-off between input sparsity and tracking performance is discussed. State-of-the-art pose estimation performance is obtained on the Total Capture (mutli-view video and IMU) and Human 3.6M (multi-view video) datasets. Finally, a live demonstrator for the approach is presented showing real-time capture, solving and character animation using a light-weight, commodity hardware setup.

Real-time 3D pose estimation by RGB-D camera for prediction of human motion

As the opportunity of interactions between human and robot increases, real-time human 3D pose estimation technique is required for robot to act adaptively to human motion. A marker-less motion capturing system, such as Kinect, sometimes miss and incorrectly estimates 3D pose when the joint-points are in a specific pose, for instance, with crossed arm. This paper describes a method of real-time 3D pose estimation using a RGB-D camera and an image-based pose estimation technique. This can be an easy-to-use real-time motion capture system. We will use this system to obtain human 3D motion data in everyday environment for predicting human motion in real time.

Development of a novel 6-DOF parallel mechanism moving platform for investigating the effects of multimodal sensory feedback information regarding postural stability in humans

The purpose of this study was to develop a six-degree-of-freedom parallel mechanism moving platform to investigate the effects of multimodal sensory feedback information while standing upright. We constructed a custom-designed disturbance-applying instrument (DAI) consisting of a support surface suspended from eight pneumatic artificial muscles. The posture of the support surface was controlled with a proportional–integral–derivative (PID) feedback system using an infra-red camera-based real-time 3D motion capture system, and was estimated by step and frequency responses. Head trajectories of two healthy subjects were recorded to evaluate the effect of the differences in the impedance of the support surface during upright standing with their eyes opened/closed. The results demonstrated that the step and frequency responses of the DAI were good enough to enhance the bodily oscillation and head motion relative to the support surface. Indeed, the head-swaying space, using the DAI in a 0.2-Hz air-supplying condition during upright standing with their eyes closed, was larger than in the other standing condition. These results suggest that the specific medio-lateral head swaying was caused by the effects of the multimodal sensory feedback information using the DAI. In conclusion, we developed a novel moving platform to investigate the effects of multimodal sensory feedback information upon upright postural control. The DAI would be particularly valuable to enhance the head-swaying space during upright standing by changing the impedance of the support surface or stimulating the translational and/or rotational sensory feedback integration.

Development of a Self-Calibrated Motion Capture System by Nonlinear Trilateration of Multiple Kinects v2

In this paper, a Kinect-based distributed and real-time motion capture system is developed. A trigonometric method is applied to calculate the relative position of Kinect v2 sensors with a calibration wand and register the sensors' positions automatically. By combining results from multiple sensors with a nonlinear least square method, the accuracy of the motion capture is optimized. Moreover, to exclude inaccurate results from sensors, a computational geometry is applied in the occlusion approach, which discovers occluded joint data. The synchronization approach is based on an NTP protocol that synchronizes the time between the clocks of a server and clients dynamically, ensuring that the proposed system is a real-time system. Experiments for validating the proposed system are conducted from the perspective of calibration, occlusion, accuracy, and efficiency. Furthermore, to demonstrate the practical performance of our system, a comparison of previously developed motion capture systems (the linear trilateration approach and the geometric trilateration approach) with the benchmark OptiTrack system is conducted, therein showing that the accuracy of our proposed system is $38.3\%$ and 24.1% better than the two aforementioned trilateration systems, respectively.

Samba: a real-time motion capture system using wireless camera sensor networks.

There is a growing interest in 3D content following the recent developments in 3D movies, 3D TVs and 3D smartphones. However, 3D content creation is still dominated by professionals, due to the high cost of 3D motion capture instruments. The availability of a low-cost motion capture system will promote 3D content generation by general users and accelerate the growth of the 3D market. In this paper, we describe the design and implementation of a real-time motion capture system based on a portable low-cost wireless camera sensor network. The proposed system performs motion capture based on the data-driven 3D human pose reconstruction method to reduce the computation time and to improve the 3D reconstruction accuracy. The system can reconstruct accurate 3D full-body poses at 16 frames per second using only eight markers on the subject's body. The performance of the motion capture system is evaluated extensively in experiments.

Experimental Comparison of Sensor Fusion Algorithms for Attitude Estimation

Inertial Measurement Unit is commonly used in various applications especially as a low-cost system for localization and attitude estimation. Some applications are: real-time motion capture system, gait analysis for rehabilitation purposes, biomedical applications, advanced robotic applications such as mobile robot localization and Unmanned Aerial Vehicles (UAV) attitude estimation. In all the mentioned applications the accuracy and the fast response are the most important requirements, thus the research is focused on the design and the implementation of highly accurate hardware systems and fast sensor data fusion algorithms, named Attitude and Heading Reference System (AHRS), aimed at estimating the orientation of a rigid body with respect to a reference frame. A large number of different solutions can be found in the literature, and an experimental comparison of the most popular is presented in this work. In particular, the algorithm based on the gradient descent method and the algorithm based on a nonlinear complementary filter are compared to a standard Extended Kalman Filter (EKF) with the aim to show that a general method can easily compete with ad-hoc solutions and even outperform them in particular conditions. In order to validate the estimation accuracy a Kuka robot is used to compute the ground truth. Moreover, in order to estimate the computational burden, the algorithms are implemented on an ARM-Cortex M4-based evaluation board.

Use of a real-time three-dimensional motion tracking system for measurement of intrafractional motion of the thoracic wall in dogs

To measure respiratory motion of the thoracic wall region in dogs using a real-time motion tracking system and compare the amount of respiratory motion between dogs positioned with and without a vacuum-formable cushion. 8 healthy adult mixed-breed dogs (median weight, 23 kg). Dogs were anesthetized and positioned in sternal and dorsal recumbency with and without a vacuum-formable cushion. Three-dimensional movement of anatomic landmarks was measured with a real-time motion capture system that tracked the locations of infrared light-emitting diodes attached externally to the dorsal or ventral and lateral aspects of the thoracic wall. Dogs positioned in sternal recumbency had significantly less cranial-to-caudal and left-to-right respiratory motion at the lateral aspect of the thoracic wall, compared with dogs positioned in dorsal recumbency, whether or not a cushion was used. For dogs treated in sternal recumbency, use of a cushion significantly increased the peak displacement vector (overall movement in 3-D space) for 3 of 4 marker locations on the dorsal thoracic wall. As respiratory rate increased, respiratory motion at the lateral and ventral aspects of the thoracic wall decreased when data for all dogs in dorsal recumbency were evaluated together. Associations between respiratory rate and respiratory motion suggested that the use of rapid, shallow ventilation may be beneficial for dogs undergoing highly conformal radiation treatment. These results provide a basis for further research on respiratory motion in anesthetized dogs.

Real-Time Animation Software for Customized Training to Use Motor Prosthetic Systems

Research on control of human movement and development of tools for restoration and rehabilitation of movement after spinal cord injury and amputation can benefit greatly from software tools for creating precisely timed animation sequences of human movement. Despite their ability to create sophisticated animation and high quality rendering, existing animation software are not adapted for application to neural prostheses and rehabilitation of human movement. We have developed a software tool known as MSMS (MusculoSkeletal Modeling Software) that can be used to develop models of human or prosthetic limbs and the objects with which they interact and to animate their movement using motion data from a variety of offline and online sources. The motion data can be read from a motion file containing synthesized motion data or recordings from a motion capture system. Alternatively, motion data can be streamed online from a real-time motion capture system, a physics-based simulation program, or any program that can produce real-time motion data. Further, animation sequences of daily life activities can be constructed using the intuitive user interface of Microsoft's PowerPoint software. The latter allows expert and nonexpert users alike to assemble primitive movements into a complex motion sequence with precise timing by simply arranging the order of the slides and editing their properties in PowerPoint. The resulting motion sequence can be played back in an open-loop manner for demonstration and training or in closed-loop virtual reality environments where the timing and speed of animation depends on user inputs. These versatile animation utilities can be used in any application that requires precisely timed animations but they are particularly suited for research and rehabilitation of movement disorders. MSMS's modeling and animation tools are routinely used in a number of research laboratories around the country to study the control of movement and to develop and test neural prostheses for patients with paralysis or amputations.

A Novel Hierarchical Information Fusion Method for Three-Dimensional Upper Limb Motion Estimation

Motion capture provides the best interface to the understanding of trajectory planning and formations in the central nervous system, which has enormously benefited the research in interactive game and learning, animation, film special effects, health care, and navigation. The existing human motion capture techniques, which use structured multiple high-resolution cameras in a dedicated studio, are complicated and expensive. With the rapid development of microsensors on chip, a ubiquitous real-time low-cost human motion capture system that uses wearable microsensors becomes possible. Because of the agility in movement, upper limb motion estimation has been regarded as the most difficult problem in human motion capture. In this paper, we take the upper limb as our research subject and present a novel upper limb motion estimation algorithm by hierarchical fusion of sensor data and human skeleton constraints. Our method represents the orientations of upper limb segments in quaternion, which is computationally effective and can avoid the singularity problem. To address the nonlinear human body segment motion, a particle filter is proposed to fuse inertial and magnetic sensor data. To compensate for the drift, which is the most challenging issue in motion estimation using inertial sensors, we present a novel solution by modeling the geometrical constraint in elbow joint and fuse the constraint in the framework of particle filter to revise the sensor fusion results and improve the estimation accuracy. The experimental results have shown that the proposed algorithm can provide accurate results compared to the ground truth.

2A1-L02 リアルタイムモーションキャプチャを用いたヒューマノイドのオプティカルコックピット

本研究では, 光学式リアルタイムモーションキャプチャを人力デバイスとして, ヒューマノイドを操作する概念である, オプティカルコックピットを実現する。モーションキャプチャにより得られた人間の運動を, 異なる自由度やリンク長を持つモデルが仮想的にマーカを体につけてモーションキャプチャした場合の運動に変換し, 関節角計算を行う。実例として, 富士通ヒューマノイドHOAP1を操作した実験結果について報告する。