6-DOF Head Research Articles

Real-time 6DoF full-range markerless head pose estimation

Head pose estimation is a fundamental function for several applications in human–computer interactions. Accurate six degrees of freedom head pose estimation (6DoF-HPE) with full-range angles make up most of these applications, which require sequential images of the human head as input. Most existing head pose estimation methods focus on a three degrees of freedom (3DoF) frontal head, which restricts their applications in real-world scenarios. This study presents a framework designed to estimate a head pose without landmark localization. The novelty of our framework is to estimate the 6DoF head poses under full-range angles in real-time. The proposed framework leverages deep neural networks to detect human heads and predict their angles using single shot multibox detector (SSD) and RepVGG-b1g4 backbone, respectively. This work uses red, green, blue, and depth (RGB-D) data to estimate the rotational and translational components relative to the camera pose. The proposed framework employs a continuous representation to predict the angles and a multi-loss approach to update the loss functions for the training strategy. The regression function combines the geodesic loss with the mean squared error. The ground-truth labels were extracted from the public dataset Carnegie Mellon university (CMU) Panoptic for full head angles. This study provides a comprehensive comparison with state-of-the-art methods using public benchmark datasets. Experiments demonstrate that the proposed method achieves or outperforms state-of-the-art methods. The code and datasets are available at: (https://github.com/Redhwan-A/6DoFHPE).

HandyGaze: A Gaze Tracking Technique for Room-Scale Environments using a Single Smartphone

We propose HandyGaze, a 6-DoF gaze tracking technique for room-scale environments that can be carried out by simply holding a smartphone naturally without installing any sensors or markers in the environment. Our technique simultaneously employs the smartphone’s front and rear cameras: The front camera estimates the user’s gaze vector relative to the smartphone, while the rear camera (and depth sensor, if available) performs self-localization by reconstructing a pre-obtained 3D map of the environment. To achieve this, we implemented a prototype that works on iOS smartphones by running an ARKit-based algorithm for estimating the user’s 6-DoF head orientation. We additionally implemented a novel calibration method that offsets the user-specific deviation between the head and gaze orientations. We then conducted a user study (N=10) that measured our technique’s positional accuracy to the gaze target under four conditions, based on combinations of use with and without a depth sensor and calibration. The results show that our calibration method was able to reduce the mean absolute error of the gaze point by 27%, with an error of 0.53 m when using the depth sensor. We also report the target size required to avoid erroneous inputs. Finally, we suggest possible applications such as a gaze-based guidance application for museums.

Investigation of role of falx and tentorium on brain simulant strain under impact loading

Mild Traumatic brain injury (mTBI) is a major health concern. The role of the falx and tentorium (i.e., membranes) in exacerbating mTBI has been conjectured due to the involvement of clinically confirmed midbrain regions. Recent brain biomechanics investigations, mainly using computational head models, also support such a hypothesis. However, data in this regard is limited. Towards this end, using a surrogate head model, we investigate the role of membranes on brain biomechanics. Two different materials—thermoplastic polyurethane with various elastic moduli values (20, 150, 205 MPa) and polylactic acid (elastic modulus 1500 MPa) were used to examine the effect of membrane stiffness on brain simulant strain. The head surrogate was mounted on the Hybrid-III neck and subjected to coronal and sagittal plane rotations using a linear impactor system. Corresponding 6-DOF head kinematics and 2D brain simulant strains in midcoronal and midsagittal planes were measured. Our results elucidate the paradigm of strain evolution in the brain simulant in the presence of membranes. The cortical strains are decreased, whereas strains in the subcortical regions are either equivalent or increased in the presence of membranes. The elastic modulus of the membranes governs the amount of strain reduction or increase. We found that the falx displacement and constraints on stress wave propagation are dominant mechanisms dictating the mechanics of the interaction of membranes with the brain simulant. Overall, these results provide novel experimental insights into the role of membranes on brain deformations, which will motivate futuristic investigations in numerous subdomains of brain injury biomechanics.

Locomotion in Place in Virtual Reality: A Comparative Evaluation of Joystick, Teleport, and Leaning.

Recent VR head-mounted displays for consumers feature 3-DOF or 6-DOF head tracking. However, position tracking (when available) is limited to a small area. Moreover, in small or cluttered physical spaces, users can safely experience VR only by staying in place, standing or seated. Different locomotion techniques have been proposed to allow users to explore virtual environments by staying in place. Two in-place locomotion techniques, frequently employed in the literature and in consumer applications, are based on joystick and teleport. Some authors explored leaning with the aim of proposing a more natural in-place locomotion technique. However, more research is needed to understand the effects of the three techniques, since no user study thoroughly compared them all together on a variety of fundamental aspects. Therefore, this paper presents a comparative evaluation with 75 users, assessing the effects of the three techniques on performance, sickness, presence, usability, and different aspects of comfort. Performance of teleport was better than the other techniques, and performance of leaning was better than joystick. Teleport also caused less nausea than the other techniques. Unexpectedly, no significant differences were found for presence. Teleport received a higher usability score than the other techniques. Finally, the techniques had different effects on comfort that we discuss in detail.

An integrated 6DoF video camera and system design

Designing a fully integrated 360° video camera supporting 6DoF head motion parallax requires overcoming many technical hurdles, including camera placement, optical design, sensor resolution, system calibration, real-time video capture, depth reconstruction, and real-time novel view synthesis. While there is a large body of work describing various system components, such as multi-view depth estimation, our paper is the first to describe a complete, reproducible system that considers the challenges arising when designing, building, and deploying a full end-to-end 6DoF video camera and playback environment. Our system includes a computational imaging software pipeline supporting online markerless calibration, high-quality reconstruction, and real-time streaming and rendering. Most of our exposition is based on a professional 16-camera configuration, which will be commercially available to film producers. However, our software pipeline is generic and can handle a variety of camera geometries and configurations. The entire calibration and reconstruction software pipeline along with example datasets is open sourced to encourage follow-up research in high-quality 6DoF video reconstruction and rendering 1 .

Communication capability of telepresence system with the miniature humanoid MH-2

Recently, several different types of telepresence robots have been developed. Their ability to communicate with a person in a remote location has been analyzed to achieve rich communication, including the presence of an operator. These studies focused on human-sized robots, small-sized non-humanoid robots, or small-sized humanoid robots without human-like proportions. A small-sized humanoid robot with human-like proportions has not been studied because of the absence of such a small humanoid. It should be noted that human-like proportions play a very important role in enhancing the presence of an operator through an avatar. In our previous work, a small-sized humanoid robot named MH-2 was developed for wearable telepresence system. The MH-2 consists of seven degrees-of-freedom (7-DOF) arms, a 3-DOF head, and a torso scaled according to human-like proportions. Wire–pulley mechanisms were employed to achieve a compact design to adopt a requirement for wearability. In this work, a telepresence system is introduced for the evaluation of the communication ability of the MH-2. In the experiment, Skype, using a flat display, was compared with the MH-2. The “Indian Poker” game was employed for the experiment. It is an ideal evaluation platform because it is a psychological game that requires careful observations among the players. The evaluations were conducted in terms of six aspects: emotion or personality, line of sight, familiarity, presence, enjoyment, and smoothness of the game progress. The experimental results showed that the MH-2 performed positively in two out of six aspects, namely the line of sight and smoothness of the game progress. On the other hand, facial expressions play important role to display individual presence. From a comprehensive standpoint, the MH-2 demonstrated good capabilities for rich remote communications.

Towards frameless maskless SRS through real-time 6DoF robotic motion compensation

Stereotactic radiosurgery (SRS) uses precise dose placement to treat conditions of the CNS. Frame-based SRS uses a metal head ring fixed to the patient’s skull to provide high treatment accuracy, but patient comfort and clinical workflow may suffer. Frameless SRS, while potentially more convenient, may increase uncertainty of treatment accuracy and be physiologically confining to some patients. By incorporating highly precise robotics and advanced software algorithms into frameless treatments, we present a novel frameless and maskless SRS system where a robot provides real-time 6DoF head motion stabilization allowing positional accuracies to match or exceed those of traditional frame-based SRS. A 6DoF parallel kinematics robot was developed and integrated with a real-time infrared camera in a closed loop configuration. A novel compensation algorithm was developed based on an iterative closest-path correction approach. The robotic SRS system was tested on six volunteers, whose motion was monitored and compensated for in real-time over 15 min simulated treatments. The system’s effectiveness in maintaining the target’s 6DoF position within preset thresholds was determined by comparing volunteer head motion with and without compensation. Comparing corrected and uncorrected motion, the 6DoF robotic system showed an overall improvement factor of 21 in terms of maintaining target position within 0.5 mm and 0.5 degree thresholds. Although the system’s effectiveness varied among the volunteers examined, for all volunteers tested the target position remained within the preset tolerances 99.0% of the time when robotic stabilization was used, compared to 4.7% without robotic stabilization. The pre-clinical robotic SRS compensation system was found to be effective at responding to sub-millimeter and sub-degree cranial motions for all volunteers examined. The system’s success with volunteers has demonstrated its capability for implementation with frameless and maskless SRS treatments, potentially able to achieve the same or better treatment accuracies compared to traditional frame-based approaches.

Towards Kilo-Hertz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras.

To maintain a reliable registration of the virtual world with the real world, augmented reality (AR) applications require highly accurate, low-latency tracking of the device. In this paper, we propose a novel method for performing this fast 6-DOF head pose tracking using a cluster of rolling shutter cameras. The key idea is that a rolling shutter camera works by capturing the rows of an image in rapid succession, essentially acting as a high-frequency 1D image sensor. By integrating multiple rolling shutter cameras on the AR device, our tracker is able to perform 6-DOF markerless tracking in a static indoor environment with minimal latency. Compared to state-of-the-art tracking systems, this tracking approach performs at significantly higher frequency, and it works in generalized environments. To demonstrate the feasibility of our system, we present thorough evaluations on synthetically generated data with tracking frequencies reaching 56.7 kHz. We further validate the method's accuracy on real-world images collected from a prototype of our tracking system against ground truth data using standard commodity GoPro cameras capturing at 120 Hz frame rate.

ミニチュアヒューマノイド用3自由度ワイヤ駆動式ジンバル機構の設計

In our previous work, a small wearable robot named telecommunicator has been developed. The telecommunicator can achieve rich communications with a remote person. Until now, we developed a prototype of the wearable miniature humanoid MH-2 as telecommunicator. It has a 3-DOF head, a 3-DOF body and 7-DOF dual arms. MH-2 also has human like shape and ability to reproduce the movements of human's arm and head. By using a real avatar, we are aiming to achieve a high fidelity communication system. However, the current head mechanism reduce reality because of its inaccuracy. In this paper, a new 3-DOF head mechanism is introduced and designed.

Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots

SUMMARYThis paper describes and evaluates the use of a head-mounted display (HMD) for the teleoperation of a field robot. The HMD presents a pair of video streams to the operator (one to each eye) originating from a pair of stereo cameras located on the front of the robot, thus providing him/her with a sense of depth (stereopsis). A tracker on the HMD captures 3-DOF head orientation data which is then used for adjusting the camera orientation by moving the robot and/or the camera position accordingly, and rotating the displayed images to compensate for the operator's head rotation. This approach was implemented in a search and rescue robot (RAPOSA), and it was empirically validated in a series of short user studies. This evaluation involved four experiments covering two-dimensional perception, depth perception, scene perception, and performing a search and rescue task in a controlled scenario. The stereoscopic display and head tracking are shown to afford a number of performance benefits. However, one experiment also revealed that controlling robot orientation with yaw input from the head tracker negatively influenced task completion time. A possible explanation is a mismatch between the abilities of the robot and the human operator. This aside, the studies indicated that the use of an HMD to create a stereoscopic visualization of the camera feeds from a mobile robot enhanced the perception of cues in a static three-dimensional environment and also that such benefits transferred to simulated field scenarios in the form of enhanced task completion times.

Prospective real-time head motion correction using inductively coupled wireless NMR probes

Head motion continues to be a major source of artifacts and data quality degradation in MRI. The goal of this work was to develop and demonstrate a novel technique for prospective, 6 degrees of freedom (6DOF) rigid body motion estimation and real-time motion correction using inductively coupled wireless nuclear magnetic resonance (NMR) probe markers. Three wireless probes that are inductively coupled with the scanner's RF setup serve as fiducials on the subject's head. A 12-ms linear navigator module is interleaved with the imaging sequence for head position estimation, and scan geometry is updated in real time for motion compensation. Flip angle amplification in the markers allows the use of extremely small navigator flip angles (∼1°). A novel algorithm is presented to identify marker positions in the absence of marker specific receive channels. Motion correction is demonstrated in high resolution 2D and 3D gradient recalled echo experiments in a phantom and humans. Significant improvement of image quality is demonstrated in phantoms and human volunteers under different motion conditions. A novel real-time 6DOF head motion correction technique based on wireless NMR probes is demonstrated in high resolution imaging at 7 Tesla.

On-Board Imaging Validation of Optically Guided Stereotactic Radiosurgery Positioning System for Conventionally Fractionated Radiotherapy for Paranasal Sinus and Skull Base Cancer

To evaluate the positioning accuracy of an optical positioning system for stereotactic radiosurgery in a pilot experience of optically guided, conventionally fractionated, radiotherapy for paranasal sinus and skull base tumors. Before each daily radiotherapy session, the positioning of 28 patients was set up using an optical positioning system. After this initial setup, the patients underwent standard on-board imaging that included daily orthogonal kilovoltage images and weekly cone beam computed tomography scans. Daily translational shifts were made after comparing the on-board images with the treatment planning computed tomography scans. These daily translational shifts represented the daily positional error in the optical tracking system and were recorded during the treatment course. For 13 patients treated with smaller fields, a three-degree of freedom (3DOF) head positioner was used for more accurate setup. The mean positional error for the optically guided system in patients with and without the 3DOF head positioner was 1.4 ± 1.1 mm and 3.9 ± 1.6 mm, respectively (p <.0001). The mean positional error drifted 0.11 mm/wk upward during the treatment course for patients using the 3DOF head positioner (p = .057). No positional drift was observed in the patients without the 3DOF head positioner. Our initial clinical experience with optically guided head-and-neck fractionated radiotherapy was promising and demonstrated clinical feasibility. The optically guided setup was especially useful when used in conjunction with the 3DOF head positioner and when it was recalibrated to the shifts using the weekly portal images.

2A1-B21 ウェアラブルミニチュアヒューマノイドロボットの開発(ウェアラブルロボティクス)

In our previous work, a personal telerobot system named "telecommunicator" has been introduced. The telecommunicator achieves communication with a remote person through a teleoperated robot. Until now, three prototypes of the wearable telecommunicator, T1, T2 and T3, have been developed as research platforms. However, these robots cannot act as an operator with realistic sensations, because of their typical robot appearance. In this paper, to tackle this problem, a wearable miniature humanoid robot is introduced as the telecommunicator. The robot includes 4-dof dual arms and a 3-dof head with a stereo camera system. Its concept, design and a developed prototype model are described.