Head Pose Tracking Research Articles

Toward Human-Centered Automated Driving: A Novel Spatiotemporal Vision Transformer-Enabled Head Tracker

Accurate dynamic driver head pose tracking is of great importance for driver–automotive collaboration, intelligent copilot, head-up display (HUD), and other human-centered automated driving applications. To further advance this technology, this article proposes a low-cost and markerless head-tracking system using a deep learning-based dynamic head pose estimation model. The proposed system requires only a red, green, blue (RGB) camera without other hardware or markers. To enhance the accuracy of the driver’s head pose estimation, a spatiotemporal vision transformer (ST-ViT) model, which takes an image pair as the input instead of a single frame, is proposed. Compared to a standard transformer, the ST-ViT contains a spatial–convolutional vision transformer and a temporal transformer, which can improve the model performance. To handle the error fluctuation of the head pose estimation model, this article proposes an adaptive Kalman filter (AKF). By analyzing the error distribution of the estimation model and the user experience of the head tracker, the proposed AKF includes an adaptive observation noise coefficient; this can adaptively moderate the smoothness of the curve. Comprehensive experiments show that the proposed system is feasible and effective, and it achieves a state-of-the-art performance.

Variational Inference and Learning of Piecewise Linear Dynamical Systems.

Modeling the temporal behavior of data is of primordial importance in many scientific and engineering fields. Baseline methods assume that both the dynamic and observation equations follow linear-Gaussian models. However, there are many real-world processes that cannot be characterized by a single linear behavior. Alternatively, it is possible to consider a piecewise linear model which, combined with a switching mechanism, is well suited when several modes of behavior are needed. Nevertheless, switching dynamical systems are intractable because their computational complexity increases exponentially with time. In this article, we propose a variational approximation of piecewise linear dynamical systems. We provide full details of the derivation of two variational expectation-maximization algorithms: a filter and a smoother. We show that the model parameters can be split into two sets: static and dynamic parameters, and that the former parameters can be estimated offline together with the number of linear modes, or the number of states of the switching variable. We apply the proposed method to the head-pose tracking, and we thoroughly compare our algorithms with several state of the art trackers.

Auritus: An Open-Source Optimization Toolkit for Training and Development of Human Movement Models and Filters Using Earables.

Smart ear-worn devices (called earables) are being equipped with various onboard sensors and algorithms, transforming earphones from simple audio transducers to multi-modal interfaces making rich inferences about human motion and vital signals. However, developing sensory applications using earables is currently quite cumbersome with several barriers in the way. First, time-series data from earable sensors incorporate information about physical phenomena in complex settings, requiring machine-learning (ML) models learned from large-scale labeled data. This is challenging in the context of earables because large-scale open-source datasets are missing. Secondly, the small size and compute constraints of earable devices make on-device integration of many existing algorithms for tasks such as human activity and head-pose estimation difficult. To address these challenges, we introduce Auritus an extendable and open-source optimization toolkit designed to enhance and replicate earable applications. Auritus serves two primary functions. Firstly, Auritus handles data collection, pre-processing, and labeling tasks for creating customized earable datasets using graphical tools. The system includes an open-source dataset with 2.43 million inertial samples related to head and full-body movements, consisting of 34 head poses and 9 activities from 45 volunteers. Secondly, Auritus provides a tightly-integrated hardware-in-the-loop (HIL) optimizer and TinyML interface to develop lightweight and real-time machine-learning (ML) models for activity detection and filters for head-pose tracking. To validate the utlity of Auritus, we showcase three sample applications, namely fall detection, spatial audio rendering, and augmented reality (AR) interfacing. Auritus recognizes activities with 91% leave 1-out test accuracy (98% test accuracy) using real-time models as small as 6-13 kB. Our models are 98-740× smaller and 3-6% more accurate over the state-of-the-art. We also estimate head pose with absolute errors as low as 5 degrees using 20kB filters, achieving up to 1.6× precision improvement over existing techniques. We make the entire system open-source so that researchers and developers can contribute to any layer of the system or rapidly prototype their applications using our dataset and algorithms.

High-Accuracy 3D Gaze Estimation with Efficient Recalibration for Head-Mounted Gaze Tracking Systems.

The problem of 3D gaze estimation can be viewed as inferring the visual axes from eye images. It remains a challenge especially for the head-mounted gaze tracker (HMGT) with a simple camera setup due to the complexity of the human visual system. Although the mainstream regression-based methods could establish the mapping relationship between eye image features and the gaze point to calculate the visual axes, it may lead to inadequate fitting performance and appreciable extrapolation errors. Moreover, regression-based methods suffer from a degraded user experience because of the increased burden in recalibration procedures when slippage occurs between HMGT and head. To address these issues, a high-accuracy 3D gaze estimation method along with an efficient recalibration approach is proposed with head pose tracking in this paper. The two key parameters, eyeball center and camera optical center, are estimated in head frame with geometry-based method, so that a mapping relationship between two direction features is proposed to calculate the direction of the visual axis. As the direction features are formulated with the accurately estimated parameters, the complexity of mapping relationship could be reduced and a better fitting performance can be achieved. To prevent the noticeable extrapolation errors, direction features with uniform angular intervals for fitting the mapping are retrieved over human’s field of view. Additionally, an efficient single-point recalibration method is proposed with an updated eyeball coordinate system, which reduces the burden of calibration procedures significantly. Our experiment results show that the calibration and recalibration methods could improve the gaze estimation accuracy by 35 percent (from a mean error of 2.00 degrees to 1.31 degrees) and 30 percent (from a mean error of 2.00 degrees to 1.41 degrees), respectively, compared with the state-of-the-art methods.

Driver's Head Pose and Gaze Zone Estimation Based on Multi-Zone Templates Registration and Multi-Frame Point Cloud Fusion.

Head pose and eye gaze are vital clues for analysing a driver’s visual attention. Previous approaches achieve promising results from point clouds in constrained conditions. However, these approaches face challenges in the complex naturalistic driving scene. One of the challenges is that the collected point cloud data under non-uniform illumination and large head rotation is prone to partial facial occlusion. It causes bad transformation during failed template matching or incorrect feature extraction. In this paper, a novel estimation method is proposed for predicting accurate driver head pose and gaze zone using an RGB-D camera, with an effective point cloud fusion and registration strategy. In the fusion step, to reduce bad transformation, continuous multi-frame point clouds are registered and fused to generate a stable point cloud. In the registration step, to reduce reliance on template registration, multiple point clouds in the nearest neighbor gaze zone are utilized as a template point cloud. A coarse transformation computed by the normal distributions transform is used as the initial transformation, and updated with particle filter. A gaze zone estimator is trained by combining the head pose and eye image features, in which the head pose is predicted by point cloud registration, and the eye image features are extracted via multi-scale spare coding. Extensive experiments demonstrate that the proposed strategy achieves better results on head pose tracking, and also has a low error on gaze zone classification.

A new head pose tracking method based on stereo visual SLAM

Real-time and reliable head pose tracking is the basis of human–computer interaction and face analysis applications. Aiming at the problems of accuracy and real time performance in current tracking method, a new head pose tracking method based on stereo visual SLAM is proposed in this paper. The sparse head map is constructed based on ORB feature points extraction and stereo matching, then the 3D-2D matching relations between 3D mappoints and 2D feature points are obtained by projection matching. Finally, the camera pose solved by the Bundle Adjustment is converted to head pose, which realizes the tracking of head pose. The experimental results show that this method can obtain high precise head pose. The mean errors of three Euler angles are all less than 1°. Therefore, the proposed head pose tracking method can track and estimate precise head pose in real time under smooth background.

Atypical Head Movement during Face-to-Face Interaction in Children with Autism Spectrum Disorder.

The present study implemented an objective head pose tracking technique-OpenFace 2.0 to quantify the three dimensional head movement. Children with autism spectrum disorder (ASD) and typical development (TD) were engaged in a structured conversation with an interlocutress while wearing an eye tracker. We computed the head movement stereotypy with multiscale entropy analysis. In addition, the head rotation range (RR) and the amount of rotation per minute (ARPM) were calculated to quantify the extent of head movement. Results demonstrated that the ASD group had significantly higher level of movement stereotypy, RR and ARPM in all the three directions of head movement. Further analyses revealed that the extent of head movement could be significantly explained by movement stereotypy, but not by the amount of visual fixation to the interlocutress. These results demonstrated the atypical head movement dynamics in children with ASD during live interaction. It is proposed that head movement might potentially provide novel objective biomarkers of ASD. LAY SUMMARY: Our study used an objective tool to quantify head movement in children with autism. Results showed that children with autism had more stereotyped and greater head movement. We suggest that head movement tracking technique be widely used in autism research.

Continuous Driver's Gaze Zone Estimation Using RGB-D Camera.

The driver gaze zone is an indicator of a driver’s attention and plays an important role in the driver’s activity monitoring. Due to the bad initialization of point-cloud transformation, gaze zone systems using RGB-D cameras and ICP (Iterative Closet Points) algorithm do not work well under long-time head motion. In this work, a solution for a continuous driver gaze zone estimation system in real-world driving situations is proposed, combining multi-zone ICP-based head pose tracking and appearance-based gaze estimation. To initiate and update the coarse transformation of ICP, a particle filter with auxiliary sampling is employed for head state tracking, which accelerates the iterative convergence of ICP. Multiple templates for different gaze zone are applied to balance the templates revision of ICP under large head movement. For the RGB information, an appearance-based gaze estimation method with two-stage neighbor selection is utilized, which treats the gaze prediction as the combination of neighbor query (in head pose and eye image feature space) and linear regression (between eye image feature space and gaze angle space). The experimental results show that the proposed method outperforms the baseline methods on gaze estimation, and can provide a stable head pose tracking for driver behavior analysis in real-world driving scenarios.

Real-Time 3D Head Pose Tracking Through 2.5D Constrained Local Models with Local Neural Fields

Tracking the head in a video stream is a common thread seen within computer vision literature, supplying the research community with a large number of challenging and interesting problems. Head pose estimation from monocular cameras is often considered an extended application after the face tracking task has already been performed. This often involves passing the resultant 2D data through a simpler algorithm that best fits the data to a static 3D model to determine the 3D pose estimate. This work describes the 2.5D constrained local model, combining a deformable 3D shape point model with 2D texture information to provide direct estimation of the pose parameters, avoiding the need for additional optimization strategies. It achieves this through an analytical derivation of a Jacobian matrix describing how changes in the parameters of the model create changes in the shape within the image through a full-perspective camera model. In addition, the model has very low computational complexity and can run in real-time on modern mobile devices such as tablets and laptops. The point distribution model of the face is built in a unique way, so as to minimize the effect of changes in facial expressions on the estimated head pose and hence make the solution more robust. Finally, the texture information is trained via local neural fields—a deep learning approach that utilizes small discriminative patches to exploit spatial relationships between the pixels and provide strong peaks at the optimal locations.

Investigating the correspondence between driver head position and glance location.

The relationship between a driver’s glance orientation and corresponding head rotation is highly complex due to its nonlinear dependence on the individual, task, and driving context. This paper presents expanded analytic detail and findings from an effort that explored the ability of head pose to serve as an estimator for driver gaze by connecting head rotation data with manually coded gaze region data using both a statistical analysis approach and a predictive (i.e., machine learning) approach. For the latter, classification accuracy increased as visual angles between two glance locations increased. In other words, the greater the shift in gaze, the higher the accuracy of classification. This is an intuitive but important concept that we make explicit through our analysis. The highest accuracy achieved was 83% using the method of Hidden Markov Models (HMM) for the binary gaze classification problem of (a) glances to the forward roadway versus (b) glances to the center stack. Results suggest that although there are individual differences in head-glance correspondence while driving, classifier models based on head-rotation data may be robust to these differences and therefore can serve as reasonable estimators for glance location. The results suggest that driver head pose can be used as a surrogate for eye gaze in several key conditions including the identification of high-eccentricity glances. Inexpensive driver head pose tracking may be a key element in detection systems developed to mitigate driver distraction and inattention.

Towards Kilo-Hertz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras.

To maintain a reliable registration of the virtual world with the real world, augmented reality (AR) applications require highly accurate, low-latency tracking of the device. In this paper, we propose a novel method for performing this fast 6-DOF head pose tracking using a cluster of rolling shutter cameras. The key idea is that a rolling shutter camera works by capturing the rows of an image in rapid succession, essentially acting as a high-frequency 1D image sensor. By integrating multiple rolling shutter cameras on the AR device, our tracker is able to perform 6-DOF markerless tracking in a static indoor environment with minimal latency. Compared to state-of-the-art tracking systems, this tracking approach performs at significantly higher frequency, and it works in generalized environments. To demonstrate the feasibility of our system, we present thorough evaluations on synthetically generated data with tracking frequencies reaching 56.7 kHz. We further validate the method's accuracy on real-world images collected from a prototype of our tracking system against ground truth data using standard commodity GoPro cameras capturing at 120 Hz frame rate.

Model-free non-rigid head pose tracking by joint shape and pose estimation

Head pose estimation under non-rigid face movement is particularly useful in applications relating to eye-gaze tracking in less constrained scenarios, where the user is allowed to move naturally during tracking. Existing vision-based head pose estimation methods often require accurate initialisation and tracking of specific facial landmarks, while methods that handle non-rigid face deformations typically necessitate a preliminary training phase prior to head pose estimation. In this paper, we propose a method to estimate the head pose in real-time from the trajectories of a set of feature points spread randomly over the face region, without requiring a training phase or model-fitting of specific facial features. Conversely, our method exploits the 3-dimensional shape of the surface of interest, recovered via shape and motion factorisation, in combination with Kalman and particle filtering to determine the contribution of each feature point to the estimation of head pose based on a variance measure. Quantitative and qualitative results reveal the capability of our method in handling non-rigid face movement without deterioration of the head pose estimation accuracy.

DRIVER HEAD MOVEMENT ESTIMATOR ON ARM BIG.LITTLE PROCESSOR

A driving environment that is changing constantly and rapidly, demands a driver to make large head turns. Drivers head movements are strong indicators of drivers focus on the road. In this paper a vision based algorithm is presented to estimate the driver head pose. Determining the head pose using vision based algorithms is a non-invasive method in intelligent driver assistance systems. Many existing state of the art vision based head pose algorithms have difficulties in monitoring the driver head movements. This is because in single camera perspective spatially large head turns disturb the facial features which are necessary to determine the head pose. A distributed camera framework and the use of Constrained Local Model algorithm for head pose tracking is presented. The proposed approach monitors the driver head over wide range of movements.

Gaze Estimation in the 3D Space Using RGB-D Sensors

We address the problem of 3D gaze estimation within a 3D environment from remote sensors, which is highly valuable for applications in human---human and human---robot interactions. To the contrary of most previous works, which are limited to screen gazing applications, we propose to leverage the depth data of RGB-D cameras to perform an accurate head pose tracking, acquire head pose invariance through a 3D rectification process that renders head pose dependent eye images into a canonical viewpoint, and computes the line-of-sight in the 3D space. To address the low resolution issue of the eye image resulting from the use of remote sensors, we rely on the appearance based gaze estimation paradigm, which has demonstrated robustness against this factor. In this context, we do a comparative study of recent appearance based strategies within our framework, study the generalization of these methods to unseen individual, and propose a cross-user eye image alignment technique relying on the direct registration of gaze-synchronized eye images. We demonstrate the validity of our approach through extensive gaze estimation experiments on a public dataset as well as a gaze coding task applied to natural job interviews.

Overview of Hybrid Tracking in Augmented Reality

Hybrid tracking is one of the most important key technologies having received rapid development over the past twenty years in the field of augmented reality (AR). It is an effective approach to realize head-pose tracking with high accuracy and strong robustness for AR application systems. This paper tries to provide a complete and comprehensive survey on hybrid tracking technology. Three important issues involved in hybrid trackers are described in detail, including hybrid tracking method, calibration, and time synchronization. The state-of-the-art applications of different hybrid trackers are introduced. The pending problems and development tendency of hybrid tracking technology are discussed, and finally the application prospect is reviewed.

A new approach for measurement of pitch, roll and yaw angles based on a circular feature

This paper presents a simple, yet very effective approach to measure the three types of rotation angles by the method of imaging analysis. This approach is suitable for many actual applications, such as head pose tracking, vehicle monitoring, robotic operation, etc. In this paper, a circular feature is fixed on the object, which can do the rotational motion. By measuring the rotational angles of the circular feature based on perspective projection, we can obtain the object’s rotational angles. We deduce the computation methods of the pure pitch, roll and yaw angles. Then we propose the methods of computing the pitch, roll and yaw angles in complex situations, which are different from the computation methods of pure pitch, roll and yaw angles. Based on the approach proposed in this paper, some experiments are conducted and the results show that the approach can achieve good performance when the rotation angles are in the scope of ±90°.

3D head pose tracking using a particle filter with nose template matching

AbstractIn this paper, we propose a method for real‐time tracking of the driver's head pose in a real vehicle environment by using multiple NIR cameras. In order to achieve real‐time performance and high accuracy, the head pose is estimated with six degrees of freedom by a particle filter with a restricted state space. First, the 3D position of the nostrils is measured by template matching of stereo images. Because the nostrils are the darkest area in the captured NTR images, we can detect the position of the nose robustly and very precisely. From the 3D position of the nose, we can make an initial estimate of the head pose. Then, each hypothesis is updated by prior probability with the constraint of the nose position. Thus, it is possible to reduce the number of particles while maintaining accuracy. Experimental results indicate that this method is effective for head pose tracking. © 2010 Wiley Periodicals, Inc. Electron Comm Jpn, 94(1): 34–42, 2011; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/ecj.10268

Multi-sensor Data Fusion for Optical Tracking of Head Pose

Accurate head pose tracking is a key issue to accomplish precise registration in indoor augmented reality systems.This paper proposes a novel approach based on multi-sensor data fusion to achieve optical tracking of head pose with high accuracy.This approach employs two extended Kalman filters and one fusion filter for multi-sensor environment to fuse the pose data from two complement optical trackers,an inside-out tracking(IOT) with a camera and an outside-in tracking(OIT) with two cameras,respectively.The aim is to reduce the pose errors from the optical tracking sensors.A representative experimental setup is designed to verify the above approach.The experimental results show that,in the static state,the pose errors of IOT and OIT are consistent to the theoretical results obtained using the rules of error covariance matrix propagation from the respective image noises to the final pose errors,and that in the dynamic state,the proposed multi-sensor data fusion approach used with our combined optical tracker can achieve more accurate and more stable outputs of position and orientation compared with using a single IOT or OIT alone.

An integrated head pose and eye gaze tracking approach to non-intrusive visual attention measurement for wide FOV simulators

Eye gaze tracking is very useful for quantitatively measuring visual attention in virtual environments. However, most eye trackers have a limited tracking range, e.g., ±35° in the horizontal direction. This paper proposed a method to combine head pose tracking and eye gaze tracking together to achieve a large range of tracking in virtual driving simulation environments. Multiple parallel multilayer perceptrons were used to reconstruct the relationship between head images and head poses. Head images were represented with the coefficients extracted from Principal Component Analysis. Eye gaze tracking provides precise results on the front view, while head pose tracking is more suitable for tracking areas of interest than for tracking points of interest on the side view.

Sensor fusion based head pose tracking for lightweight flight cockpit systems

Flight cockpit head tracking systems (HTS) are one of the most important impetuses for head pose tracking in the field of augmented reality. For the purpose of implementing the natural interaction between pilot and the complete internal environment in our lightweight flight cockpit system, a head tracking system consisting of inside-out tracking (IOT) and outside-in tracking (OIT) is designed and a novel approach using sensor fusion is proposed to dynamically track pilot's head pose. The proposed approach utilizes a sensor fusion framework, composed of extended Kalman filters and fusion filter, to fuse the poses from complementary IOT and OIT. An experimental setup is established to simulate the cockpit HTS and verify the proposed approach. Experimental results show that the proposed tracking scheme based on sensor fusion is capable of achieving more accurate and stable pose outputs, extending tracking range as well as better robustness compared with single IOT or OIT.