User's Head Movement Research Articles

Deep Variational Learning for 360° Adaptive Streaming

Prediction of head movements in immersive media is key to designing efficient streaming systems able to focus the bandwidth budget on visible areas of the content. However, most of the numerous proposals made to predict user head motion in 360° images and videos do not explicitly consider a prominent characteristic of the head motion data: its intrinsic uncertainty. In this article, we present an approach to generate multiple plausible futures of head motion in 360° videos, given a common past trajectory. To our knowledge, this is the first work that considers the problem of multiple head motion prediction for 360° video streaming. We introduce our discrete variational multiple sequence (DVMS) learning framework, which builds on deep latent variable models. We design a training procedure to obtain a flexible, lightweight stochastic prediction model compatible with sequence-to-sequence neural architectures. Experimental results on 4 different datasets show that our method DVMS outperforms competitors adapted from the self-driving domain by up to 41% on prediction horizons up to 5 sec., at lower computational and memory costs. To understand how the learned features account for the motion uncertainty, we analyze the structure of the learned latent space and connect it with the physical properties of the trajectories. We also introduce a method to estimate the likelihood of each generated trajectory, enabling the integration of DVMS in a streaming system. We hence deploy an extensive evaluation of the interest of our DVMS proposal for a streaming system. To do so, we first introduce a new Python-based 360° streaming simulator that we make available to the community. On real-world user, video, and networking data, we show that predicting multiple trajectories yields higher fairness between the traces, the gains for 20 to 30% of the users reaching up to 10% in visual quality for the best number K of trajectories to generate.

Continuous Gaze Tracking With Implicit Saliency-Aware Calibration on Mobile Devices

Gaze tracking is a useful human-to-computer interface, which plays an increasingly important role in a range of mobile applications. Gaze calibration is an indispensable component of gaze tracking, which transforms the eye coordinates to the screen coordinates. The existing approaches of gaze tracking either have limited accuracy or require the user's cooperation in calibration and in turn hurt the quality of experience. We in this paper propose vGaze, continuous gaze tracking with implicit saliency-aware calibration on mobile devices. The design of vGaze stems from our insight on the temporal and spatial dependent relation between the visual saliency and the user's gaze. vGaze is implemented as a light-weight software that identifies video frames with "useful" saliency information, sensing the user's head movement, performs opportunistic calibration using only those "useful" frames, and leverages historical information for accelerating saliency detection. We implement vGaze on a commercial mobile device and evaluate its performance in various scenarios. The results show that vGaze can work at real time with video playback applications. The average error of gaze tracking is 1.51 cm (2.884 degree) which decreases to 0.99 cm (1.891 degree) with historical information and 0.57 cm (1.089 degree) with an indicator.

A hat-shape error microphone array for user-friendly spatial active noise control

Spatial active noise control (ANC) has been developed in the past decades to reduce unwanted acoustic noise over a desired region by generating an anti-noise field with secondary loudspeakers. Unlike conventional multi-channel ANC systems, the integration of spherical harmonic-based sound field analysis allows the ANC system to reduce noise over a continuous region surrounding the user's head rather than discrete points. However, the use of a spherical error microphone array on the boundary of the region for spherical harmonic analysis poses a limitation as it obstructs user access to the controlled area. In this work, we introduce a hat-shaped error microphone array design which can be placed above the listener's head. The proposed error microphone array can achieve spherical harmonic-based residual noise field analysis and is more user-friendly, allowing unrestricted user head movement. We construct the hat-shaped error microphone array and present the results of ANC experiments conducted using the array.

Wheelchair-Centered Omnidirectional Gaze-Point Estimation in the Wild

In this article, we propose a dataset named Wheelchair-OmniGaze and a method of estimating the gaze point of a wheelchair user that can cope with large head motion as the first step for understanding the intention of a wheelchair user. In the proposed method, the wheelchair user's face is observed by two remote cameras, called face cameras, mounted on the two front sides of the wheelchair, while a spherical camera with a 360° field of view, called a spherical scene camera, is mounted to the top rear to observe the surrounding scenes. A user's gaze point at the environment is determined by combining the facial information acquired by the face cameras with the spherical image acquired by the spherical scene camera. Concretely, a convolutional neural network–long short-term memory network is used to learn gaze points of the spherical image from the two face image sequences. Using two face cameras enables a user's face to be observed even for a large head motion. Using a spherical scene camera mounted on a wheelchair, the representation of a user's gaze points at a wheelchair-centered spherical camera coordinate can be obtained, which is independent of a user's head motion. The experimental results show that our method achieved competitive performance in comparison with the state-of-the-art method. Since our problem definition is different from the existing research, the experimental results of this article can be seen as a baseline for our built dataset.

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display.

This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

56‐2: A Distraction‐free Display System using Embedded Asynchronous Time Warp

We propose a distraction‐free display system and present display controller design to realize the system. The designed controller utilized embedded asynchronous time warp (ATW) for dynamic view change in accordance with a user's head motion. Proposed design can be used for lightweight information displays to provide better user experience as an affordable solution even without application processor‐centric designs.

Dynamic Viewport Selection-Based Prioritized Bitrate Adaptation for Tile-Based 360° Video Streaming

Interactive 360° remote video applications have seen booming advancements due to the proliferation of smart display devices that enable a truly immersive experience. Compared to regular monoscopic videos, 360° videos have different requirements related to content preparation, packaging, transmission, specialized viewing equipment, and display factors (e.g., brightness, contrast, delay, frame rate, resolution, image quality, etc. In addition, 360° video requires substantial network and computational resources, which are challenging to achieve with conventional transmission and rendering infrastructure. Viewport-adaptive streaming is a common way to ensure visual quality under limited bandwidth resources. However, identifying, extracting, and rendering the true viewport in response to drastic head rotations can adversely affect user experience. This paper proposes two dynamic viewport selection approaches, which adapt the streamed regions based on content complexity variations and positional information to ensure viewport availability and smooth visual angles for VR users. They incorporate content information as well as user head movement patterns to support tile-based prioritized 360° video streaming. Moreover, a practical, prioritized bitrate adaptation approach, which requests selected tiles at appropriate quality levels, is also proposed to reduce the impact of inefficient bandwidth utilization in the VR scene. Experimental evaluations under real 4G bandwidth logs demonstrate that the proposed solutions outperform the closest state-of-the-art algorithms across multiple performance metrics, i.e., viewport overlap, perceived quality levels, quality fluctuations, and viewport bandwidth utilization.

Viewport History as a Heuristic for Quality Enhancement and Quality Variation Control in Viewport-Aware Tile-Based 360-Degree Video Streaming

Despite the growing popularity of Virtual Reality (VR), 360-degree videos are often regarded as challenging to stream due to their large bandwidth requirement. As a solution, the 360-degree video content is spatially divided into tiles, and the quality level for each tile is selected based on the user’s network environment and viewport information. To determine the high quality tiles, viewport prediction and viewport history methods are used to estimate the user’s viewport. However, due to the unpredictability of user head movements, generating accurate viewport estimates are difficult, which can severely degrade the Quality of Experience (QoE) for the user. In this paper, to sustain high user QoE, we detail a novel tile quality selection algorithm that employs viewport prediction, viewport history, viewport extensions, and a viewport tile count limit. In addition, we also include comparison analysis on six 360-degree videos that vary in content pace. Based from simulations, using viewport history as a heuristic for tile quality selection demonstrated a significant increase in the perceived quality while suppressing quality variation inside the viewport and across segments compared to eight reference methods; and secondly, 360-degree videos slow in content pace tended to result in lower viewport prediction accuracy, QoE performance, and weaker viewport history trends compared to 360-degree videos fast in content pace.

Enhancing Virtual Walking Sensation Using Self-Avatar in First-Person Perspective and Foot Vibrations

Walking is a fundamental physical activity in humans. Various virtual walking systems have been developed using treadmill or leg-support devices. Using optic flow, foot vibrations simulating footsteps, and a walking avatar, we propose a virtual walking system that does not require limb action for seated users. We aim to investigate whether a full-body or hands-and-feet-only walking avatar with either the first-person (experiment 1) or third-person (experiment 2) perspective can convey the sensation of walking in a virtual environment through optic flows and foot vibrations. The viewing direction of the virtual camera and the head of the full-body avatar were linked to the actual user's head motion. We discovered that the full-body avatar with the first-person perspective enhanced the sensations of walking, leg action, and telepresence, either through synchronous or asynchronous foot vibrations. Although the hands-and-feet-only avatar with the first-person perspective enhanced the walking sensation and telepresence, compared with the no-avatar condition, its effect was less prominent than that of the full-body avatar. However, the full-body avatar with the third-person perspective did not enhance the sensations of walking and leg action; rather, it impaired the sensations of self-motion and telepresence. Synchronous or rhythmic foot vibrations enhanced the sensations of self-motion, waking, leg action, and telepresence, irrespective of the avatar condition. These results suggest that the full-body or hands-and-feet avatar is effective for creating virtual walking experiences from the first-person perspective, but not the third-person perspective, and that the foot vibrations simulating footsteps are effective, regardless of the avatar condition.

Implementasi Algoritma Complementary Filter untuk Merancang Bangun Interaksi Manusia dan Smartphone Menggunakan Gerakan Kepala

This study proposes an approach to design and build interaction systems for smartphone users with hand disabilities using the user's head movements through the implementation of the fusion complementary filter algorithm to get the angle of orientation and position of the user’s head. The main objective of this experimental research is to get the best constant function of complementary filters that have the highest accuracy result. The results of this study indicate that the fusion complementary filter algorithm can be used to measure the angle of the user's head movements using a combination of 3 sensors; namely, the accelerometer, gyroscope, and magnetometer, with the best constants function are 0.99 for the gyroscope and 0.01 for the accelerometer. Complementary filter performance results using fusion of 3 sensors to measure the user's head angle obtained from the calculation of the root mean squared error (RMSE) with an average value of 0.364 on the x-axis (roll), 0.578 on the y-axis (pitch), and 0.767 on the z-axis (yaw).

TRACK: A New Method From a Re-Examination of Deep Architectures for Head Motion Prediction in 360 ° Videos.

We consider predicting the user's head motion in 360 ° videos, with 2 modalities only: the past user's positions and the video content (not knowing other users' traces). We make two main contributions. First, we re-examine existing deep-learning approaches for this problem and identify hidden flaws from a thorough root-cause analysis. Second, from the results of this analysis, we design a new proposal establishing state-of-the-art performance. First, re-assessing the existing methods that use both modalities, we obtain the surprising result that they all perform worse than baselines using the user's trajectory only. A root-cause analysis of the metrics, datasets and neural architectures shows in particular that (i) the content can inform the prediction for horizons longer than 2 to 3 sec. (existing methods consider shorter horizons), and that (ii) to compete with the baselines, it is necessary to have a recurrent unit dedicated to process the positions, but this is not sufficient. Second, from a re-examination of the problem supported with the concept of Structural-RNN, we design a new deep neural architecture, named TRACK. TRACK achieves state-of-the-art performance on all considered datasets and prediction horizons, outperforming competitors by up to 20 percent on focus-type videos and horizons 2-5 seconds. The entire framework (codes and datasets) is online and received an ACM reproducibility badge https://gitlab.com/miguelfromeror/head-motion-prediction.

Visual and Gyroscope Sensor for Head Movement Controller System on Meal-Assistance Application

Head movement utilizes gestures to aid people with disabilities so that they can have hands-free human-computer interaction. Currently, motion-based sensor is the most widely used approach to recognize head gestures. Identification of head movement is important to control a robotic manipulator in an assisting device. However, the most effective methodologies to assess head angular movements are yet to be discovered. This paper combines two algorithms, the visual sensor and the gyro sensor, to identify head orientation movement with high precision. Head orientations were measured using data distribution and this was done with a meal-assistance robot manipulator used in a sitting position. Evaluation of the accuracy of the system shows a visual sensor and gyro sensor. Experimental results show that a correct head movement with the average accuracy is 82%. Therefore, we propose the application of position control of meal assistive robot based on user's head movement in a sitting position.

Predictive Adaptive Streaming to Enable Mobile 360-Degree and VR Experiences

As 360-degree videos and virtual reality (VR) applications become popular for consumer and enterprise use cases, the desire to enable truly mobile experiences also increases. Delivering 360-degree videos and cloud/edge-based VR applications require ultra-high bandwidth and ultra-low latency <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> , challenging to achieve with mobile networks. A common approach to reduce bandwidth is streaming only the field of view (FOV). However, extracting and transmitting the FOV in response to user head motion can add high latency, adversely affecting user experience. In this paper, we propose a predictive adaptive streaming approach, where the predicted view with high predictive probability is adaptively encoded in relatively high quality according to bandwidth conditions and transmitted in advance, leading to a simultaneous reduction in bandwidth and latency. The predictive adaptive streaming method is based on a deep-learning-based viewpoint prediction model we develop, which uses past head motions to predict where a user will be looking in the 360-degree view. Using a very large dataset consisting of head motion traces from over 36,000 viewers for nineteen 360-degree/VR videos, we validate the ability of our predictive adaptive streaming method to offer high-quality view while simultaneously significantly reducing bandwidth.

Errata to "RingText: Dwell-Free and Hands-Free Text Entry for Mobile Head-Mounted Displays Using Head Motions".

In this paper, we present a case for text entry using a circular keyboard layout for mobile head-mounted displays (HMDs) that is dwell-free and does not require users to hold a dedicated input device for letter selection. To support the case, we have implemented RingText whose design is based on a circular layout with two concentric circles. The outer circle is subdivided into regions containing letters. Selection is made by using a virtual cursor controlled by the user's head movements—entering a letter region triggers a selection and moving back into the inner circle resets the selection. The design of RingText follows an iterative process, where we initially conduct one first study to investigate the optimal number of letters per region, inner circle size, and alphabet starting location. We then optimize its design by selecting the most suitable features from the first study: one letter per region, narrowing the trigger area to lower error rates, and creating candidate regions that incorporate two suggested words to appear next to the current letter region (close to the cursor) using a dynamic (rather than fixed) approach. Our second study compares text entry performance of RingText with four other hands-free techniques and the results show that RingText outperforms them. Finally, we run a third study lasting four consecutive days with 10 participants (5 novice users and 5 expert users) doing two daily sessions and the results show that RingText is quite efficient and yields a low error rate. At the end of the eighth session, the novice users can achieve a text entry speed of 11.30 WPM after 60 minutes of training while the expert (more experienced) users can reach an average text entry speed of 13.24 WPM after 90 minutes of training.

Viewport Prediction for Live 360-Degree Mobile Video Streaming Using User-Content Hybrid Motion Tracking

360-degree video streaming has been gaining popularities recently with the rapid growth of adopting mobile head mounted display (HMD) devices in the consumer video market, especially for live broadcasts. The 360-degree video streaming introduces brand new bandwidth and latency challenges in live streaming due to the significantly increased video data. However, most of the existing bandwidth saving approaches based on viewport prediction have only focused on the video-on-demand (VOD) use cases leveraging historical user behavior data, which is not available in live broadcasts. We develop a new viewport prediction scheme for live 360-degree video streaming using video content-based motion tracking and dynamic user interest modeling. To obtain real-time performance, we implement the Gaussian mixture model (GMM) and optical flow algorithms for motion detection and feature tracking. Then, the user's future viewport of interest is generated by leveraging a dynamic user interest model that weighs all the features and motion information abstracted from the live video frames. Furthermore, we develop two enhancement techniques that take into consideration of user feedback for fast error recovery and view updates. Consequently, our predicted viewports are irregular and dynamically adjusted to cover the maximum portions of the actual user viewports and thus ensure a high prediction accuracy. We evaluate our viewport prediction approach using a public user head movement dataset, which contains the data of 48 users watching 6 360-degree videos. The experimental results show that the proposed approach supports sophisticated user head movement patterns and outperforms the existing velocity-based approach in terms of prediction accuracy. In addition, the motion tracking scheme introduces minimum latency overhead to ensure the quality of live streaming experience.

An Optimal Tile-Based Approach for Viewport-Adaptive 360-Degree Video Streaming

In this paper, we propose a new adaptation approach for viewport-adaptive streaming of 360-degree videos over the Internet. The proposed approach is able to systematically decide versions of tiles according to user head movements and network conditions by taking into account not only viewport estimation errors, but also users head movements in each segment duration. The experimental results show that the proposed approach can effectively adapt 360-degree videos to both varying network conditions and user’s head movements. Compared to existing approaches, the proposed approach can improve the average viewport quality by up to 3.8 dB and reduce the standard deviation of the viewport quality by up to 1.1 dB. Also, the impacts of the segment duration and the buffer size are investigated. It is found that long segment durations and large buffer sizes have significant impacts on the performances of tile selection methods.

Behavioral Analysis of Kinetic Telepresence for Small Symmetric Group-to-Group Meetings

Nonverbal behavior analysis revealed the effect of MMSpace , a kinetic telepresence developed for social telepresence, on small symmetric group-to-group conversations. MMSpace consists of kinetic avatars, equipped with flat projection screen panels as faces, that can change their pose and position automatically to mirror the remote user's head motions. The advantage is the realistic kinetic expression of human head movements, which form gestures like nodding and indicate the focus of visual attention, through the use of four degree-of-freedom low-latency precision actuators. Another feature is the support of eye contact among remote participants, which is made possible by the avatar's kinetic pose changes and by adaptive camera selection for orienting the user's face toward the remote addressee. Its limitation is its room-scale infrastructure and restricted participant positions. Targeting a symmetric 2 $\times$ 2 setting, participants’ nonverbal behaviors, including gaze directions and head gestures, were compared among three conditions, MMSpace with/without physical motions and face-to-face settings. There was a significant difference between the conditions in terms of the duration of glance/mutual glances, total gaze transition time, amount of head gesturing, and co-occurrences of head gestures in the remote participants. The results indicate that the avatar's physical motion can elicit longer (mutual) glances with a shorter total transition time and more (co-)occurrences of head gestures, and it makes MMSpace -based conversations closer, in terms of these nonverbal statistics, to face-to-face ones compared with those of a static version of MMSpace without physical motion.

51‐5: Late-News Paper: Fast Motion Picture Response Color Filter LCOS for Wearable Applications

The display for augmented reality (AR) applications has to update image promptly in order to cope with the user's head movement or any environment change. It makes image blur become noticeable if the motion picture response time of the AR display is longer than 1.5ms. We propose an image blur free Color Filter LCOS (CFLCOS) which equipped with 1.9ms liquid crystal response time; analog frame buffer and 480fps refreshing rate. It is found that the MPRT of such device can be as fast as &lt;1.5ms.

Human-computer interaction for users with cerebral palsy based on head orientation. Can cursor’s movement be modeled by Fitts’s law?

Abstract This paper presents an experiment to validate a head-mounted inertial interface for human-computer interaction (HCI) developed for people with cerebral palsy (CP). The method is based on Fitts’s law, an empirical model of human motor performance for aimed movements. Head motion is recorded in a series of goal-crossing tasks and a regression model of the movement time ( MT ) is estimated for each user. Values of R 2 above 0.9 are indicators of a strong correlation of those motion patterns with the linear model proposed by Fitts. The analysis of MT confirmed that head movements of users without disability follow Fitts’s law and showed that 3 users with CP (MACS IV and V) had the same behavior. There was a weaker correlation ( R 2 =0.839) for one individual with cervical dystonia and ballistic movements and no correlation for two users with cervical hypotonia and dyskinetic CP. Results show the impact of ballistic movements and poor postural control in computer interaction. They also provide the foundation for new interaction techniques to develop a universal computer interface for motor impaired users.

Individual differences in virtual reality: Are spatial presence and spatial ability linked?

One aim of virtual reality technology is to immerse the user in a digital environment that is distinct from physical reality. Feeling spatially located in this digital environment is central to the experience and is more formally known as spatial presence. Experiences of spatial presence differ between individuals; prominent theories assume that these differences may, in part, be explained by differences in more general spatial abilities. Whilst there is some support for this claim with desktop systems, there is currently no direct empirical evidence to support this with more immersive technologies such as head-mounted displays (HMDs). In this study, participants completed three different measures of spatial ability before experiencing two virtual environments. These measures included a self-report of visuospatial imagery; the mental rotations test; and a test of topographical memory. After completing the measures, participants briefly experienced a virtual city and a virtual train ride through a HMD. The user's head movements were tracked, and visual displays were updated to give the sense of a full 360° environment. After each experience, the participants reported how present they felt and the extent to which they had a mental model of the environment. Self-reports of imagery were positively correlated with reports of spatial presence, consistent with the previous literature. However, spatial presence was not related to performance on either of the more objective tests. Whilst this provides confirmatory evidence that self-reports of imagery can predict presence, it is still unclear which more basic spatial abilities, if any, could underlie this relationship.