Commercial Head-mounted Display Research Articles

Quantitative Comparison of Hand Kinematics Measured with a Markerless Commercial Head-Mounted Display and a Marker-Based Motion Capture System in Stroke Survivors.

Upper-limb paresis is common after stroke. An important tool to assess motor recovery is to use marker-based motion capture systems to measure the kinematic characteristics of patients' movements in ecological scenarios. These systems are, however, very expensive and not readily available for many rehabilitation units. Here, we explored whether the markerless hand motion capabilities of the cost-effective Oculus Quest head-mounted display could be used to provide clinically meaningful measures. A total of 14 stroke patients executed ecologically relevant upper-limb tasks in an immersive virtual environment. During task execution, we recorded their hand movements simultaneously by means of the Oculus Quest's and a marker-based motion capture system. Our results showed that the markerless estimates of the hand position and peak velocity provided by the Oculus Quest were in very close agreement with those provided by a marker-based commercial system with their regression line having a slope close to 1 (maximum distance: mean slope = 0.94 ± 0.1; peak velocity: mean slope = 1.06 ± 0.12). Furthermore, the Oculus Quest had virtually the same sensitivity as that of a commercial system in distinguishing healthy from pathological kinematic measures. The Oculus Quest was as accurate as a commercial marker-based system in measuring clinically meaningful upper-limb kinematic parameters in stroke patients.

Full body video-based self-avatars for mixed reality: from E2E system to user study

In this work, we explore the creation of self-avatars through video pass-through in mixed reality (MR) applications. We present our end-to-end system, including custom MR video pass-through implementation on a commercial head-mounted display (HMD), our deep learning-based real-time egocentric body segmentation algorithm, and our optimized offloading architecture, to communicate the segmentation server with the HMD. To validate this technology, we designed an immersive VR experience where the user has to walk through a narrow tile path over an active volcano crater. The study was performed under three-body representation conditions: virtual hands, video pass-through with color-based full-body segmentation, and video pass-through with deep learning full-body segmentation. This immersive experience was carried out by 30 women and 28 men. To the best of our knowledge, this is the first user study focused on evaluating video-based self-avatars to represent the user in a MR scene. Results showed no significant differences between the different body representations in terms of presence, with moderate improvements in some Embodiment components between the virtual hands and full-body representations. Visual Quality results showed better results from the deep-learning algorithms in terms of the whole body perception and overall segmentation quality. In this study, we provide some discussion regarding the use of video-based self-avatars and some reflections on the evaluation methodology. The proposed E2E solution is in the boundary of the state-of-the-art, so there is still room for improvement before it reaches maturity. However, this solution serves as a crucial starting point for MR applications where users can feel immersed and interact with their own bodies.

Examining the feasibility of integrating pupillometry measures of listening effort into clinical audiology assessments

Pure-tone thresholds are often a poor predictor of hearing difficulties experienced by patients in audiology clinics. Approximately 25 million Americans may experience hearing difficulties despite having audiometric thresholds within normal limits. Pupil diameter increases with increasing listening effort and could provide an objective measure for determining the presence of such subclinical hearing problems. However, the cost and technical requirements of research-grade eye trackers have limited their use in audiology clinics. This study aims to determine the feasibility of using a commercially-available head mounted display (HMD) with integrated eye tracking to collect pupillometry measures during audiological testing. Pupillary data were collected for N = 46 younger normal-hearing adults while they completed clinically-relevant tests including pure tone audiometry, gaps-in-noise (GIN), dichotic digits, and speech-in-noise tests. Across tasks, task-evoked pupillary responses were generally sensitive to differences in listening demands. Results suggest audiologic testing with commercial HMD eye trackers is feasible and provides additional objective information related to listening effort that may help clinicians better understand and address patient needs. [The views expressed in this abstract are those of theauthors and do not reflect the official policy of the Department of Army/Navy/Air Force, Department of Defense, or U.S. Government.]

ThumbAir

Typing while wearing a standalone Head Mounted Display (HMD)---systems without external input devices or sensors to support text entry---is hard. To address this issue, prior work has used external trackers to monitor finger movements to support in-air typing on virtual keyboards. While performance has been promising, current systems are practically infeasible: finger movements may be visually occluded from inside-out HMD based tracking systems or, otherwise, awkward and uncomfortable to perform. To address these issues, this paper explores an alternative approach. Taking inspiration from the prevalence of thumb-typing on mobile phones, we describe four studies exploring, defining and validating the performance of ThumbAir, an in-air thumb-typing system implemented on a commercial HMD. The first study explores viable target locations, ultimately recommending eight targets sites. The second study collects performance data for taps on pairs of these targets to both inform the design of a target selection procedure and also support a computational design process to select a keyboard layout. The final two studies validate the selected keyboard layout in word repetition and phrase entry tasks, ultimately achieving final WPMs of 27.1 and 13.73. Qualitative data captured in the final study indicate that the discreet movements required to operate ThumbAir, in comparison to the larger scale finger and hand motions used in a baseline design from prior work, lead to reduced levels of perceived exertion and physical demand and are rated as acceptable for use in a wider range of social situations.

Immersive Technologies for Vestibular Rehabilitation

Vestibular vertigo accounts for about a quarter of dizziness complaints. Loss of vestibular function is a debilitating condition that causes nausea, spontaneous nystagmus, or disequilibrium, which is known to highly influence day-to-day tasks. Therefore, recovery should start as soon as possible, targeting the affected side of the vestibular apparatus, by performing a set of prescribed exercises. Aside from being a long and tiresome process, patients must perform these exercises, while trying to stimulate optokinetic, angular vestibulo-ocular, and vestibulospinal reflexes. The article presents the development of virtual reality serious games that can be played at home or in a rehabilitation clinic. The main objective is to increase patients' motivation, specifically during telerehabilitation which is essential for a faster recovery process. A preliminary evaluation was carried out to compare the users' experiences using a smartphone-based headset and a standalone commercial head-mounted display, the Oculus Quest.

Wearable Immersive Virtual Reality Device for Promoting Physical Activity in Parkinson's Disease Patients.

Parkinson’s disease (PD) is a neurological disorder that usually appears in the 6th decade of life and affects up to 2% of older people (65 years and older). Its therapeutic management is complex and includes not only pharmacological therapies but also physiotherapy. Exercise therapies have shown good results in disease management in terms of rehabilitation and/or maintenance of physical and functional capacities, which is important in PD. Virtual reality (VR) could promote physical activity in this population. We explore whether a commercial wearable head-mounted display (HMD) and the selected VR exergame could be suitable for people with mild–moderate PD. In all, 32 patients (78.1% men; 71.50 ± 11.80 years) were a part of the study. Outcomes were evaluated using the Simulator Sickness Questionnaire (SSQ), the System Usability Scale (SUS), the Game Experience Questionnaire (GEQ post-game module), an ad hoc satisfaction questionnaire, and perceived effort. A total of 60 sessions were completed safely (without adverse effects (no SSQ symptoms) and with low scores in the negative experiences of the GEQ (0.01–0.09/4)), satisfaction opinions were positive (88% considered the training “good” or “very good”), and the average usability of the wearable HMD was good (75.16/100). Our outcomes support the feasibility of a boxing exergame combined with a wearable commercial HMD as a suitable physical activity for PD and its applicability in different environments due to its safety, usability, low cost, and small size. Future research is needed focusing on postural instability, because it seems to be a symptom that could have an impact on the success of exergaming programs aimed at PD.

Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform

In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.

Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360$^\circ$ Videos: ITU-T Rec. P.919

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> sequences. The following factors&#x2019; influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.

A Structured Approach to Test the Signal Quality of Electroencephalography Measurements During Use of Head-Mounted Displays for Virtual Reality Applications.

Joint applications of virtual reality (VR) systems and electroencephalography (EEG) offer numerous new possibilities ranging from behavioral science to therapy. VR systems allow for highly controlled experimental environments, while EEG offers a non-invasive window to brain activity with a millisecond-ranged temporal resolution. However, EEG measurements are highly susceptible to electromagnetic (EM) noise and the influence of EM noise of head-mounted-displays (HMDs) on EEG signal quality has not been conclusively investigated. In this paper, we propose a structured approach to test HMDs for EM noise potentially harmful to EEG measures. The approach verifies the impact of HMDs on the frequency- and time-domain of the EEG signal recorded in healthy subjects. The verification task includes a comparison of conditions with and without an HMD during (i) an eyes-open vs. eyes-closed task, and (ii) with respect to the sensory- evoked brain activity. The approach is developed and tested to derive potential effects of two commercial HMDs, the Oculus Rift and the HTC Vive Pro, on the quality of 64-channel EEG measurements. The results show that the HMDs consistently introduce artifacts, especially at the line hum of 50 Hz and the HMD refresh rate of 90 Hz, respectively, and their harmonics. The frequency range that is typically most important in non-invasive EEG research and applications (<50 Hz) however, remained largely unaffected. Hence, our findings demonstrate that high-quality EEG recordings, at least in the frequency range up to 50 Hz, can be obtained with the two tested HMDs. However, the number of commercially available HMDs is constantly rising. We strongly suggest to thoroughly test such devices upfront since each HMD will most likely have its own EM footprint and this article provides a structured approach to implement such tests with arbitrary devices.

A Comparison of Eye Tracking Latencies Among Several Commercial Head-Mounted Displays.

A number of virtual reality head-mounted displays (HMDs) with integrated eye trackers have recently become commercially available. If their eye tracking latency is low and reliable enough for gaze-contingent rendering, this may open up many interesting opportunities for researchers. We measured eye tracking latencies for the Fove-0, the Varjo VR-1, and the High Tech Computer Corporation (HTC) Vive Pro Eye using simultaneous electrooculography measurements. We determined the time from the occurrence of an eye position change to its availability as a data sample from the eye tracker (delay) and the time from an eye position change to the earliest possible change of the display content (latency). For each test and each device, participants performed 60 saccades between two targets 20° of visual angle apart. The targets were continuously visible in the HMD, and the saccades were instructed by an auditory cue. Data collection and eye tracking calibration were done using the recommended scripts for each device in Unity3D. The Vive Pro Eye was recorded twice, once using the SteamVR SDK and once using the Tobii XR SDK. Our results show clear differences between the HMDs. Delays ranged from 15 ms to 52 ms, and the latencies ranged from 45 ms to 81 ms. The Fove-0 appears to be the fastest device and best suited for gaze-contingent rendering.

Commercial head-mounted display virtual reality for upper extremity rehabilitation in chronic stroke: a single-case design study

BackgroundRehabilitation is crucial for maximizing recovery after stroke. Rehabilitation activities that are fun and rewarding by themselves can be more effective than those who are not. Gamification with virtual reality (VR) exploits this principle. This single-case design study probes the potential for using commercial off-the-shelf, room-scale head-mounted virtual reality for upper extremity rehabilitation in individuals with chronic stroke, the insights of which can inform further research.MethodsA heterogeneous volunteer sample of seven participants living with stroke were recruited through advertisement. A single-case design was employed with a 5-week baseline (A), followed by a 10-week intervention (B) and a 6-month follow-up. Upper extremity motor function was assessed with validated kinematic analysis of drinking task. Activity capacity was assessed with Action Research Arm Test, Box and Block Test and ABILHAND questionnaire. Assessments were done weekly and at follow-up. Playing games on a VR-system with head-mounted display (HTC Vive) was used as rehabilitation intervention. Approximately 300 games were screened and 6 tested. Visual analysis and Tau-U statistics were used to interpret the results.ResultsVisual analysis of trend, level shift and overlap as well as Tau-U statistics indicated improvement of Action Research Arm Test in six participants. Four of these had at least a moderate Tau-U score (0.50–0.92), in at least half of the assessed outcomes. These four participants trained a total of 361 to 935 min. Two out of four participants who were able to perform the drinking task, had the highest training dose (> 900 min) and showed also improvements in kinematics. The predominant game played was Beat Saber. No serious adverse effects related to the study were observed, one participant interrupted the intervention phase due to a fall at home.ConclusionsThis first study of combining commercial games, a commercial head-mounted VR, and commercial haptic hand controls, showed promising results for upper extremity rehabilitation in individuals with chronic stroke. By being affordable yet having high production values, as well as being an easily accessible off-the-shelf product, this variant of VR technology might facilitate widespread adaption. Insights garnered in this study can facilitate the execution of future studies.Trial registration The study was registered at researchweb.org (project number 262331, registered 2019-01-30, https://www.researchweb.org/is/vgr/project/262331) prior to participant enrolment.

Influence of field of view in visual prostheses design: Analysis with a VR system

Objective. Visual prostheses are designed to restore partial functional vision in patients with total vision loss. Retinal visual prostheses provide limited capabilities as a result of low resolution, limited field of view and poor dynamic range. Understanding the influence of these parameters in the perception results can guide prostheses research and design. Approach. In this work, we evaluate the influence of field of view with respect to spatial resolution in visual prostheses, measuring the accuracy and response time in a search and recognition task. Twenty-four normally sighted participants were asked to find and recognize usual objects, such as furniture and home appliance in indoor room scenes. For the experiment, we use a new simulated prosthetic vision system that allows simple and effective experimentation. Our system uses a virtual-reality environment based on panoramic scenes. The simulator employs a head-mounted display which allows users to feel immersed in the scene by perceiving the entire scene all around. Our experiments use public image datasets and a commercial head-mounted display. We have also released the virtual-reality software for replicating and extending the experimentation. Main results. Results show that the accuracy and response time decrease when the field of view is increased. Furthermore, performance appears to be correlated with the angular resolution, but showing a diminishing return even with a resolution of less than 2.3 phosphenes per degree. Significance. Our results seem to indicate that, for the design of retinal prostheses, it is better to concentrate the phosphenes in a small area, to maximize the angular resolution, even if that implies sacrificing field of view.

Volumetric Head-Mounted Display With Locally Adaptive Focal Blocks.

A commercial head-mounted display (HMD) for virtual reality (VR) presents three-dimensional imagery with a fixed focal distance. The VR HMD with a fixed focus can cause visual discomfort to an observer. In this article, we propose a novel design of a compact VR HMD supporting near-correct focus cues over a wide depth of field (from 18 cm to optical infinity). The proposed HMD consists of a low-resolution binary backlight, a liquid crystal display panel, and focus-tunable lenses. In the proposed system, the backlight locally illuminates the display panel that is floated by the focus-tunable lens at a specific distance. The illumination moment and the focus-tunable lens' focal power are synchronized to generate focal blocks at the desired distances. The distance of each focal block is determined by depth information of three-dimensional imagery to provide near-correct focus cues. We evaluate the focus cue fidelity of the proposed system considering the fill factor and resolution of the backlight. Finally, we verify the display performance with experimental results.

Use of Commercial Virtual Reality Technology to Assess Verticality Perception in Static and Dynamic Visual Backgrounds.

The Subjective Visual Vertical (SVV) test and the closely related Rod and Disk Test (RDT) are measures of perceived verticality measured in static and dynamic visual backgrounds. However, the equipment used for these tests is variable across clinics and is often too expensive or too primitive to be appropriate for widespread use. Commercial virtual reality technology, which is now widely available, may provide a more suitable alternative for collecting these measures in clinical populations. This study was designed to investigate verticality perception in symptomatic patients using a modified RDT paradigm administered through a head-mounted display (HMD). A group of adult patients referred by a physician for vestibular testing based on the presence of dizziness symptoms and a group of healthy adults without dizziness symptoms were included. We investigated degree of visual dependence in both groups by measuring SVV as a function of kinematic changes to the visual background. When a dynamic background was introduced into the HMD to simulate the RDT, significantly greater shifts in SVV were found for the patient population than for the control population. In patients referred for vestibular testing, the SVV measured with the HMD was significantly correlated with traditional measures of SVV collected in a rotary chair when accounting for head tilt. This study provides initial proof of concept evidence that reliable SVV measures in static and dynamic visual backgrounds can be obtained using a low-cost commercial HMD system. This initial evidence also suggests that this tool can distinguish individuals with dizziness symptomatology based on SVV performance in dynamic visual backgrounds.

Evaluating the effects of collaboration and competition in navigation tasks and spatial knowledge acquisition within virtual reality environments

Current virtual reality environments, due to their higher viewing resolution and degree of immersion, can help users to acquire spatial knowledge with more ease. In addition, they are considered suitable environments that can facilitate collaborative engagement. With recent advances, we have a new wave of virtual reality technologies in the form of portable head-mounted displays (HMD), like the HTC Vive and Oculus RIFT. In this research, we investigate the effects of collaboration and competition activities, that are carried out within HMD VR environments, on users’ navigation behavior and acquisition of spatial knowledge. To do this investigation, we have developed a 3D virtual shopping mall with 3 floors and a number of shops. This mall resembles one that people would normally visit in real life. We then have conducted an experiment with two groups of paired users and one group of single users. Their task is twofold: (1) to navigate within the virtual mall and collect a set of items placed throughout the mall; and (2) to remember the locations of the items and several designated shops. While they have to do this in the shortest time possible, one group of paired users have to work in collaboration with each other, and the other group in competition. Paired users in both groups are physically co-located and are allowed to talk to each other. The third group of users have to work alone. After the experiment, each user is requested to pinpoint the location of the items and shops on a map with the layout of the virtual mall. The results show different patterns of user behavior for the two paired groups and single users. In addition, they indicate gender differences in behavior and performance. Overall, the results help us understand the effects of competition and cooperation in navigation behavior and spatial memory recall using commercial HMD VR systems.

Photosensor-Based Latency Measurement System for Head-Mounted Displays.

In this paper, a photosensor-based latency measurement system for head-mounted displays (HMDs) is proposed. The motion-to-photon latency is the greatest reason for motion sickness and dizziness felt by users when wearing an HMD system. Therefore, a measurement system is required to accurately measure and analyze the latency to reduce these problems. The existing measurement system does not consider the actual physical movement in humans, and its accuracy is also very low. However, the proposed system considers the physical head movement and is highly accurate. Specifically, it consists of a head position model-based rotary platform, pixel luminance change detector, and signal analysis and calculation modules. Using these modules, the proposed system can exactly measure the latency, which is the time difference between the physical movement for a user and the luminance change of an output image. In the experiment using a commercial HMD, the latency was measured to be up to 47.05 ms. In addition, the measured latency increased up to 381.17 ms when increasing the rendering workload in the HMD.

Evaluating Head Mounted Display Efficiency by User Study

Head Mounted Display (HMD) has been widely used in the area of wearable computing, augmented/virtual reality etc. The different image sources and optical subsystems of HMDs result in the diverse comprehensive display properties and optical properties. The traditional method to evaluate a HMD is modular transfer function. It requires special instruments, software and professional backgrounds knowledge, and thus leads to the lack of usability when it applied by common users. On the basis of human transfer function and Fitts’ model, in this paper we proposed an evaluation method, which has the advantage of easy applying. We evaluated this method targeting at a commercial HMD and a custom HMD, and the result shows that our method has a high usability.

Super Wide Field of View Head Mounted Display Using Catadioptrical Optics

Many virtual reality, mixed reality, and telepresence applications use head mounted displays (HMD). HMD systems are portable and can display stereoscopic images. However, the field of view (FOV) of commercial HMD systems is too narrow for conveying the feeling of immersion. The horizontal FOV is typically around 60°, significantly narrower than that of the human eye. In this paper, we propose new display optics for a super wide FOV head mounted display. The proposed optics consists of an ellipsoidal and a hyperboloidal mirror that will display distortionless images by using the characteristics of the mirrors, even if the image has a large FOV. We constructed a prototype HMD system with a 180° horizontal × 60° vertical FOV that includes the peripheral vision of the human eye. The FOV has a 60° × 60° overlap area that can display stereoscopic images. We estimated the resolution, focus, and aberration of the prototype in an optical simulation and experimentally confirmed that the prototype displays distortionless wide FOV images.

Super wide viewer using catadioptrical optics

Many applications have used a head-mounted display (HMD), such as in virtual and mixed realities and telepresence. However, the field of view (FOV) of commercial HMD systems is too narrow for feeling immersion. In this paper, we propose a super-wide field of view head-mounted display consisting of an ellipsoidal mirror and a hyperboloidal curved mirror. The horizontal FOV of the proposed HMD is 180 degrees and includes the peripheral vision of humans. It increases the reality and immersion for users.

2A1-C12 屈折反射光学系を用いた広視野角ヘッドマウントディスプレイ

Many applications have used a Head-Mounted Display (HMD), such as in virtual and mixed realities, and tele-presence. The advantage of HMD systems is the ease of feeling a 3D world in the display of animation or movies. However, the field of view (FOV) of commercial HMD systems is too narrow for feeling immersion. The horizontal FOV of many commercial HMDs is around 60 degrees, significantly narrower than that of humans. In this paper, we propose a super wide field of view catadioptrical head-mounted display consisting of an ellipsoidal and a hyperboloidal curved mirror. The horizontal FOV of the proposed HMD is 180 degrees and includes the peripheral view of humans. It increases reality and immersion of users. As well, the central region (60 degrees) of the FOV can measure 3D distances using stereoscopics.