Immersive Virtual Reality Environment Research Articles

Post-processing integration and semi-automated analysis of eye-tracking and motion-capture data obtained in immersive virtual reality environments to measure visuomotor integration.

Mobile eye-tracking and motion-capture techniques yield rich, precisely quantifiable data that can inform our understanding of the relationship between visual and motor processes during task performance. However, these systems are rarely used in combination, in part because of the significant time and human resources required for post-processing and analysis. Recent advances in computer vision have opened the door for more efficient processing and analysis solutions. We developed a post-processing pipeline to integrate mobile eye-tracking and full-body motion-capture data. These systems were used simultaneously to measure visuomotor integration in an immersive virtual environment. Our approach enables calculation of a 3D gaze vector that can be mapped to the participant's body position and objects in the virtual environment using a uniform coordinate system. This approach is generalizable to other configurations, and enables more efficient analysis of eye, head, and body movements together during visuomotor tasks administered in controlled, repeatable environments.

Exploring the Effects of Immersive Virtual Reality Environments on Sensory Perception of Beef Steaks and Chocolate.

Virtual reality (VR) technology is emerging as a tool for simulating different eating environments to better understand consumer sensory response to food. This research explored the impact of different environmental contexts on participants’ hedonic ratings of two different food products: beef steaks, and milk chocolate, using VR as the context-enhancing technology. Two separate studies were conducted. For beef, two different contextual conditions were compared: traditional sensory booths and a VR restaurant. For chocolate, data were generated under three different contextual conditions: traditional sensory booths, VR Irish countryside; VR busy city (Dublin, Ireland). All VR experiences were 360-degree video based. Consumer level of engagement in the different contextual settings was also investigated. The results showed that VR had a significant effect on participants’ hedonic responses to the food products. Beef was rated significantly higher in terms of liking for all sensory attributes when consumed in the VR restaurant. While for chocolate, the VR countryside context generated significantly higher hedonic scores for flavour and overall liking in comparison to the sensory booth. Taken together, both studies demonstrate how specific contextual settings can impact participants’ sensory response to food products, when compared to a traditional sensory laboratory condition.

‘And then again, I draw myself to the detail’

This article illustrates the creative process and conceptual impetus for the development of the piece Expanded Fields; a moving image installation with live performance, sound and virtual reality which was first presented in the Limerick City Gallery of Art in November 2019. The work highlights multiple filters through which the audience can encounter a piece of choreography, using a range of different media to invite the viewer into various degrees of proximity with the dancers in movement. Co-authors Ruth Gibson and Jenny Roche contextualise this collaboration undertaken with artist Bruno Martelli, composer Mel Mercier and the dancers, Kévin Coquelard, Henry Montes and Ursula Robb and articulate the application of process tools utilised to create the work. They draw on contributions from these collaborators on their experience of the making process as well as through materials produced by all involved in the development of the piece. The process tools included Micro-phenomenological Interviewing (MPI), movement initiated writing, Skinner Releasing Technique (SRT), motion capture, filmmaking and audio recordings. These methods were interwoven to reveal the tapestry of experiential layers that can be accessed when dancing or viewing a piece of dance. Placing them together within a gallery installation space and grounding these perspectives through episodic live performance, allowed the material to be encountered through various sensorial frames, from the immersive VR environment where the avatars of the dancers moved close to the viewer’s body, to the foyer space where footage of the dancers from studio rehearsals could be viewed on tv screens. The article situates this creative work in relation to writing by theorists such as Teresa Brennan on affect and André Lepecki on Derrida’s notion of the trace, alongside articulating potential alignments between SRT and MPI.

Development of a Novel Omnidirectional Treadmill-Based Locomotion Interface Device with Running Capability

To achieve an immersive virtual reality (VR) environment, omnidirectional treadmills (ODTs) allow users to perform locomotion in any direction. However, existing ODTs are heavy and complex, and operate at low speeds. This limits fast user motion and prevents natural interactions in real applications such as military training programs and interactive games. In this paper, we introduce a novel locomotion interface device with running capability, which uses an omnidirectional treadmill with a new power transmission mechanism and a locomotion controller that enables the user to make fast movements. As a result of the improved power transmission performance due to the simple and relatively lightweight structure, the proposed two-dimensional treadmill can generate a maximum speed of 3 m/s, with an acceleration of 3 m/s2. Moreover, through a pilot test with the proposed locomotion interface device, we verified that the fast directional changes during walking and running with the designed speed adaptation controller do not exceed the acceleration performance of the proposed system. Due to its wide range of movement speeds and acceleration capabilities, and lack of any motion constraints, the proposed locomotion interface device with a novel ODT can be used as a representative platform in various VR environments to enhance the immersive experience.

Effect of Cognitive Training in Fully Immersive Virtual Reality on Visuospatial Function and Frontal-Occipital Functional Connectivity in Predementia: Randomized Controlled Trial.

BackgroundCognitive training can potentially prevent cognitive decline. However, the results of recent studies using semi-immersive virtual reality (VR)-assisted cognitive training are inconsistent.ObjectiveWe aimed to examine the hypothesis that cognitive training using fully immersive VR, which may facilitate visuospatial processes, could improve visuospatial functioning, comprehensive neuropsychological functioning, psychiatric symptoms, and functional connectivity in the visual brain network in predementia.MethodsParticipants over 60 years old with subjective cognitive decline or mild cognitive impairment from a memory clinic were randomly allocated to the VR (n=23) or the control (n=18) group. The VR group participants received multidomain and neuropsychologist-assisted cognitive training in a fully immersive VR environment twice a week for 1 month. The control group participants did not undergo any additional intervention except for their usual therapy such as pharmacotherapy. Participants of both groups were evaluated for cognitive function using face-to-face comprehensive neuropsychological tests, including the Rey-Osterrieth Complex Figure Test (RCFT) copy task; for psychiatric symptoms such as depression, apathy, affect, and quality of life; as well as resting-state functional magnetic resonance imaging (rsfMRI) at baseline and after training. Repeated-measures analysis of variance was used to compare the effect of cognitive training between groups. Seed-to-voxel–based analyses were used to identify the cognitive improvement–related functional connectivity in the visual network of the brain.ResultsAfter VR cognitive training, significant improvement was found in the total score (F1,39=14.69, P=.001) and basic components score of the RCFT copy task (F1,39=9.27, P=.005) compared with those of the control group. The VR group also showed improvements, albeit not significant, in naming ability (F1,39=3.55, P=.07), verbal memory delayed recall (F1,39=3.03, P=.09), and phonemic fluency (F1,39=3.08, P=.09). Improvements in psychiatric symptoms such as apathy (F1,39=7.02, P=.01), affect (F1,39=14.40, P=.001 for positive affect; F1,39=4.23, P=.047 for negative affect), and quality of life (F1,39=4.49, P=.04) were found in the VR group compared to the control group. Improvement in the RCFT copy task was associated with a frontal-occipital functional connectivity increase revealed by rsfMRI in the VR group compared to the control group.ConclusionsFully immersive VR cognitive training had positive effects on the visuospatial function, apathy, affect, quality of life, and increased frontal-occipital functional connectivity in older people in a predementia state. Future trials using VR cognitive training with larger sample sizes and more sophisticated designs over a longer duration may reveal greater improvements in cognition, psychiatric symptoms, and brain functional connectivity.Trial RegistrationClinical Research Information Service KCT0005243; https://tinyurl.com/2a4kfasa

Kinematic of the Position and Orientation Synchronization of the Posture of a n DoF Upper-Limb Exoskeleton with a Virtual Object in an Immersive Virtual Reality Environment

Exoskeletons are an external structural mechanism with joints and links that work in tandem with the user, which increases, reinforces, or restores human performance. Virtual Reality can be used to produce environments, in which the intensity of practice and feedback on performance can be manipulated to provide tailored motor training. Will it be possible to combine both technologies and have them synchronized to reach better performance? This paper consists of the kinematics analysis for the position and orientation synchronization between an n DoF upper-limb exoskeleton pose and a projected object in an immersive virtual reality environment using a VR headset. To achieve this goal, the exoskeletal mechanism is analyzed using Euler angles and the Pieper technique to obtain the equations that lead to its orientation, forward, and inverse kinematic models. This paper extends the author’s previous work by using an early stage upper-limb exoskeleton prototype for the synchronization process.

Combining cues to judge distance and direction in an immersive virtual reality environment.

When we move, the visual direction of objects in the environment can change substantially. Compared with our understanding of depth perception, the problem the visual system faces in computing this change is relatively poorly understood. Here, we tested the extent to which participants’ judgments of visual direction could be predicted by standard cue combination rules. Participants were tested in virtual reality using a head-mounted display. In a simulated room, they judged the position of an object at one location, before walking to another location in the room and judging, in a second interval, whether an object was at the expected visual direction of the first. By manipulating the scale of the room across intervals, which was subjectively invisible to observers, we put two classes of cue into conflict, one that depends only on visual information and one that uses proprioceptive information to scale any reconstruction of the scene. We find that the sensitivity to changes in one class of cue while keeping the other constant provides a good prediction of performance when both cues vary, consistent with the standard cue combination framework. Nevertheless, by comparing judgments of visual direction with those of distance, we show that judgments of visual direction and distance are mutually inconsistent. We discuss why there is no need for any contradiction between these two conclusions.

FixationNet: Forecasting Eye Fixations in Task-Oriented Virtual Environments.

Human visual attention in immersive virtual reality (VR) is key for many important applications, such as content design, gaze-contingent rendering, or gaze-based interaction. However, prior works typically focused on free-viewing conditions that have limited relevance for practical applications. We first collect eye tracking data of 27 participants performing a visual search task in four immersive VR environments. Based on this dataset, we provide a comprehensive analysis of the collected data and reveal correlations between users' eye fixations and other factors, i.e. users' historical gaze positions, task-related objects, saliency information of the VR content, and users' head rotation velocities. Based on this analysis, we propose FixationNet - a novel learning-based model to forecast users' eye fixations in the near future in VR. We evaluate the performance of our model for free-viewing and task-oriented settings and show that it outperforms the state of the art by a large margin of 19.8% (from a mean error of 2.93° to 2.35°) in free-viewing and of 15.1% (from 2.05° to 1.74°) in task-oriented situations. As such, our work provides new insights into task-oriented attention in virtual environments and guides future work on this important topic in VR research.

Cognitive Modifiability in 3D-IVR and 2D Computerized Environments: The Effects of Rotation of Information Resources and Shift of Viewing Angles

Research findings indicate that cognitive achievements are significantly improved by practice of cognitive abilities in the 3D Immersive Virtual Reality (3D-IVR) environment. The current study focuses on the effects of two spatial characteristics of the computer environment, Rotation of Information Resources (RIR) and Shift of Viewing Angles (SVA), on cognitive modifiability as measured in a dynamic assessment (DA) procedure. The DA was composed of modified versions of the Analogies Subtest (AN) from the Cognitive Modifiability Battery (CMB) adapted for the computerized environment and includes pre-teaching, teaching, and post-teaching phases. The analogies contain dimensions of color, number, height, and position. In the teaching phase, children mediated various problem-solving strategies. The sample was composed of children in Grades 1 and 2 (n = 73). They were randomly assigned to either 3D-IVR or 2D conditions. Higher frequency of use of SVA contributed significantly to pre- to post-teaching improvement of analogical thinking. Higher improvements were found in dimensions of height and position than in color and number. The dimensions of height and position are specifically connected to spatial perception, hence the higher improvement. The findings are explained in relation to the importance of the use of SVA and RIR as crucial spatial characteristics for developing cognitive maps formation, and cognitive performance.

Simulation-based assessments of fire emergency preparedness and response in virtual reality

The current study aimed at evaluating the prospects of a three-dimensional gas power plant (GPP) simulation in an immersive virtual reality (IVR) environment for fire emergency preparedness and response (EPR). To achieve this aim, the study assessed the possibility of safety situational awareness, evacuation drills and hazard mitigation exercises during a fire emergency simulation scenario. The study likewise evaluated the safety and ergonomics of the environment while addressing this aim. We employed the virtual reality accident causation model (VR-ACM) for the assessment with 54 participants individually in IVR. Participants were grouped into two according to whether they had work experience in engineering or not. The obtained results suggested that IVR can be realistic and safe, with the potential for presenting hazardous scenarios necessary for fire EPR. Furthermore, the results indicated that there were no statistically significant differences in the perceptions of both groups regarding the prospects of IVR towards EPR.

EEG Cortical Activities and Networks Altered by Watching 2D/3D Virtual Reality Videos

Abstract. Virtual reality (VR), which can represent real-life events and situations, is being increasingly applied to many fields, such as education, entertainment, and medical rehabilitation. Correspondingly, the neural information processing of VR has attracted attention. However, the underlying neural mechanisms of VR environments have not yet been fully revealed. The purpose of this study was to examine the possible differences in brain activities and networks between the less immersive 2D and the fully immersive 3D VR environments. 3D VR videos and the same 2D scenes were presented to the participants and the scalp electroencephalogram (EEG) was recorded, respectively. Power spectral density (PSD) and the functional connectivity of these EEG signals were analyzed. The results showed that 3D VR videos significantly enhanced the PSD of θ rhythm (4–7 Hz) in the frontal lobe; decreased the PSD of α rhythm (8–13 Hz) in the parietal and the occipital lobes; increased the PSD of β rhythm (14–30 Hz) in the frontal, the parietal, the temporal, and the occipital lobes, relative to 2D VR watching. Furthermore, 3D versus 2D VR-induced alterations in the patterns of brain networks were similar to the patterns of PSD. Specifically, for the θ rhythm, 3D VR significantly enhanced the frontal and the temporal brain functional connectivity; for the α rhythm, 3D VR increased the parietal and the occipital networks; for the β rhythm, 3D VR remarkably increased the frontal, the occipital, the frontal-temporal and the frontal-occipital brain functional connectivity, relative to 2D VR. These significant differences between 3D and 2D VR video-watching suggest that the neural information processing of cortical activities and networks is correlated to the degree of immersion. The present results, collected with previous researches, implicate that some visual-related information processes, such as visual attention, visual perception, and visual immersion are more robust in 3D VR environments.

User capabilities in eyes-free spatial target acquisition in immersive virtual reality environments

In immersive virtual reality (VR) environments, users rely on the vision channel to search for objects. Such eyes-engaged interactive techniques may significantly degrade the interaction efficiency and user experience, particularly when users have to turn their head frequently to search for a target object in the limited field of view (FOV) of a head-mounted display (HMD). In this study, we systematically investigated user capabilities in eyes-free spatial target acquisition considering different horizontal angles, vertical angles, distances from the user's body, and body sides. Our results show that high acquisition accuracy and low task load are achieved for target locations at front and middle horizontal angles as well as those at middle vertical angles. Meanwhile, a trade-off cannot be achieved between the acquisition accuracy and the task load for target locations at long distances from the user's body. In addition, the acquisition accuracy and task load for the target locations vary with the body side. Our research findings can provide a deeper understanding of user capability in eyes-free target acquisition and offer concrete design guidelines for appropriate target arrangement for eyes-free target acquisition in immersive VR environments.

The effect of attentional focus on movement accuracy in an immersive and interactive virtual reality environment

The effects of attentional focus (AF) instruction were examined in a reciprocal aiming task implemented in a 3-dimensional, fully immersive virtual environment (VE). Within the VE, participants (N = 19) moved a cube between two targets at two paces (750 ms and 500 ms) while being asked to focus externally (EXF) and internally (INF). Performance accuracy was measured as two-dimensional error and its variability between the center of the target to the center of the cube and one-dimensional bias (undershooting/overshooting behavior in the anteroposterior and mediolateral directions). The results indicated better performance, specifically greater accuracy and lower one-dimensional bias in the anteroposterior direction when adopting an EXF compared to an INF. Our findings reveal that the beneficial effects of an EXF on motor performance are not restricted movements within physical environments only but also work in VE. This has implications for rehabilitation and training protocols in VE.

Effects of exposure to immersive videos and photo slideshows of forest and urban environments

A large number of studies have demonstrated the benefits of natural environments on people’s health and well-being. For people who have limited access to nature (e.g., elderly in nursing homes, hospital patients, or jail inmates), virtual representations may provide an alternative to benefit from the illusion of a natural environment. For this purpose and in most previous studies, conventional photos of nature have been used. Immersive virtual reality (VR) environments, however, can induce a higher sense of presence compared to conventional photos. Whether this higher sense of presence leads to increased positive impacts of virtual nature exposure is the main research question of this study. Therefore, we compared exposure to a forest and an urban virtual environment in terms of their respective impact on mood, stress, physiological reactions, and cognition. The environments were presented via a head-mounted display as (1) conventional photo slideshows or (2) 360^{circ } videos. The results show that the forest environment had a positive effect on cognition and the urban environment disturbed mood regardless of the mode of presentation. In addition, photos of either urban or forest environment were both more effective in reducing physiological arousal compared to immersive 360^{circ } videos.

Spatial Skills and Perceptions of Space: Representing 2D Drawings as 3D Drawings inside Immersive Virtual Reality

Rapid freehand drawings are of great importance in the early years of university studies of architecture, because both the physical characteristics of spaces and their sensory characteristics can be communicated through them. In order to draw architectural spaces, it is necessary to have the ability to visualize and manipulate them mentally, which leads us to the concept of spatial skills; but it also requires a development of spatial perception to express them in the drawings. The purpose of this research is to analyze the improvement of spatial skills through the full-scale sketching of architectural spaces in virtual immersive environments and to analyze spatial perception in reference to the capture of spatial sensations in virtual immersive environments. Spatial skills training was created based on the freehand drawing of architectural spaces using Head Mounted Displays (HMD) and registered the spatial sensations experienced also using HMD, but only in previously modeled realistic spaces. It was found that the training significantly improved orientation, rotation and visualization, and that the sensory journey and experimentation of architectural spaces realistically modeled in immersive virtual reality environments allows for the same sensations that the designer initially sought to convey.

IMHOTEP: cross-professional evaluation of a three-dimensional virtual reality system for interactive surgical operation planning, tumor board discussion and immersive training for complex liver surgery in a head-mounted display

BackgroundVirtual reality (VR) with head-mounted displays (HMD) may improve medical training and patient care by improving display and integration of different types of information. The aim of this study was to evaluate among different healthcare professions the potential of an interactive and immersive VR environment for liver surgery that integrates all relevant patient data from different sources needed for planning and training of procedures.Methods3D-models of the liver, other abdominal organs, vessels, and tumors of a sample patient with multiple hepatic masses were created. 3D-models, clinical patient data, and other imaging data were visualized in a dedicated VR environment with an HMD (IMHOTEP). Users could interact with the data using head movements and a computer mouse. Structures of interest could be selected and viewed individually or grouped. IMHOTEP was evaluated in the context of preoperative planning and training of liver surgery and for the potential of broader surgical application. A standardized questionnaire was voluntarily answered by four groups (students, nurses, resident and attending surgeons).ResultsIn the evaluation by 158 participants (57 medical students, 35 resident surgeons, 13 attending surgeons and 53 nurses), 89.9% found the VR system agreeable to work with. Participants generally agreed that complex cases in particular could be assessed better (94.3%) and faster (84.8%) with VR than with traditional 2D display methods. The highest potential was seen in student training (87.3%), resident training (84.6%), and clinical routine use (80.3%). Least potential was seen in nursing training (54.8%).ConclusionsThe present study demonstrates that using VR with HMD to integrate all available patient data for the preoperative planning of hepatic resections is a viable concept. VR with HMD promises great potential to improve medical training and operation planning and thereby to achieve improvement in patient care.

Interactive 3D Modeling with Virtual Reality

The immersive Virtual Reality (VR) environment transforms the way people learn, work, and communicate. While the traditional modeling tools such as Blender and AutoCAD are commonly used for industrial design today, handling 3D objects on a two-dimensional screen with a keyboard and mouse is very challenging. In this work, we introduce a VR modeling system named The Virtual Workshop supporting the design and manipulation of various 3D objects in virtual environments. The proposed system was developed for the Oculus Rift platform allowing a user to efficiently and precisely design 3D objects using two hands directly. The friendly system GUI supports the user to create new 3D objects from scratch using premade “basic objects” or alternatively to import an existing 3D object made in other applications. Meanwhile, the finished 3D models are stored in the standard OBJ file format and exported later for the development of 3D scenarios in other applications such as the Unity engine. From concept to design, the VR modeling system provides an open platform where the designers and the clients can better share their ideas as well as interactively refine the rendered virtual models in collaboration.

Exposure to an Immersive Virtual Reality Environment can Modulate Perceptual Correlates of Endogenous Analgesia and Central Sensitization in Healthy Volunteers

Virtual reality (VR) has been shown to produce analgesic effects during different experimental and clinical pain states. Despite this, the top-down mechanisms are still poorly understood. In this study, we examined the influence of both a real and sham (ie, the same images in 2D) immersive arctic VR environment on conditioned pain modulation (CPM) and in a human surrogate model of central sensitization in 38 healthy volunteers. CPM and acute heat pain thresholds were assessed before and during VR/sham exposure in the absence of any sensitization. In a follow-on study, we used the cutaneous high frequency stimulation model of central sensitization and measured changes in mechanical pain sensitivity in an area of heterotopic sensitization before and during VR/sham exposure. There was an increase in CPM efficiency during the VR condition compared to baseline (P < .01). In the sham condition, there was a decrease in CPM efficiency compared to baseline (P < .01) and the real VR condition (P < .001). Neither real nor sham VR had any effect on pain ratings reported during the conditioning period or on heat pain threshold. There was also an attenuation of mechanical pain sensitivity during the VR condition indicating a lower sensitivity compared to sham (P < .05). We conclude that exposure to an immersive VR environment has no effect over acute pain thresholds but can modulate dynamic CPM responses and mechanical hypersensitivity in healthy volunteers. PerspectiveThis study has demonstrated that exposure to an immersive virtual reality environment can modulate perceptual correlates of endogenous pain modulation and secondary hyperalgesia in a human surrogate pain model. These results suggest that virtual reality could provide a novel mechanism-driven analgesic strategy in patients with altered central pain processing.

A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke.

Given the rising incidence of stroke, several technology-driven methods for rehabilitation have recently been developed. Virtual reality (VR) is a promising therapeutic technology among them. We recently developed a neuroscientifically grounded VR system to aid recovery of motor function poststroke. The developed system provides unilateral and bilateral upper extremity (UE) training in a fully immersive virtual environment that may stimulate and activate mirror neurons (MNs) in the brain necessary for UE rehabilitation. Twenty-three participants were randomized to a VR group (n=12) to receive VR intervention (8 h within 2 weeks) plus 8-h occupational therapy (OT) or a control group (n=11) to receive time-matched OT alone. Treatment effects on motor recovery and cortical reorganization were investigated using the Barthel Index (BI), Fugl-Meyer Upper Extremity (FM-UE), and resting-state fMRI. Both groups significantly improved BI (P < 0.05), reflecting the recovery of UE motor function. The VR group revealed significant improvements on FM-UE scores (P < 0.05) than the control group. Neural activity increased after the intervention, particularly in the brain areas implicating MNs, such as in the primary motor cortex. Overall, results suggested that using a neuroscientifically grounded VR system might offer additional benefits for UE rehabilitation in patients receiving OT.

Perceptual Quality Assessment of Omnidirectional Images as Moving Camera Videos.

Omnidirectional images (also referred to as static 360 ° panoramas) impose viewing conditions much different from those of regular 2D images. How do humans perceive image distortions in immersive virtual reality (VR) environments is an important problem which receives less attention. We argue that, apart from the distorted panorama itself, two types of VR viewing conditions are crucial in determining the viewing behaviors of users and the perceived quality of the panorama: the starting point and the exploration time. We first carry out a psychophysical experiment to investigate the interplay among the VR viewing conditions, the user viewing behaviors, and the perceived quality of 360 ° images. Then, we provide a thorough analysis of the collected human data, leading to several interesting findings. Moreover, we propose a computational framework for objective quality assessment of 360 ° images, embodying viewing conditions and behaviors in a delightful way. Specifically, we first transform an omnidirectional image to several video representations using different user viewing behaviors under different viewing conditions. We then leverage advanced 2D full-reference video quality models to compute the perceived quality. We construct a set of specific quality measures within the proposed framework, and demonstrate their promises on three VR quality databases.