Interactive Virtual Reality Environment Research Articles

Immersive presentations of real-world medical equipment through interactive VR environment populated with the high-fidelity 3D model of mobile MRI unit

The primary goal behind the system presented in this paper is to investigate the efficacy of using virtual reality (VR) for showcasing sizable medical equipment. Specifically, we focused on a mobile magnetic resonance imaging (MRI) scanner mounted on a truck trailer. The latter is integral to the mobile MRI setup and must be presented as part of the immersive experience. Therefore, we not only have to depict the medical apparatus but also provide the means of understanding its surroundings. This is especially important to radiologists and other medical personnel to ascertain if a given mobile medical facility fulfills their needs and wants. Furthermore, despite such MRI devices being designed for mobility, their long-distance transportation can be time-consuming, troublesome and expensive. Therefore, we can observe the need for showcasing such mobile MRI units without additional cost and burden related to transportation. To achieve this, we designed an immersive environment in which the users can interact with the real-life scale 3D model of a mobile MRI. In addition, we also verified the usability and expressiveness of our system using established heuristical approaches.

Human-computer interaction in virtual reality environments for educational and business purposes

This paper examines the integration of Human-Computer Interaction (HCI) in Virtual Reality (VR) environments, with a focus on its application in educational and business domains. It explores the transformative potential of VR in enhancing user interface design, optimizing user experience, and integrating advanced technologies like natural language processing and gesture recognition. The research highlights VR's efficacy in improving learning outcomes and operational efficiencies, demonstrating significant advancements in immersive and intuitive interaction within virtual settings. The study underscores the critical role of user-centered design in VR systems and the impact of integrating natural language processing and gesture recognition technologies in creating more natural and intuitive user interactions. Through comprehensive analysis and case studies, the paper presents the latest developments in VR technology and its application, offering insights into the future of immersive digital environments in education and business sectors.

Thelxinoë: Recognizing Human Emotions Using Pupillometry and Machine Learning

In this study, we present a method for emotion recognition in Virtual Reality (VR) using pupillometry. We analyze pupil diameter responses to both visual and auditory stimuli via a VR headset and focus on extracting key features in the time-domain, frequency-domain, and time-frequency domain from VR-generated data. Our approach utilizes feature selection to identify the most impactful features using Maximum Relevance Minimum Redundancy (mRMR). By applying a Gradient Boosting model, an ensemble learning technique using stacked decision trees, we achieve an accuracy of 98.8% with feature engineering, compared to 84.9% without it. This research contributes significantly to the Thelxinoë framework, aiming to enhance VR experiences by integrating multiple sensor data for realistic and emotionally resonant touch interactions. Our findings open new avenues for developing more immersive and interactive VR environments, paving the way for future advancements in virtual touch technology.

Pilgrimage to Pureland: Art, Perception and the Wutai Mural VR Reconstruction

Virtual reality (VR) supports audiences to engage with cultural heritage proactively. We designed an easy-to-access and guided VR reconstruction of the murals in Mogao Grottoes, Pilgrimage To Pureland, to offer the general public an accessible and engaging way to experience the Dunhuang murals. We explored an immersive VR reconstruction approach that can efficiently convert complex 2D mural art into 3D digital VR content while maintaining its originality. Specifically, we reconstructed the Mt. Wutai pilgrimage mural in Cave 61 into a plot-based interactive VR environment that offers users a more accessible and immersive narrative to visit and appreciate Dunhuang murals’ complex religious, historical, and artistic value. We conducted a user study with twenty participants to further investigate how this reconstruction approach and the interactive narratives would affect users’ immersive experiences and knowledge acquisition, compared with other forms of virtual access as the control group. Findings from our user study revealed that participants from the VR experimental group reported a significantly higher level of immersion and understanding of the knowledge. To conclude, our 2D-to-VR reconstruction methods provided a VR reconstruction approach and insights for creating engaging user experiences and improving users understanding of cultural and historical concepts while preserving the originality of the cultural heritage.

Various Types of Virtual Reality-Based Therapy for Eating Disorders: A Systematic Review.

(1) Background: Interactive VR (virtual reality) environments (i.e., using three-dimensional graphics presented with a head-mounted display) have recently become a popular professional tool for the treatment of patients with eating disorders (EDs). However, there are no published review reports that have analyzed the original papers between 2015 and 2021, which additionally focused only on HMD (head-mounted display) 3DVR (three-dimensional virtual reality) exposure and included only three therapeutic categories for ED patients. (2) Methods: The EbscoHost and Scopus databases were searched to identify relevant papers on VR research employing VR in the assessment and treatment of eating disorders. (3) Results: In addition to the known therapeutic divisions for ED, there are new forms of therapy based on 360 cameras, eye-tracking, and remote therapy. (4) Conclusions: The potential of VR in combination with different therapies may offer an alternative for future research. More rigorous testing, especially in terms of larger sample sizes, the inclusion of control groups or multisessions, and follow-up measures, is still needed. The current state of research highlights the importance of the nature and content of VR interventions for ED patients. Future research should look to incorporate more home-based and remote forms of VR tools.

Co-Designing a Virtual Reality Application to Enhance Reminiscence Therapy for Persons with Dementia

Reminiscence therapy for persons with dementia is often being conducted by employing analog media including pictures and videos organized and presented by a caregiver. However, such media lacks the immersive experience to support patient engagement and successful recollection of reminiscence events. Recently, Virtual Reality (VR) is gaining momentum as a potential technological tool to support dementia care due to its increased immersion, presence, and embodiment. Haptic artifacts can be used to enrich reminiscence therapy as part of the multi-sensory stimuli to increase immersion and patient engagement, as well as improving social connectedness and cognitive health. The purpose of this project is to explore the use of VR application to advance reminiscence therapy for persons with dementia. We have prototyped an immersive and non-immersive VR framework that allows caregivers to deliver reminiscence therapy for persons with dementia with varying stages in their disease progression. These reminiscence therapy sessions are built by employing a narrative storyboard and content management through a series of co-designing sessions with content experts at the Geriatric Dementia Unit in Ontario, Canada. A caregiver-led VR framework will be adopted to enable the caregiver to guide the persons with dementia to safely navigate through the interactive VR environment, while allowing the patients to engage with the interactive VR elements using a point-and-pinch gesture approach. We anticipate that the VR experiences hold the potential for improving the interactions between persons with dementia and caregivers, as well as enhancing the reminiscence experiences to promote the maximal therapeutic benefit of patient’s recovery.

4D modelling using virtual collaborative planning and scheduling

During the last decades, the introduction of Building Information Models (BIM) has opened new possibilities to ensure better communication and a shared understanding between stakeholders in construction projects. Similar benefits have been found in 4D simulations of the schedule. While BIM and 4D use has seen a steady increase, the potential benefits have not fully been realised, primarily due to tools not supporting and enabling the full potential of collaborative stakeholder involvement in scheduling. The benefits of 4D simulations come from improving communication between stakeholders. While traditional 4D modelling connects an existing BIM model with an existing schedule, a move from this type of passive 4D visualisation toward social co-creation enables all stakeholders to be involved in the scheduling process. This connection is further enhanced with the use of Virtual Reality (VR). Recent research has shown VR to enhance understanding and perception of the space and details and, thus, a better understanding of the project. This paper maps collaborative planning and scheduling method and traditional 4D scheduling using process modelling. This mapping is followed by implementing the 4D collaborative planning and scheduling method in the virtual project planning system with support for a multi-user interactive VR environment. The system enables social co-creation both during the planning and scheduling and during 4D modelling; it improves understanding of the project and the planning and scheduling.

CellexalVR: A virtual reality platform to visualize and analyze single-cell omics data

SummarySingle-cell RNAseq is a routinely used method to explore heterogeneity within cell populations. Data from these experiments are often visualized using dimension reduction methods such as UMAP and tSNE, where each cell is projected in two or three dimensional space. Three-dimensional projections can be more informative for larger and complex datasets because they are less prone to merging and flattening similar cell-types/clusters together. However, visualizing and cross-comparing 3D projections using current software on conventional flat-screen displays is far from optimal as they are still essentially 2D, and lack meaningful interaction between the user and the data. Here we present CellexalVR (www.cellexalvr.med.lu.se), a feature-rich, fully interactive virtual reality environment for the visualization and analysis of single-cell experiments that allows researchers to intuitively and collaboratively gain an understanding of their data.

Effect of visual elements on Indoor soundscape perception in open-plan office

This study examined the effect of changes in visual elements on spatial comfort and work productivity in the aspect of indoor soundscape perception in the open-plan office (OPO) sound environment. Various OPO visual stimuli were simulated using computer software (Unity 3D engine) to change the visual environment by varying variables such as worker density, window ratio, green ratio, and ceiling height. An interactive virtual reality environment was implemented to perform a specific task while experiencing the audio-visual stimuli combining the general OPO noise stimulus and the simulated OPO visual stimulus. Subjective evaluation was performed on a total of 30 subjects to evaluate indoor soundscape quality and work performance for each stimulus. Based on the results of this study, a pleasant OPO design guideline was proposed. Keywords: Open-plan office, indoor soundscape, interactive VR test, spatial comfort

The effect of attentional focus on movement accuracy in an immersive and interactive virtual reality environment

The effects of attentional focus (AF) instruction were examined in a reciprocal aiming task implemented in a 3-dimensional, fully immersive virtual environment (VE). Within the VE, participants (N = 19) moved a cube between two targets at two paces (750 ms and 500 ms) while being asked to focus externally (EXF) and internally (INF). Performance accuracy was measured as two-dimensional error and its variability between the center of the target to the center of the cube and one-dimensional bias (undershooting/overshooting behavior in the anteroposterior and mediolateral directions). The results indicated better performance, specifically greater accuracy and lower one-dimensional bias in the anteroposterior direction when adopting an EXF compared to an INF. Our findings reveal that the beneficial effects of an EXF on motor performance are not restricted movements within physical environments only but also work in VE. This has implications for rehabilitation and training protocols in VE.

Expert system for identifying and analyzing the IoT devices using Augmented Reality Approach

Biometrics In conjunction with the new development of the Internet of Things (IoT), augmented reality (AR) systems are evolving to visualize 3D virtual models of the real world into an intelligent and interactive virtual reality environment that facilitates physical identification of objects and defines their specifications efficiently. The integration between AR and IoT in a complementary way helps identify network-related items' specifications and interact with the Internet of Things more efficiently. An identity is a dedicated, publicly known attribute or set of names for an individual device. Typically, identifiers operate within a specific area or network, making it difficult to identify things globally. This paper explores the use of Augmented Reality (AR) Technology for identifying devices and displaying relevant information about the device to the user. Based on the developed model network, the developed system of identification of IoT devices was tested. Also, the traffic generated by the AR device when generating requests to the organization server was investigated. According to the test results, the system is undemanding to the main network indicators. The system-generated traffic is self-similar. The test results show that the server software can solve the problems of identifying IoT devices through interaction with augmented reality devices.

Developmental differences across the lifespan in the use of perceptual information to guide action-based decisions

Perceptual information about unfolding events is important for guiding decisions about when and how to move in real-world action situations. As an exemplary case, road-crossing is a perceptual-motor task where age has been shown to be a strong predictor of risk due to errors in action-based decisions. The present study investigated age differences between three age groups (Children: 10–12 years old; Adults: 19–39 years old; Older Adults: 65 + year olds) in the use of perceptual information for selection, timing, and control of action when crossing a two-way street in an immersive, interactive virtual reality environment. Adults and children selected gaps to cross that were consistent with the use of a time-based information variable (tau), whereas older adults tuned less into the time-based variable (tau) to guide road-crossing decisions. For action initiation and control, children and adults also showed a strong ability to precisely time their entry with respect to the lead vehicle maximising the available time to cross and coordinating walking movements with the tail vehicle to ensure they were not on a collision course. In contrast, older adults delayed action initiation and showed difficulty coordinating self-movement with the approaching vehicle. This study and its results tie together age-based differences in the three components of action decision-making (selection, timing and control) within a unified framework based on perceptual information. The implications of these age-related differences in action decisions and crossing behaviours are discussed in the context of road safety.

Virtual Reality for Monitor and Control of Electrical Substations.

This project presents the results obtained from a new strategy based on Virtual Reality techniques, which intends to minimize the operational issues caused in electric power substations due to the lack of spatial and functional information on the traditional operation interfaces. For this purpose, a three-dimensional interactive virtual reality environment was built in a realistic and accurate way regarding an energy supplier substation in Minas Gerais - Brazil and subsequently implanted it in its operation center for tasks related to its functioning. Lastly, tests were applied to operators to obtain results aiming at the contextualized problems.

A Comparative Usability Study of Bare Hand Three-Dimensional Object Selection Techniques in Virtual Environment

Object selection is the basis of natural user–computer interaction (NUI) in a virtual environment (VE). Among the three-dimensional object selection techniques employed in virtual reality (VR), bare hand-based finger clicking interaction and ray-casting are two convenient approaches with a high level of acceptance. This study involved 14 participants, constructed a virtual laboratory environment in VR, and compared the above two finger-based interaction techniques in terms of aspects of the task performance, including the success rate, total reaction time, operational deviation, and accuracy, at different spatial positions. The results indicated that the applicable distance range of finger clicking interaction and finger ray-casting was 0.2 to 1.4 m and over 0.4 m, respectively. Within the shared applicable distance, the finger clicking interaction achieved a shorter total reaction time and higher clicking accuracy. The performance of finger clicking interaction varied remarkably at the center and edge of the horizontal field of view, while no significant difference was found among ray-casting at various horizontal azimuths. The current findings could be directly applied to the application of bare-hand interaction in VR environments.

Production and use of multimedia speech content for perceptual experiments in virtual environments

Consumer virtual reality devices have become inexpensive and readily available, and offer high quality motion tracking, low latency, and sufficient resolution to conduct ecologically valid perceptual experiments. Unfortunately, high quality multimedia source material can be difficult to produce or obtain due to the specialized equipment and facilities required for capture. To facilitate a wide array of perceptual experiments in multi-modal speech perception, we have generated a multimedia corpus that includes stereoscopic and spherical videos, as well as anechoic audio. These materials can easily be placed into an interactive virtual reality environment and delivered over a head-mounted display to evaluate ventriloquism, lip-reading, spatial release from masking, or other perceptual effects that depend on audio-visual integration. The corpus replicates the Coordinate Response Measure sentences, as well as the Harvard IEEE corpus word list. Subjects were recorded at three different distances to preserve accurate binocular disparities in the videos and can be positioned at arbitrary azimuthal positions. The corpus and a simple virtual reality application for positioning and viewing the videos on a head-mounted display have been made publicly available online for download. We will detail the methods used to produce this content, as well as use of the accompanying viewing application.

Graphics and Virtual Environments for Serious Games

The articles in this special section focus on computer graphics and virtual environments for serious games. & SINCE THE SWIFT and dramatic emergence of serious games as a field of study in its own right, numerous challenges have been revealed. Serious games share most of their infrastructure with entertainment games and other interactive virtual and augmented reality (AR) environments and progressive development of games technologies and interactive techniques also improve serious games; however, the transfer of existing and new technologies and solutions to the younger application domain of serious games is not always straightforward, requiring creative solutions. For this Special Issue, we have accepted five articles that demonstrate the breadth of techniques for graphics and virtual environments for serious games. These articles are a mix of original submissions and of extended and revised versions of selected best papers presented at the VSGames 2019 conference.

The effects of different levels of realism on the training of CNNs with only synthetic images for the semantic segmentation of robotic instruments in a head phantom.

The manual generation of training data for the semantic segmentation of medical images using deep neural networks is a time-consuming and error-prone task. In this paper, we investigate the effect of different levels of realism on the training of deep neural networks for semantic segmentation of robotic instruments. An interactive virtual-reality environment was developed to generate synthetic images for robot-aided endoscopic surgery. In contrast with earlier works, we use physically based rendering for increased realism. Using a virtual reality simulator that replicates our robotic setup, three synthetic image databases with an increasing level of realism were generated: flat, basic, and realistic (using the physically-based rendering). Each of those databases was used to train 20 instances of a UNet-based semantic-segmentation deep-learning model. The networks trained with only synthetic images were evaluated on the segmentation of 160 endoscopic images of a phantom. The networks were compared using the Dwass-Steel-Critchlow-Fligner nonparametric test. Our results show that the levels of realism increased the mean intersection-over-union (mIoU) of the networks on endoscopic images of a phantom ([Formula: see text]). The median mIoU values were 0.235 for the flat dataset, 0.458 for the basic, and 0.729 for the realistic. All the networks trained with synthetic images outperformed naive classifiers. Moreover, in an ablation study, we show that the mIoU of physically based rendering is superior to texture mapping ([Formula: see text]) of the instrument (0.606), the background (0.685), and the background and instruments combined (0.672). Using physical-based rendering to generate synthetic images is an effective approach to improve the training of neural networks for the semantic segmentation of surgical instruments in endoscopic images. Our results show that this strategy can be an essential step in the broad applicability of deep neural networks in semantic segmentation tasks and help bridge the domain gap in machine learning.

Use of Virtual Reality Simulation to Identify Vision-Related Disability in Patients With Glaucoma

Clinical assessment of vision-related disability is hampered by the lack of instruments to assess visual performance in real-world situations. Interactive virtual reality (VR) environments displayed in a binocular stereoscopic VR headset have been designed, presumably simulating day-to-day activities to evaluate vision-related disability. To investigate the application of VR to identify vision-related disability in patients with glaucoma. In a cross-sectional study, 98 patients with glaucoma and 50 healthy individuals were consecutively recruited from a university eye clinic; all participants were Chinese. The study was conducted between August 30, 2016, and July 31, 2017; data analysis was performed from December 1, 2017, to October 30, 2018. Measurements of visual acuity, contrast sensitivity, visual field (VF), National Eye Institute 25-item Visual Function Questionnaire Rasch score, and VR disability scores determined from 5 VR simulations: supermarket shopping, stair and city navigations in daytime, and stair and city navigations in nighttime. Duration required to complete the simulation, number of items incorrectly identified, and number of collisions were measured to compute task-specific and overall VR disability scores. Vision-related disability was identified when the VR disability score was outside the normal age-adjusted 95% confidence region. Virtual reality disability score. In the 98 patients with glaucoma, mean (SD) age was 49.8 (11.6) years and 60 were men (61.2%); in the 50 healthy individuals, mean (SD) age was 48.3 (14.8) years and 16 were men (32.0%). The patients with glaucoma had different degrees of VF loss (122 eyes [62.2%] had moderate or advanced VF defects). The time required to complete the activities by patients with glaucoma vs healthy individuals was longer by 15.2 seconds (95% CI, 5.5-24.9 seconds) or 34.1% (95% CI, 12.4%-55.7%) for the shopping simulation, 72.8 seconds (95% CI, 23.0-122.6 seconds) or 33.8% (95% CI, 10.7%-56.9%) for the nighttime stair navigation, and 38.1 seconds (95% CI, 10.9-65.2 seconds) or 30.8% (95% CI, 8.8%-52.8%) for the nighttime city navigation. The mean (SD) duration was not significantly different between the glaucoma and healthy groups in daytime stair (203.7 [93.7] vs 192.9 [89.1] seconds, P = .52) and city (118.7 [41.5] vs 117.0 [52.3] seconds, P = .85) navigation. For each decibel decrease in binocular VF sensitivity, the risk of collision increased by 15% in nighttime stair (hazard ratio [HR], 1.15; 95% CI, 1.08-1.22) and city (HR, 1.15; 95% CI, 1.08-1.23) navigations. Fifty-eight patients (59.1%) with glaucoma had vision-related disability in at least 1 simulated daily task; a higher proportion of patients had vision-related disability in nighttime city (27 of 88 [30.7%]) and stair (27 of 90 [30.0%]) navigation than in daytime city (7 of 88 [8.0%]) and stair (19 of 96 [19.8%]) navigation. The overall VR disability score was associated with the National Eye Institute 25-item Visual Function Questionnaire Rasch score (R2 = 0.207). These findings suggest that vision-related disability is associated with lighting condition and task in patients with glaucoma. Virtual reality may allow eye care professionals to understand the patients' perspectives on how visual impairment imparts disability in daily living and provide a new paradigm to augment the assessment of vision-related disability.

A Digital Reconstruction of a Historical Building and Virtual Reintegration of Mural Paintings to Create an Interactive and Immersive Experience in Virtual Reality

Nowadays, virtual reality technologies and immersive virtual reality (VR) apps allow people to view, explore, engage with and learn about historic monuments and buildings, historic sites, and even historic scenes. To preserve our cultural heritage for future generations. it is essential that damaged and dilapidated historic artefacts are accurately documented, and that steps are taken to improve user experiences in the areas of virtual visits, science and education. This paper describes an approach to reconstruct and restore historic buildings and mural paintings. The work process uses digital models that are then inserted into an interactive and immersive VR environment. Windows-Mixed Reality is used to visualize the said VR environment. The work method was applied at a United Nations Educational, Scientific and Cultural Organisation (UNESCO) World Heritage Site in Tenerife (Canary Islands, Spain), thereby creating a virtual three dimensional (3D) rendering of the architectural structures of the St Augustine Church in La Laguna and its murals. A combination of topography and terrestrial photogrammetry was used to reconstruct its architectural features, and the digital imaging tool DStretch® to recover its murals. The resulting 3D model was then inserted into an immersive and interactive VR environment created using the cross-platform game engine Unity. One of the greatest challenges of this project revolved around recovering the dilapidated and virtually nonexistent mural paintings using DStretch®. However, the final result is an immersive and interactive VR environment containing architectural and artistic information created within the video game engine Unity, which thereby allows the user to explore, observe and interact with a cultural heritage site in real time.

Haptic Interaction in Virtual Reality Environments for Manual Assembly Validation

Based on an increased level of product complexity in product development processes new technologies and validation approaches are researched and applied in industry to validate assembly processes. Especially technologies that are focusing on Mixed or Virtual Reality environments are getting very popular in the field of assembly validation, because of a huge potential to reduce the necessary number of physical prototypes. Therefore costs and time within product integration phase can be saved. These technologies can provide a high level of immersion for pick-and place or assembly tasks and allow a scale 1:1 interaction with the future product or production system. Still plenty of limitations for a complete substitution of physical prototypes exist due to the lack of physical and haptic behavior of these virtual prototypes. Within this paper, a development method and applications for haptic interaction with force-feedback devices for assembly validation is presented. Focusing especially on the development process of a mixed prototype with virtual and physical elements, so called hybrid prototypes, a hard- and software framework is presented. Nowadays virtual assembly methods for Virtual Reality and computer haptics are still based on a high level of expertise in providing technology and testbeds. To keep a high level of flexibility regarding virtual validation models, easily applicable procedures for modelling, im-/exporting and configuration of the prototype are needed for a virtual assembly use case. In this paper, a new and connected processing pipeline is presented that describes and guides a non-expert user from CAD environment to the implementation and configuration of prototypes for haptic interaction.