3D Immersive Environment Research Articles

Visual Perceptive and Blockchain-based Decentralized Metaverse Systems, Machine Vision and Geolocation Data Processing Algorithms, and Virtual Twin and Deep Learning-based Sensing Technologies in 3D Immersive Environments

Visual Perceptive and Blockchain-based Decentralized Metaverse Systems, Machine Vision and Geolocation Data Processing Algorithms, and Virtual Twin and Deep Learning-based Sensing Technologies in 3D Immersive Environments

Deep Learning-based Sensing and Extended Reality Technologies, Visual Recognition and Geospatial Mapping Tools, and Virtual Simulation and Spatial Cognition Algorithms in Digital Twin Cities and Immersive 3D Environments

Deep Learning-based Sensing and Extended Reality Technologies, Visual Recognition and Geospatial Mapping Tools, and Virtual Simulation and Spatial Cognition Algorithms in Digital Twin Cities and Immersive 3D Environments

Virtual Data Modeling and Geospatial Mapping Tools, Data-driven Internet of Things and Sustainable Urban Computing Systems, and Digital Twin Simulation and Extended Reality Technologies in Augmented Reality-powered Immersive Spaces and 3D Digital Environments

Virtual Data Modeling and Geospatial Mapping Tools, Data-driven Internet of Things and Sustainable Urban Computing Systems, and Digital Twin Simulation and Extended Reality Technologies in Augmented Reality-powered Immersive Spaces and 3D Digital Environments

GDOM: An Immersive Experience of Intangible Heritage through Spatial Storytelling

This paper presents the design, development, and evaluation of GDOM (Geelong Digital Outdoor Museum) application that integrates intangible heritage stories into places of public significance through a 3D virtual immersive environment. The project makes an important contribution to community-centered intangible heritage, while bridging the gap between theory and practice of location-based, non-linear storytelling. Research by Design methodology has been adopted to integrate highly cross-disciplinary insights into the creation, reproduction, and evaluation of a tangible application. A web-based 360° panoramic image viewer platform has been utilised to design and curate an interactive heritage experience, by spatially linking stories (content) to specific locations. The key innovation is the location-based, non-linear and spatial storytelling inside a 3D immersive virtual space where users have the opportunity to interact with intangible heritage stories. GDOM application opened up new opportunities to connect people with intangible heritage to facilitate new forms of environmental knowing, spatial and cultural understanding, and the creation of a sense of place. An in-depth evaluation of GDOM, with both expert and non-expert user groups, confirmed the GDOM application as an effective tool to experience intangible heritage to facilitate better understanding of places compared to a physical experience of heritage in a museum. Potential context of application with immediate benefits have been reported as education and cultural tourism sectors.

A walk in the black-box: 3D visualization of large neural networks in virtual reality

Within the last decade Deep Learning has become a tool for solving challenging problems like image recognition. Still, Convolutional Neural Networks (CNNs) are considered black-boxes, which are difficult to understand by humans. Hence, there is an urge to visualize CNN architectures, their internal processes and what they actually learn. Previously, virtual realityhas been successfully applied to display small CNNs in immersive 3D environments. In this work, we address the problem how to feasibly render large-scale CNNs, thereby enabling the visualization of popular architectures with ten thousands of feature maps and branches in the computational graph in 3D. Our software ”DeepVisionVR” enables the user to freely walk through the layered network, pick up and place images, move/scale layers for better readability, perform feature visualization and export the results. We also provide a novel Pytorch module to dynamically link PyTorch with Unity, which gives developers and researchers a convenient interface to visualize their own architectures. The visualization is directly created from the PyTorch class that defines the Pytorch model used for training and testing. This approach allows full access to the network’s internals and direct control over what exactly is visualized. In a use-case study, we apply the module to analyze models with different generalization abilities in order to understand how networks memorize images. We train two recent architectures, CovidResNet and CovidDenseNet on the Caltech101 and the SARS-CoV-2 datasets and find that bad generalization is driven by high-frequency features and the susceptibility to specific pixel arrangements, leading to implications for the practical application of CNNs. The code is available on Github https://github.com/Criscraft/DeepVisionVR.

Designing Guided User Tasks in VR Embodied Experiences

Virtual reality (VR) offers extraordinary opportunities in user behavior research to study and observe how people interact in immersive 3D environments. A major challenge of designing these 3D experiences and user tasks, however, lies in bridging the inter-relational gaps of perception between the designer, the user, and the 3D scene. Paul Dourish identified three gaps of perception: ontology between the scene representation and the user and designer interpretation, intersubjectivity of task communication between designer and user, and intentionality between the user's intentions and designer's interpretations. We present the GUsT-3D framework for designing Guided User Tasks in embodied VR experiences, i.e., tasks that require the user to carry out a series of interactions guided by the constraints of the 3D scene. GUsT-3D is implemented as a set of tools that support a 4-step workflow to (1) annotate entities in the scene with navigation and interaction possibilities, (2) define user tasks with interactive and timing constraints, (3) manage interactions, task validation, and user logging in real-time, and (4) conduct post-scenario analysis through spatio-temporal queries using ontology definitions. To illustrate the diverse possibilities enabled by our framework, we present two case studies with an indoor scene and an outdoor scene, and conducted a formative evaluation involving six expert interviews to assess the framework and the implemented workflow. Analysis of the responses show that the GUsT-3D framework fits well into a designer's creative process, providing a necessary workflow to create, manage, and understand VR embodied experiences.

Enhancing Intrinsic Motivation of Librarian Students using Virtual Reality for Education in the Context of Culture Heritage Museums

In this study the effects of intrinsic motivation and learning competency of virtual realitybased learning is compared to traditional text-based learning. In the study 28 students were involved from the second and third years of the library and information science faculty. The application of the fully immersive virtual 3D environment of Wieng Yong House was developed to support the virtual realitybased learning in the context of museums for culture heritage. To verify the hypothesis, the control group (traditional text-based learning), consisted of 14 students, used the course module of metadata for culture heritage in the museum whereas the experimental group (control group), consisted of 14 students, was provided virtual reality-based learning in the virtual museum. The tools of measurement were the pre-test/post-test quiz for learning competency and Intrinsic Motivation Inventory (IMI) with interview. The statistical model of paired sampled t-test assumptions was chosen for the statistical investigation for the hypothesis. The results suggest that constructing a virtual reality for learning in the context of a museum for cultural heritage might be a good way to enhance intrinsic motivation but cannot sufficiently and significantly improve the student's learning competency.

Residents’ views on adaptable housing: a virtual reality-based study

Housing adaptability is often said to be a characteristic that benefits the residents. But is there an actual demand from residents? This study asks whether and what kind of value residents place on adaptable housing. The research, situated in Finland, combines interviews involving seven varied households together with their experiences of adaptable flats in a 3D CAVE immersive virtual reality environment. The participants were first interviewed about their current life, housing situations and expected future changes in household composition or housing needs. For the virtual viewings, flats with systematically transformable floor plans were designed to anticipate situations that could occur during people’s housing careers (e.g. the birth of a child, working from home, a child leaving home, additional healthcare, etc.). The participants walked through virtual flats selected for them based on the first interview. Afterwards, they were re-interviewed, focusing on the value they place on the presented adaptability. All households identified functional benefits for their current or anticipated situation, and many expressed a willingness to buy a home with the demonstrated adaptability characteristics. Most also described economic security that adaptability provides (e.g. renting or selling a part of the flat separately) and the benefits of remaining long term in a flat. <em><strong>Practice relevance</strong></em> In the adaptability discourse, residents’ opinions have been heard to a very limited degree. More typically, adaptability researchers or architects make the case for adaptability on the residents’ behalf. According to recent research, some architects assume developers are disinterested in building more adaptable flats partly because no real market demand by inhabitants is believed to exist. The findings of the current study demonstrate that residents need to be properly informed about the benefits of adaptability by the housing provider. Virtual reality can be a useful tool for demonstrating adaptability characteristics. When sufficiently informed, the participants of the current study considered adaptability as functionally beneficial for the family and a desirable feature that would increase their willingness to buy and pay a premium. To demonstrate a potential market or its absence in a given context, residents’ views should be sought more widely and more often.

Simulation and Virtualization Technologies, Artificial Intelligence Diagnostic Tools, and Metaverse Healthcare Systems in 3D Immersive Environments

Simulation and Virtualization Technologies, Artificial Intelligence Diagnostic Tools, and Metaverse Healthcare Systems in 3D Immersive Environments

Digital Twin-based Cyber-Physical Production Systems in Immersive 3D Environments: Virtual Modeling and Simulation Tools, Spatial Data Visualization Techniques, and Remote Sensing Technologies

Digital Twin-based Cyber-Physical Production Systems in Immersive 3D Environments: Virtual Modeling and Simulation Tools, Spatial Data Visualization Techniques, and Remote Sensing Technologies

Digital Twin Simulation and Modeling Tools, Computer Vision Algorithms, and Urban Sensing Technologies in Immersive 3D Environments

Digital Twin Simulation and Modeling Tools, Computer Vision Algorithms, and Urban Sensing Technologies in Immersive 3D Environments

Immersive Virtual Reality Experience of Historical Events Using Haptics and Locomotion Simulation

Virtual reality (VR) and 3D modeling technologies have become increasingly powerful tools for multiple fields, such as education, architecture, and cultural heritage. Museums are no longer places for only placing and exhibiting collections and artworks. They use such technologies to offer a new way of communicating art and history with their visitors. In this paper, we present the initial results of a proposed workflow towards highlighting and interpreting a historic event with the use of an immersive and interactive VR experience and the utilization of multiple senses of the user. Using a treadmill for navigating and haptic gloves for interacting with the environment, combined with the detailed 3D models, deepens the sense of immersion. The results of our study show that engaging multiple senses and visual manipulation in an immersive 3D environment can effectively enhance the perception of visual realism and evoke a stronger sense of presence, amplifying the educational and informative experience in a museum.

Teleoperation and Visualization Interfaces for Remote Intervention in Space

Approaches to robotic manufacturing, assembly, and servicing of in-space assets range from autonomous operation to direct teleoperation, with many forms of semi-autonomous teleoperation in between. Because most approaches require one or more human operators at some level, it is important to explore the control and visualization interfaces available to those operators, taking into account the challenges due to significant telemetry time delay. We consider one motivating application of remote teleoperation, which is ground-based control of a robot on-orbit for satellite servicing. This paper presents a model-based architecture that: 1) improves visualization and situation awareness, 2) enables more effective human/robot interaction and control, and 3) detects task failures based on anomalous sensor feedback. We illustrate elements of the architecture by drawing on 10 years of our research in this area. The paper further reports the results of several multi-user experiments to evaluate the model-based architecture, on ground-based test platforms, for satellite servicing tasks subject to round-trip communication latencies of several seconds. The most significant performance gains were obtained by enhancing the operators’ situation awareness via improved visualization and by enabling them to precisely specify intended motion. In contrast, changes to the control interface, including model-mediated control or an immersive 3D environment, often reduced the reported task load but did not significantly improve task performance. Considering the challenges of fully autonomous intervention, we expect that some form of teleoperation will continue to be necessary for robotic in-situ servicing, assembly, and manufacturing tasks for the foreseeable future. We propose that effective teleoperation can be enabled by modeling the remote environment, providing operators with a fused view of the real environment and virtual model, and incorporating interfaces and control strategies that enable interactive planning, precise operation, and prompt detection of errors.

An Immersive and Interactive Platform for Cognitive Assessment and Rehabilitation (bWell): Design and Iterative Development Process.

BackgroundImmersive technologies like virtual reality can enable clinical care that meaningfully aligns with real-world deficits in cognitive functioning. However, options in immersive 3D environments are limited, partly because of the unique challenges presented by the development of a clinical care platform. These challenges include selecting clinically relevant features, enabling tasks that capture the full breadth of deficits, ensuring longevity in a rapidly changing technology landscape, and performing the extensive technical and clinical validation required for digital interventions. Complicating development, is the need to integrate recommendations from domain experts at all stages.ObjectiveThe Cognitive Health Technologies team at the National Research Council Canada aims to overcome these challenges with an iterative process for the development of bWell, a cognitive care platform providing multisensory cognitive tasks for adoption by treatment providers.MethodsThe team harnessed the affordances of immersive technologies while taking an interdisciplinary research and developmental approach, obtaining active input from domain experts with iterative deliveries of the platform. The process made use of technology readiness levels, agile software development, and human-centered design to advance four main activities: identification of basic requirements and key differentiators, prototype design and foundational research to implement components, testing and validation in lab settings, and recruitment of external clinical partners.ResultsbWell was implemented according to the findings from the design process. The main features of bWell include multimodal (fully, semi, or nonimmersive) and multiplatform (extended reality, mobile, and PC) implementation, configurable exercises that pair standardized assessment with adaptive and gamified variants for therapy, a therapist-facing user interface for task administration and dosing, and automated activity data logging. bWell has been designed to serve as a broadly applicable toolkit, targeting general aspects of cognition that are commonly impacted across many disorders, rather than focusing on 1 disorder or a specific cognitive domain. It comprises 8 exercises targeting different domains: states of attention (Egg), visual working memory (Theater), relaxation (Tent), inhibition and cognitive control (Mole), multitasking (Lab), self-regulation (Butterfly), sustained attention (Stroll), and visual search (Cloud). The prototype was tested and validated with healthy adults in a laboratory environment. In addition, a cognitive care network (5 sites across Canada and 1 in Japan) was established, enabling access to domain expertise and providing iterative input throughout the development process.ConclusionsImplementing an interdisciplinary and iterative approach considering technology maturity brought important considerations for the development of bWell. Altogether, this harnesses the affordances of immersive technology and design for a broad range of applications, and for use in both cognitive assessment and rehabilitation. The technology has attained a maturity level of prototype implementation with preliminary validation carried out in laboratory settings, with next steps to perform the validation required for its eventual adoption as a clinical tool.

Does the Size-Arrival Effect Occur With an Active Collision-Avoidance Task in an Immersive 3D Virtual Reality Environment?

Determine whether the size-arrival effect (SAE) occurs with immersive, 3D visual experiences and active collision-avoidance responses. When a small near object and a large far object approach the observer at the same speeds, the large object appears to arrive before the small object, known as the size-arrival effect (SAE), which may contribute to crashes between motorcycles and cars. Prior studies of the SAE were limited because they used two dimensional displays and asked participants to make passive judgments. Participants viewed approaching objects using a virtual reality (VR) headset. In an active task, participants ducked before the object hit them. In a passive prediction-motion (PM) judgment, the approaching object disappeared, and participants pressed a button when they thought the object would hit them. In a passive relative TTC judgment, participants reported which of two approaching objects would reach them first. The SAE occurred with the PM and relative TTC tasks but not with the ducking task. The SAE can occur in immersive 3D environments but is limited by the nature of the task and display. Certain traffic situations may be more prone to the SAE and have higher risk for collisions. For example, in left-turn scenarios (e.g., see Levulis, 2018), drivers make passive judgments when oncoming vehicles are far and optical expansion is slow, and binocular disparity putatively is ineffective. Collision-avoidance warning systems may be needed more in such scenarios than when vehicles are near and drivers' judgments of TTC may be more accurate (DeLucia, 2008).

UnityMol prototype for FAIR sharing of molecular-visualization experiences: from pictures in the cloud to collaborative virtual reality exploration in immersive 3D environments.

Motivated by the current COVID-19 pandemic, which has spurred a substantial flow of structural data, the use of molecular-visualization experiences to make these data sets accessible to a broad audience is described. Using a variety of technology vectors related to the cloud, 3D and virtual reality gear, how to share curated visualizations of structural biology, modeling and/or bioinformatics data sets for interactive and collaborative exploration is examined. FAIR is discussed as an overarching principle for sharing such visualizations. Four initial example scenes related to recent COVID-19 structural data are provided, together with a ready-to-use (and share) implementation in the UnityMol software.

The Development of a Virtual World Problem-Based Learning Tutorial and Comparison With Interactive Text-Based Tutorials

Collaborative learning through case-based or problem-based learning (PBL) scenarios is an excellent way to acquire and develop workplace knowledge associated with specific competencies. At St George's, University of London we developed an interactive online form of decision-based PBL (D-PBL) for our undergraduate medical course using web-based virtual patients (VPs). This method of delivery allowed students to consider options for clinical management, to take decisions and to explore the consequences of their chosen actions. Students had identified this as a more engaging type of learning activity compared to conventional paper-based/linear PBL and demonstrated improved exam performance in controlled trials. We explored the use of Second Life (SL), a virtual world and immersive 3D environment, as a tool to provide greater realism than our interactive image and text-based D-PBL patient cases. Eighteen separate tutorial groups were provided with their own experience of the same patient scenario in separate locations within the virtual world. The study found that whilst a minority of students reported that the Second Life experience felt more realistic, most did not. Students favored the simpler interaction of the web-based VPs, which already provided them with the essential learning needed for practice. This was in part due to the time proximity to exams and the extra effort required to learn the virtual world interface. Nevertheless, this study points the way towards a scalable process for running separate PBL sessions in 3D environments.

Design and evaluation of a 3D serious game for communication learning in nursing education

BackgroundCommunication is essential for nursing practice; such competence is not intuitive and must be taught. Serious games in 3D virtual reality can support the nursing teaching-learning process; students connected to the virtual world can apply what they have learned in their professional practice. AimTo describe the design and heuristic evaluation of a serious game in 3D immersive VR for the development of communication competence. MethodA methodological study was conducted to develop and evaluate the serious game in a 3D immersive virtual environment using Oculus Rift®, called Comunica-Enf. The game consists of the user asking for authorization to the patient's avatar to perform a nursing procedure using communication abilities. Professors from six nursing courses and 30 students from two undergraduate nursing courses were recruited to evaluate the usability and appearance of the Comunica-Enf serious game. After using the game, the participants completed the Heuristic Evaluation of Digital Educational Games. A maximum of 25% of high or urgent priority problems in the whole game or heuristics is necessary to validate the game. ResultThe participants considered Comunica-Enf suitable for nursing education because none of the heuristics reached more than 25% of high or urgent priority problems. The participants made suggestions that were analyzed and incorporated into the Comunica-Enf when possible. ConclusionComunica-Enf is an educational technology that uses Oculus Rift® and a 3D virtual reality perspective as an innovative resource. It had its appearance and usability validated for nursing students, providing a safe and pleasant environment for communication competence.

Decreasing Stress Through a Spatial Audio and Immersive 3D Environment: A Pilot Study With Implications for Clinical and Medical Settings

There is evidence that music-based interventions can be effective in treating clinical and non-clinical populations. With the emergence of audio-visual technologies, there are new opportunities for developing web-based applications that have therapeutic effects in mental health and medical settings. Here we conducted a pilot study ( N = 100) to test if an immersive web-based spatial audio application can decrease stress and negative mood states. Results showed that the application was effective for both clinical ( n = 40) and non-clinical ( n = 60) groups, and that the effect was most profound for individuals diagnosed with depression and anxiety disorders. Though the present study needs to be replicated with physiological methods, the findings provide initial evidence that web-based spatial audio applications can be effective for short-term stress reduction and have the potential to be a supplement to clinical music interventions, but not a replacement or substitute for such interventions.

Metrics for Evaluating Cyber Security Data Visualizations in Virtual Reality

Abstract Cyber security analysts use data visualizations to speed up ingestion of security data. These visualizations typically take the form of 2D graphics displayed on computer monitors. Virtual reality has the potential to improve these visualizations with immersive 3D environments and unique interaction mechanics. However, research into this newly synergized area lacks evaluation, leading to unfounded claims of effectiveness. A potential cause for these missing evaluations was identified as a lack of guidance detailing how evaluations should be conducted in this area. Additionally, the small amount of research that does include evaluation incorrectly relies on subjective participant opinions to objectively measure system effectiveness. An example of this misuse is asking participants which system they thought was quicker, rather than timing them. The objective of this article is to propose a solution to these issues in the form of a surveyed, categorized, and analyzed set of evaluation metrics. A total of 49 metrics was identified from 41 papers. The presented metrics detail which dependent variables should be considered when evaluating works in the combined fields of cyber security, data visualization, and virtual reality. These metrics can be used to produce more accurate evaluations in future works in this area.