Immersive System Research Articles

Latency Reduction in Immersive Systems through Request Scheduling with Digital Twin Networks in Collaborative Edge Computing

Immersive systems, which offer immersive interactive experiences by tightly coupling the real and virtual worlds, encompassing Augmented Reality (AR) and Virtual Reality (VR), have undergone rapid development in recent years. The inherent latency issue of immersive systems poses considerable challenges for AR/VR applications that are highly sensitive to delay. To address this challenge, we propose utilizing edge collaboration as an efficient means to schedule AR/VR requests, owing to the inherent ability of edge nodes to seamlessly coordinate within the same geographical region and optimally allocate request resources to minimize latency. However, when it comes to edge nodes across different regions, a significant hurdle emerges. These nodes often lack real-time knowledge about cached resources across other regions, making cross-regional edge collaboration for AR/VR request scheduling an extremely challenging problem. We formally define and prove the NP-hardness of this problem. To tackle this complexity, we introduce a Federated Digital Twin Model Construction (FedDT), which establishes a global digital twin network connecting different edge nodes. This network simulates and reflects the resource status and availability of each edge node in real time. Subsequently, we devise an extended Dijkstra algorithm to determine the shortest path between AR/VR requests and edge nodes, taking into account current network conditions to optimize service delay. Both theoretical analysis and experimental results confirm that compared with baseline methods, our proposed solution consistently achieves an average delay reduction of 36.78% across various scenarios, effectively demonstrating its superior performance.

Developmental changes in how pedestrians cross streets in single- vs. dual-lane traffic conditions.

Pedestrian injury is a leading contributor to childhood deaths. This study compared the crossing behaviors of children with adults when crossing in virtual single-lane vs. two-lane traffic conditions. Using a fully immersive virtual reality system, children 7-10 years and adults crossed streets with traffic in one lane (near lane) or in two lanes (near and far lane) with 0 offset so cars from opposing directions arrived at the child's crossing line at the same time. On each trial, pedestrians made a decision on when to cross and then completed the crossing, with measures of attention and crossing behaviors automatically recorded by the system as they did so. In comparison to crossing a single lane of traffic, crossing two-lane roads increased injury risk for all pedestrians, though children experienced significantly greater risk than adults. Children predominantly crossed by stopping before entering the far lane, whereas adults showed greater synchronization of self-movement to traffic flow and more often crossed both lanes without stopping. Children experience more high-risk outcomes than adults when crossing single-lane roads. Crossing two-lane roads elevates risk for pedestrians of both ages, though this risk is significantly greater for child than adult pedestrians. The predominant strategy used to cross two lanes of traffic shows significant developmental changes. Implications for injury prevention are discussed.

Feedback on the creation of a scientific show using immersive sound

This presentation is devoted to feedback on the creation of a scientific show produced in collaboration between two acoustic laboratories, musicians and a loudspeaker manufacturer. The aim of the show is to simply explain sound perception to a wide audience using an immersive sound system. The presentation focuses on the creative process (including scenario writing, musical compositions, choice of technical solutions for the audio rendering) and the compromises made between scientific rigor and the artistic aspect. The various phases of the work carried out over almost two years are explained in order to formalize the creative process and the need to find a specific method that is suitable for both artists and scientists.

Clinical Feasibility of Applying Immersive Virtual Reality during Robot-Assisted Gait Training for Individuals with Neurological Diseases: A Pilot Study.

Immersive virtual reality has the potential to motivate and challenge patients who need and want to relearn movements in the process of neurorehabilitation. The aim of this study was to evaluate the feasibility and user acceptance of an innovative immersive virtual reality system (head-mounted display) used in combination with robot-assisted gait training in subjects suffering from neurological diseases. Fifteen participants suffering from cerebrovascular accident or spinal cord injury completed a single session of immersive virtual reality using a head-mounted display during a Lokomat® gait session. Training parameters and safety indicators were collected, and acceptance was investigated among participants and therapists. The results suggest that an immersive virtual reality system is feasible in terms of safety and tolerance. Furthermore, the very positive overall acceptance of the system suggests that it has the potential to be included in a robot-assisted gait training session using Lokomat®. Overall, this study demonstrates that a fully immersive virtual reality system based on a head-mounted display is both feasible and well received by cerebrovascular accident and spinal cord injury patients and their therapists during robot-assisted gait training. This study suggests that such a virtual reality system could be a viable alternative to the screen-based training games currently used in neurorehabilitation. It may be especially suitable for enhancing patient motivation and adherence to training, particularly if the application is enjoyable and not mentally taxing.

A virtual reality-based immersive teleoperation system for remote human-robot collaborative manufacturing

In recent years, the cutting-edge technologies in smart manufacturing have presented promising opportunities for the utilization of human-robot collaborative teleoperation in personalized manufacturing tasks. To effectively leverage the creative capabilities of humans while benefiting from the efficiency and stability of robots, the provision of an intuitive teleoperation interface assumes paramount importance. However, current teleoperation systems still face limitations in terms of intuitive operability. In this study, we present a virtual reality-based teleoperation system that offers operators a more intuitive interaction platform for robot control, thereby facilitating personalized manufacturing processes. The overall system framework design, as well as the main components are elaborated in detail. Furthermore, an evaluative case study based on the battery disassembly task is conducted to assess the performance of the proposed system. The results demonstrate that the proposed teleoperation system exhibits improved intuitiveness.

Learner evaluation of an immersive virtual reality mass casualty incident simulator for triage training

BackgroundTo minimize loss of life, modern mass casualty response requires swift identification, efficient triage categorization, and rapid hemorrhage control. Current training methods remain suboptimal. Our objective was to train first responders to triage a mass casualty incident using Virtual Reality (VR) simulation and obtain their impressions of the training’s quality and effectiveness.We trained subjects in a triage protocol called Sort, Assess, Lifesaving interventions, and Treatment and/or Transport (SALT) Triage then had them respond to a terrorist bombing of a subway station using a fully immersive virtual reality simulation. We gathered learner reactions to their virtual reality experience and post-encounter debriefing with a custom electronic survey. The survey was designed to gather information about participants’ demographics and prior experience, including roles, triage training, and virtual reality experience. We then asked them to evaluate the training and encounter and the system’s potential for training others.ResultsWe received 375 completed evaluation surveys from subjects who experienced the virtual reality encounter. Subjects were primarily paramedics, but also included medical learners as well as other emergency medical service (EMS) professionals. Most participants (95%) recommended the experience for other first responders and rated the simulation (95%) and virtual patients (91%) as realistic. Ninety-four percent (94%) of participants rated the virtual reality simulator as “excellent” or “good.” We observed some differences between emergency medical service and medical professionals regarding their prior experience with disaster response training and their opinions on how much the experience contributed to their learning. We observed no differences between subjects with extensive virtual reality experience and those without.ConclusionsOur virtual reality simulator is an automated, customizable, fully immersive virtual reality system for training and assessing personnel in the proper response to a mass casualty incident. Participants perceived the simulator as an adequate alternative to traditional triage and treatment training and believed that the simulator was realistic and effective for training. Prior experience with virtual reality was not a prerequisite for the use of this system.

Ten Years of Operating a Center with Large-Scale Virtual-Reality Installations: Developments and Learnings

Abstract Virtual reality (VR) can be found in many fields including art history, education, research, and industry 4.0. Next to lightweight head-mounted displays there are also VR installations, such as immersive 3D screens, large-scale displays, and CAVE systems, that are used in research, scientific visualization, and also the automotive industry. These systems offer high visual quality and collaborative VR experiences for researchers and have long been used in research. We present learnings and insights from ten years of operating and maintaining a visualization center with large-scale immersive displays and installations. Our report aims to answer questions on the benefits and challenges of such a location-based VR center. We broke down the ten years into three phases and discuss the installations themselves, software, and other various developments of the center over time, illustrated with exemplary use cases. Finally, we cover our experiences at the beginning of phase four with the installation of a (novel) LED CAVE and the future (forward-looking) developments we expect for location-based VR centers.

The Digital Twin Immersive Design Process and Its Potential Disruption to Healthcare Design through a User-Centered Approach

This applied research proposes a solution to the static government design process for Thai healthcare architecture to better serve the needs of its elderly society. In its place, a novel real-time design process, termed the Digital Twin Immersive Design Process (DT-IDP), repurposes aspects of digital twin and virtual reality technologies into a ‘unitary’ immersive design system. This system accesses ‘experiential’ user-centered data, helping enhance the design of Thai healthcare space beyond a standardized government response. This text builds a rationale for departing from the current design process by describing the formation and advantages of the DT-IDP process. To test its credibility, the DT-IDP process is used to build and compare two digital versions of an existing healthcare space. In these spaces, participants are immersed (elderly patients n = 30; nursing staff n = 5; government healthcare architects n = 5) to assess visitor experiences based on daylighting, artificial lighting, and views of nature. Following immersion, government healthcare architects are interviewed in-depth to evaluate the process’s efficacy and their willingness to adopt it. Results confirm the potential for this process to capture ‘user-centered’ insights, otherwise unobtainable without immersion. Consequently, healthcare architects express a unanimous preference for DT-IDP, acknowledging its unique capacity to bridge a market gap through an experiential component that could better assist them in creating a superior final product. Ultimately, they assert that demand for these features could have a disruptive impact to the current healthcare design process, helping to re-envisage the design of future Thai healthcare space.

Geo-spatial hypertext in virtual reality: mapping and navigating global news event spaces

ABSTRACT Every day, a myriad of events take place that are documented and shared online through news articles from a variety of sources. As a result, as users navigate the Web, the volume of data can lead to information overload, making it difficult to find specific details about an event. We present News in Time and Space (NiTS) to address this issue: NiTS is a fully immersive system integrated into Va.Si.Li-Lab that organises textual information in a geospatial hypertext system in virtual reality. With NiTS, users can visualise, filter and interact with information currently based on GDELT on a virtual globe providing document networks to analyse global events and trends. The article describes NiTS, its event semantics and architecture. It evaluates NiTS in comparison to a classic search engine website, extended by NiTS's information filtering capabilities to make it comparable. Our comparison with this website technology, which is directly linked to the user's usage habits, shows that NiTS enables comparable information exploration even if the users have little or no experience with VR. That is, we observe an equivalent search result behaviour, but with the advantage that VR allows users to get their results with a higher level of usability without distracting them from their tasks. Through its integration with Va.Si.Li-Lab, a simulation-based learning environment, NiTS can be used in simulations of learning processes aimed at studying critical online reasoning, where Va.Si.Li-Lab guarantees that this can be done in relation to individual or groups of learners.

Development and Evaluation of Training Scenarios for the Use of Immersive Assistance Systems

Emerging assistance systems are designed to enable operators to perform tasks better, faster, and with a lower workload. However, in line with the productivity paradox, the full potential of automation and digitalisation is not being realised. One reason for this is insufficient training. In this study, the statistically significant differences among three different training scenarios on performance, acceptance, workload, and technostress during the execution of immersive measurement tasks are demonstrated. A between-subjects design was applied and analysed using ANOVAs involving 52 participants (with a statistical overall power of 0.92). The ANOVAs were related to three levels of the independent variable: quality training, manipulated as minimal, personal, and optimised training. The results show that the quality of training significantly influences immersive assistance systems. Hence, this article deduces tangible design guidelines for training, with consideration of the system-level hardware, operational system, and immersive application. Surprisingly, an appropriate mix of training approaches, rather than detailed, personalised training, appears to be more effective than e-learning or ‘getting started’ tools for immersive systems. In contrast to most studies in the related work, our article is not about learning with AR applications but about training scenarios for the use of immersive systems.

Investigating Professional Analyst Strategies in Immersive Space to Think.

Existing research on sensemaking in immersive analytics systems primarily focuses on understanding how users complete analysis within these systems with quantitative and qualitative datasets. However, these user studies mainly concentrate on understanding analysis styles and methodologies from a predominantly novice user study population. While this approach provides excellent initial insights into what users may do within IA systems, it fails to address how professionals may utilize an immersive analytic system for analysis tasks. In our work, we build upon an existing immersive analytics concept - "Immersive Space to Think" to understand how professional user populations differ from novice users in immersive analytic system usage. We conducted a user study with 11 professional intelligence analysts who completed three analysis sessions each. Using our results from this study, we provide deep analysis into how professional users complete sensemaking within immersive analytic systems, compare our findings to previously published findings with a novice user population, and provide insights into how to develop better IA systems to support the professional analyst's strategies within these systems.

A virtual-reality spatial matching algorithm and its application on equipment maintenance support: System design and user study

Equipment maintenance support is an important technical measure to maintain the equipment’s expected performance. However, the current maintenance supports are mainly completed by maintainers under the guidance of technical manual or additional experts, which may be insufficient for some advanced equipment with rapid update rate and complex inner structure. The rising technology of augmented reality (AR) provides a new solution for equipment maintenance support, while one of the key issues limiting the practical application of AR in maintenance field is the spatial matching issue between virtual space and reality space. In this paper, a virtual-reality spatial matching algorithm is designed to accurately superimpose the virtual information to the corresponding actual scene on the AR glasses. In this algorithm, two methods are proposed to help achieve the stable matching of virtual space and reality space. In detail, to obtain the saliency map with less background interference and improved saliency detection accuracy, a saliency detection method is designed based on the super-pixel segmentation. To deal with the problems of uneven distribution on the feature points and weak robustness to the light changes, a feature extraction and matching method is proposed for acquiring the feature point matching set with the utilization of the obtained saliency map. Finally, an immersive equipment maintenance support system (IEMSS) is developed based on this spatial matching algorithm, which provides the maintainers with immediate and immersive guidance to improve the efficiency and safety in the maintenance task, as well as offers maintenance training for inexperienced maintainers with expanded virtual information in case of limited experts. Several comparative experiments are implemented to verify the effectiveness of proposed methods, and a user study of real system application is carried out to further evaluate the superiority of these methods when applied in the IEMSS.

Immersive analytics with augmented reality in meteorology: an exploratory study on ontology and linked data

Although Augmented Reality (AR) has been extensively studied in supporting Immersive Analytics (IA), there are still many challenges in visualising and interacting with big and complex datasets. To deal with these datasets, most AR applications utilise NoSQL databases for storing and querying data, especially for managing large volumes of unstructured or semi-structured data. However, NoSQL databases have limitations in their reasoning and inference capabilities, which can result in insufficient support for certain types of queries. To fill this gap, we aim to explore and evaluate whether an intelligent approach based on ontology and linked data can facilitate visual analytics tasks with big datasets on AR interface. We designed and implemented a prototype of this method for meteorological data analytics. An experiment was conducted to evaluate the use of a semantic database with linked data compared to a conventional approach in an AR-based immersive analytics system. The results significantly highlight the performance of semantic approach in helping the users analysing meteorological datasets and their subjective appreciation in working with the AR interface, which is enhanced with ontology and linked data.

Toward More Comprehensive Evaluations of 3D Immersive Sketching, Drawing, and Painting.

To understand current practice and explore the potential for more comprehensive evaluations of 3D immersive sketching, drawing, and painting, we present a survey of evaluation methodologies used in existing 3D sketching research, a breakdown and discussion of important phases (sub-tasks) in the 3D sketching process, and a framework that suggests how these factors can inform evaluation strategies in future 3D sketching research. Existing evaluations identified in the survey are organized and discussed within three high-level categories: 1) evaluating the 3D sketching activity, 2) evaluating 3D sketching tools, and 3) evaluating 3D sketching artifacts. The new framework suggests targeting evaluations to one or more of these categories and identifying relevant user populations. In addition, building upon the discussion of the different phases of the 3D sketching process, the framework suggests to evaluate relevant sketching tasks, which may range from low-level perception and hand movements to high-level conceptual design. Finally, we discuss limitations and challenges that arise when evaluating 3D sketching, including a lack of standardization of evaluation methods and multiple, potentially conflicting, ways to evaluate the same task and user interface usability; we also identify opportunities for more holistic evaluations. We hope the results can contribute to accelerating research in this domain and, ultimately, broad adoption of immersive sketching systems.

Personalized rehabilitation approach for reaching movement using reinforcement learning

Musculoskeletal disorders challenge significantly the performance of many daily life activities, thus impacting the quality of life. The efficiency of the traditional physical therapy programs is limited by ecological parameters such as intervention duration and frequency, number of caregivers, geographic accessibility, as well as by subjective factors such as patient’s motivation and perseverance in training. The implementation of VR rehabilitation systems may address these limitations, but the technology still needs to be improved and clinically validated. Furthermore, current applications generally lack flexibility and personalization. A VR rehabilitation game simulation is developed, which focuses on the upper-limb movement of reaching, an essential movement involved in numerous daily life activities. Its novelty consists in the integration of a machine learning algorithm, enabling highly adaptive and patient-customized therapeutic intervention. An immersive VR system for the rehabilitation of reaching movement using a bubble popping game is proposed. In the virtual space, the patient is presented with bubbles appearing at different locations and is asked to reach the bubble with the injured limb and pop it. The implementation of a Q-learning algorithm enables the game to adjust the location of the next bubble according to the performance of the patient, represented by his kinematic characteristics. Two test cases simulate the performance of the patient during a training program of 10 days/sessions, in order to validate the effectiveness of the algorithm, demonstrated by the spatial and temporal distribution of the bubbles in each evolving scenario. The results show that the algorithm learns the patient’s capabilities and successfully adapts to them, following the reward policy dictated by the therapist; moreover, the algorithm is highly responsive to kinematic features’ variation, while demanding a reasonable number of iterations. A novel approach for upper limb rehabilitation is presented, making use of immersive VR and reinforcement learning. The simulation suggests that the algorithm offers adaptive capabilities and high flexibility, needed in the comprehensive personalization of a rehabilitation process. Future work will demonstrate the concept in clinical trials.

Electronic Visualization Laboratory's 50th Anniversary Retrospective: Look to the Future, Build on the Past

Abstract September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL's introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations—pillars of VR. But EVL's culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person's expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL's past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans.

A 15-Year Retrospective on Immersive Visualization Lessons from the Applied Visualization Laboratory

Abstract This paper shares the experiences of operating an immersive visualization laboratory over a 15-year period. The paper discusses valuable insights into the lessons learned from various projects, including the challenges they faced, such as technical difficulties, user adoption, and practical solutions for overcoming them. One crucial element for successful immersive visualization projects is interdisciplinary collaboration. The paper presents the advantages of immersive visualization technology, such as improved data comprehension, enhanced communication, and increased engagement in complex scenarios. We present the dynamic realm of virtual and augmented reality (VR/AR) technology in the context of an immersive laboratory system. We spotlight rising trends inside the integration of VR/AR tools for visualization. We also present our experience in dealing with the hardware and software additives in immersive VR/AR labs, shedding light on the demanding situations and successes encountered throughout everyday operations. Furthermore, we offer practical knowledge and guidance based on years of experience in the field, which can help you overcome challenges and achieve success in your project.

An efficient image classification and segmentation method for crime investigation applications

AbstractThe field of forensic science is experiencing significant growth, largely driven by the increasing integration of holographic and immersive technologies, along with their associated head-mounted displays. These immersive systems have become increasingly vital in resolving critical crimes as they facilitate communication, interaction, and collaboration. Given the sensitive nature of their work, crime investigators require substantial technical support. There is a pressing need for accurate documentation and archiving of crime scenes, which can be addressed by leveraging 3D scanned scenes to accurately represent evidence and expected scenarios. This study aims to develop an enhanced AR. system that can be deployed on hologram facilities such as the Microsoft HoloLens. The proposed system encompasses two main approaches, namely image classification and image segmentation. Image classification utilizes various deep learning models, including lightweight convolutional neural networks (CNNs) and convolutional Long-Short Term Memory (ConvLSTM). Additionally, the image segmentation approach is based on the fuzzy active contour model (FACM). The effectiveness of the proposed system was evaluated for both classification and segmentation tasks, utilizing metrics such as accuracy, sensitivity, precision, and F1 score. The simulation results indicate that the proposed system achieved a 99% accuracy rate in classification and segmentation tasks, positioning it as an effective solution for detecting bloodstain patterns in AR applications.

An Efficient Immersive Self-Training System for Hip-Hop Dance Performance with Automatic Evaluation Features

As a significant form of physical expression, dance demands ongoing training for skill enhancement, particularly in expressiveness. However, such training often faces restrictions related to location and time. Moreover, the evaluation of dance performance tends to be subjective, which necessitates the development of effective training methods and objective evaluation techniques. In this research, we introduce a self-training system for dance that employs VR technology to create an immersive training environment that facilitates a comprehensive understanding of three-dimensional dance movements. Furthermore, the system incorporates markerless motion capture technology to accurately record dancers’ movements in real time and translate them into the VR avatar. Additionally, the use of deep learning enables multi-perspective dance performance assessment, providing feedback to users to aid their repetitive practice. To enable deep learning-based dance evaluations, we established a dataset that incorporates data from beginner-level dances along with expert evaluations of those dances. This dataset was specifically curated for practitioners in a dance studio setting by using a total of four cameras to record dances. Expert annotations were obtained from various perspectives to provide a comprehensive evaluation. This study also proposes three unique automatic evaluation models. A comparative analysis of the models, particularly contrastive learning (and autoencoder)-based expression learning and a reference-guided model (where a model dancer’s performance serves as a reference), revealed that the reference-guided model achieved superior accuracy. The proposed method was able to predict dance performance ratings with an accuracy of approximately ±1 point on a 10-point scale, compared to ratings by professional coaches. Our findings open up novel possibilities for future dance training and evaluation systems.

The impact of virtual and augmented reality on presence, user experience and performance of Information Visualisation

The fast growth of virtual reality (VR) and augmented reality (AR) head-mounted displays provides a new medium for interactive visualisations and visual analytics. Presence is the experience of consciousness within extended reality, and it has the potential to increase task performance. This project studies the impact that a sense of presence has on data visualisation performance and user experience under AR and VR conditions. A within-subjects design recruited 38 participants to complete interactive visualisation tasks within the novel immersive data analytics system for genomic data in AR and VR, and measured speed, accuracy, preference, presence, and user satisfaction. Open-ended user experience responses were also collected. The results implied that VR was more conducive to efficiency, effectiveness, and user experience as well as offering insight into possible cognitive load benefits for VR users.