Interactive Virtual Reality Research Articles

Interacting with virtual characters, objects and environments: investigating immersive virtual reality in rehabilitation

Purpose This pilot study aimed to (a) investigate opportunities for immersive Virtual Reality (VR) technology in communication, physical, and visual rehabilitation by examining the interaction of people without disabilities in a range of structured virtual environments; and (b) validate research protocols that might be used to evaluate the physical, visual, and verbal interaction of users in virtual worlds, and their safety while using the technology. Methods Thirteen adults identifying as people without disability were exposed to VR via a head-mounted display. A video-review method was used to qualitatively code and analyse each participant’s communication, movement, orientation, and support needs. Results All participants oriented to their virtual environments sufficiently to use applications. Their spoken language was effective for interaction, although unconventional social behaviours were also observed. Two participants reported minor adverse reactions consistent with mild cybersickness. Conclusion The results provide insight into the types of environments and characters that support the greatest communicative, physical, and visual interaction in immersive VR. The tested protocols are useful to assess safety when using VR, and to observe communicative, physical, and visual interaction with virtual environments and characters. Implications for future research and use of VR with people with communication, physical and visual disability are discussed.

Towards believable and educational conversations with virtual patients

Virtual Reality (VR) technology allows the design and application of realistic but adaptive learning environments in medical education. In particular, virtual patient systems have logistical and methodological advantages compared to non-computerized interventions. However, evidence for their effectiveness is fragmented as any educational domain introduces its requirements regarding learning goals, measurements of learning outcomes, and application design. In this context, we present preliminary results of evaluating a VR training application for conducting a clinical interview to diagnose mental disorders in children and adolescents using virtual patients. The evaluation focuses on design elements related to the virtual patient’s appearance and natural language capabilities. Our results indicate that our virtual patient design is highly believable and that our dialog system is satisfying. However, conversational flow requires optimization. We discuss design directions and potential enhancements for learner-virtual patient interactions in VR and address future operations to evaluate the effectiveness of our approach.

GazeIntent: Adapting Dwell-time Selection in VR Interaction with Real-time Intent Modeling

The use of ML models to predict a user's cognitive state from behavioral data has been studied for various applications which includes predicting the intent to perform selections in VR. We developed a novel technique that uses gaze-based intent models to adapt dwell-time thresholds to aid gaze-only selection. A dataset of users performing selection in arithmetic tasks was used to develop intent prediction models (F1 = 0.94). We developed GazeIntent to adapt selection dwell times based on intent model outputs and conducted an end-user study with returning and new users performing additional tasks with varied selection frequencies. Personalized models for returning users effectively accounted for prior experience and were preferred by 63% of users. Our work provides the field with methods to adapt dwell-based selection to users, account for experience over time, and consider tasks that vary by selection frequency.

Advances in emerging digital technologies for energy efficiency and energy integration in smart cities

Advances and fast development in emerging digital technologies trigger the next generation revolution in energy areas and smart cities, while roles and mechanisms of digital technologies for smart and sustainable transition is unclear. Furthermore, energy flexibility enhancement in intermittent renewable-stochastic demand power management and survival capability in minimizing frequency of power outrage are still not clear, when suffering from climate change and extreme events with emerging digital technologies. In this study, advances in emerging digital technologies have been systematically and comprehensively reviewed, in terms of current development status and mechanisms for energy efficiency and energy integration in energy-efficient systems. Afterwards, roles of energy digitalization technologies are provided for high-efficiency, low-carbon and intelligent building energy systems, including artificial intelligence for dynamic performance predictions, advanced model predictive controls and optimisations of nonlinear systems (e.g., PVs, heat pumps, heat recovery systems and multi-energy storages). Furthermore, digital twin-based building energy digitalization technologies are applied for 3D modeling, monitoring, real-time visualization and virtual reality interaction. Frontier multi-agent based distributed energy systems are comprehensively proposed with multi-agent energy management, including RE-battery-building-EV, RE-building-hydrogen vehicles, and both centralised and distributed energy management systems. Considering the multi-energy system capability for power management (intermittent renewable energy and energy demands) and survival capability when suffering from high-impact and low-probability events, both energy flexibility and energy resilience with energy digitalization technologies are interconnected, for climate change adaption and internet of energy. This study provides a systematic and comprehensive review on emerging digital technologies for energy efficiency and energy integration in smart cities, providing guidelines on sustainable and smart transitions with multi-agent based distributed energy management and energy digitalization technologies.

A Proof-of-Concept of an Integrated VR and AI Application to Develop Classroom Management Competencies in Teachers in Training

We designed an interactive virtual reality (VR) application to provide a controlled and yet unpredictable environment for the development of classroom management skills. The simulated environment allows teachers in training to interact with virtual students in realistic and meaningful ways. The VR application allows rich verbal interaction by using artificial intelligence (AI). Initial findings suggest it is a successful proof of concept. In this paper, we focus on the technical implementation. Predictions on educational effectiveness and the educational challenges of pre-service teacher education are discussed. Future developments include rigorous testing and incorporating non-verbal communication based on a multi-dimensional interpersonal behavior model.

Developing a virtual reality and AI-based framework for advanced digital manufacturing and nearshoring opportunities in Mexico.

The growing expansion of the manufacturing sector, particularly in Mexico, has revealed a spectrum of nearshoring opportunities yet is paralleled by a discernible void in educational tools for various stakeholders, such as engineers, students, and decision-makers. This paper introduces a state-of-the-art framework, incorporating virtual reality (VR) and artificial intelligence (AI) to metamorphose the pedagogy of advanced manufacturing systems. Through a case study focused on the design, production, and evaluation of a robotic platform, the framework endeavors to offer an exhaustive educational experience via an interactive VR environment, encapsulating (1) Robotic platform system design and modeling, enabling users to immerse themselves in the design and simulation of robotic platforms under varied conditions; (2) Virtual manufacturing company, presenting a detailed virtual manufacturing setup to enhance users' comprehension of manufacturing processes and systems, and problem-solving in realistic settings; and (3) Product evaluation, wherein users employ VR to meticulously assess the robotic platform, ensuring optimal functionality and customer satisfaction. This innovative framework melds theoretical acumen with practical application in advanced manufacturing, preparing entities to navigate Mexico's manufacturing sector's vibrant and competitive nearshoring landscape. It creates an immersive environment for understanding modern manufacturing challenges, fostering Mexico's manufacturing sector growth, and maximizing nearshoring opportunities for stakeholders.

The effect of virtual tourism experience on tourist responses: the lens from cognitive appraisal theory

ABSTRACT Virtual reality (VR) can enhance a tourist’s travel experience and the effectiveness of destination marketing. Drawing on cognitive appraisal theory, this study developed a theoretical framework that examined VR tourism interaction, immersion, sense of presence in relation to pleasure and arousal, and the effect of these emotional responses on behavioral intention. Our results showed that VR interactivity, immersion, and sense of presence influence users’ emotional responses, however, the interactive experience has no positive effect on arousal, and arousal has no positive effect on behavioral intention. Our study contributes to the development of cognitive appraisal theory and suggests valuable practical recommendations for tourism marketing managers.

An exploratory study on practising listening comprehension skills in high‐immersion virtual reality

AbstractHolding learners' attention is challenging, especially when they are asked to listen to long passages. High‐immersion virtual reality (VR) can immerse learners in listening tasks, even in such complex languages as Chinese. This exploratory study examined the effect of VR on 43 Chinese language learners' listening comprehension, enjoyment, sense of presence, and cognitive load. Participants were self‐selected into two groups without knowing the purpose of the study or the details of their activity. The experimental group (n1 = 23) experienced an interactive multimedia story in VR, and the comparison group (n2 = 20) watched a screencast video recording of the same story. Multivariate analysis of covariance (MANCOVA) indicated that VR may have a positive effect on the development of listening skills. Compared with the video group, the VR group had significantly higher listening comprehension scores, reported significantly more enjoyment and sense of presence, and reported experiencing less cognitive load. Thus, the findings suggest that VR could be used as a pedagogical tool to enhance foreign language listening skills.Practitioner notesWhat is already known about this topic Practising listening skills is key for language development. Virtual reality (VR) is motivational and engaging. Research on practising listening skills in VR is scarce. What this paper adds VR can be beneficial for practising listening skills. Language learners highly enjoy practising listening in VR. Interactivity can improve engagement and support listening comprehension. Implications for practice and/or policy Learners should be provided pretraining and scaffolding prior to using VR. The duration of VR activities should be considered. Interactivity in VR may increase its effectiveness in attending listening tasks.

Bridging theory and practice: CFD simulation and interactive VR for conduction heat transfer learning

AbstractSoftware simulation programs and virtual reality (VR) have become powerful tools for several educational purposes, and recently, they were used in a wide range of applications. In cases of inaccessibility to labs, workshops, or industries, as happened before in the coronavirus disease 2019 pandemic, these tools could be effectively integrated with practical lab experiments. In this study, a computational fluid dynamics (CFD) simulation and a VR module were utilized to simulate heat transfer by conduction through various experimental cases. The CFD simulation results were obtained using ANSYS/FLUENT software. Meantime, the experimental data were obtained by carrying out three experiments of heat conduction with different heat transfer rates through simple, composite, and different cross‐sectional area bars. At last, the experimental procedure and devices used were virtually constructed using SolidWorks software as three‐dimensional models, which were then extruded into VR and augmented reality models. It was found that the simulation results closely align with the experimental ones, and the temperature profile in both cases has the same behavior with small differences, which indicates the validity of the developed module to be used as a simulation of the actual experiments. In addition to improving knowledge of heat transfer principles, this combination of simulation and VR technology advances many Sustainable Development Goals (SDGs), including advancing quality education (SDG 4) and innovation in higher education (SDG 9). Additionally, this method assists in achieving the course learning objectives by mimicking real‐world lab experiments, guaranteeing that students graduate from the course with the required information and abilities.

, virtual reality embodying philosophical and conceptual heritage

ABSTRACT In this paper, we introduce the <Sunghaksipdo VR> project, which is based on Sunghaksipdo (聖學十圖). Known as the Ten Diagrams on Sage Learning, Sunghaksipdo was a heritage created by Toegye, the Korean Confucian scholar of the Joseon Dynasty. While conveying philosophical concepts such as Li (理) and Gyung (敬) and their spiritual values of mindfulness from the heritage to VR, we have focused on reconnecting the original visual diagrams and texts into perceivable VR experiences. We approached this task through the following: (1) by constructing conceptual and philosophical VR space and environments based on the original diagrams, (2) by creating conceptual objects inside VR, and (3) by designing diverse user interactions with such virtual conceptual objects. Particularly the interactions allow the user’s embodied experience and performative engagement, which become a way of understanding the meaning of the philosophical message. While explaining these with diverse exemplary cases, we hope to show how new media and interactive VR technologies can be used to realize philosophical concepts and spiritual value. Through this, we hope to discuss how VR could be imagined as a tangible philosophy discovering the plane of immanence.

The Aural Eye

Human aural experience can be equally considered along spatial and temporal continuums. We hear at all times of the day and night, and within all places and spaces: built, natural, public, private and virtual. The territory – both physical and philosophical – between music composition for interior listening and traditional sound based research within environmental acoustics is gradually being occupied by the listener-centred approach of soundscape studies. Soundscape design is emerging as an interdisciplinary field within design education, and one that not only challenges the ocular-centric nature of most design education, but one that could provide a useful mode through which to investigate the coincidences between different design disciplines. This paper draws on the author’s own practice as a sound designer in a variety of spatial sound projects in built and virtual contexts to discuss ideas of landscape, interiority, space and place as experienced through listening. This will include aspects of Canopies: chimerical acoustic environments for the Southgate soundscape system, Ecstasis: human presence in digital environments for an interactive VR system and stereoscopic projections, and The Occupation of Space: Soundsites project with the Melbourne blind community.2 The ideas and technologies underpinning these projects also form the basis of a new pedagogy of sound and listening housed in the Spatial Information Architecture Laboratory’s (SIAL) Sound Studios at RMIT University. The place and role of the Sound Studio’s program in providing an aural perspective that compliments the visual methodologies of co-located design disciplines is discussed.

Augmenting collaborative interaction with shared visualization of eye movement and gesture in VR

AbstractVirtual Reality (VR)‐enabled multi‐user collaboration has been gradually applied in academic research and industrial applications, but it still has key problems. First, it is often difficult for users to select or manipulate objects in complex three‐dimesnional spaces, which greatly affects their operational efficiency. Second, supporting natural communication cues is crucial for cooperation in VR, especially in collaborative tasks, where ambiguous verbal communication cannot effectively assign partners the task of selecting or manipulating objects. To address the above issues, in this paper, we propose a new interaction method, Eye‐Gesture Combination Interaction in VR, to enhance the execution of collaborative tasks by sharing the visualization of eye movement and gesture data among partners. We conducted user experiments and showed that using dots to represent eye gaze and virtual hands to represent gestures can help users complete tasks faster than other visualization methods. Finally, we developed a VR multi‐user collaborative assembly system. The results of the user study show that sharing gaze points and gestures among users can significantly improve the productivity of collaborating users. Our work can effectively improve the efficiency of multi‐user collaborative systems in VR and provide new design guidelines for collaborative systems in VR.

Design of a lightweight and easy‐to‐wear hand glove with multi‐modal tactile perception for digital human

AbstractWithin the field of human–computer interaction, data gloves play an essential role in establishing a connection between virtual and physical environments for the realization of digital human. To enhance the credibility of human‐virtual hand interactions, we aim to develop a system incorporating a data glove‐embedded technology. Our proposed system collects a wide range of information (temperature, bending, and pressure of fingers) that arise during natural interactions and afterwards reproduce them within the virtual environment. Furthermore, we implement a novel traversal polling technique to facilitate the streamlined aggregation of multi‐channel sensors. This mitigates the hardware complexity of the embedded system. The experimental results indicate that the data glove demonstrates a high degree of precision in acquiring real‐time hand interaction information, as well as effectively displaying hand posture in real‐time using Unity3D. The data glove's lightweight and compact design facilitates its versatile utilization in virtual reality interactions.

PetPresence: Investigating the Integration of Real-World Pet Activities in Virtual Reality.

For VR interaction, the home environment with complicated spatial setup and dynamics may hinder the VR user experience. In particular, pets' movement may be more unpredictable. In this paper, we investigate the integration of real-world pet activities into immersive VR interaction. Our pilot study showed that the active pet movements, especially dogs, could negatively impact users' performance and experience in immersive VR. We proposed three different types of pet integration, namely semitransparent real-world portal, non-interactive object in VR, and interactive object in VR. We conducted the user study with 16 pet owners and their pets. The results showed that compared to the baseline condition without any pet-integration technique, the approach of integrating the pet as interactive objects in VR yielded significantly higher participant ratings in perceived realism, joy, multisensory engagement, and connection with their pets in VR.

Privacy-Preserving Gaze Data Streaming in Immersive Interactive Virtual Reality: Robustness and User Experience.

Eye tracking is routinely being incorporated into virtual reality (VR) systems. Prior research has shown that eye tracking data, if exposed, can be used for re-identification attacks [14]. The state of our knowledge about currently existing privacy mechanisms is limited to privacy-utility trade-off curves based on data-centric metrics of utility, such as prediction error, and black-box threat models. We propose that for interactive VR applications, it is essential to consider user-centric notions of utility and a variety of threat models. We develop a methodology to evaluate real-time privacy mechanisms for interactive VR applications that incorporate subjective user experience and task performance metrics. We evaluate selected privacy mechanisms using this methodology and find that re-identification accuracy can be decreased to as low as 14% while maintaining a high usability score and reasonable task performance. Finally, we elucidate three threat scenarios (black-box, black-box with exemplars, and white-box) and assess how well the different privacy mechanisms hold up to these adversarial scenarios. This work advances the state of the art in VR privacy by providing a methodology for end-to-end assessment of the risk of re-identification attacks and potential mitigating solutions. f.

Enhancing surgical training through immersive and interactive virtual reality experiences in robotic surgery

Enhancing surgical training through immersive and interactive virtual reality experiences in robotic surgery

Multimodal Physiological Analysis of Impact of Emotion on Cognitive Control in VR.

Cognitive control is often perplexing to elucidate and can be easily influenced by emotions. Understanding the individual cognitive control level is crucial for enhancing VR interaction and designing adaptive and self-correcting VR/AR applications. Emotions can reallocate processing resources and influence cognitive control performance. However, current research has primarily emphasized the impact of emotional valence on cognitive control tasks, neglecting emotional arousal. In this study, we comprehensively investigate the influence of emotions on cognitive control based on the arousal-valence model. A total of 26 participants are recruited, inducing emotions through VR videos with high ecological validity and then performing related cognitive control tasks. Leveraging physiological data including EEG, HRV, and EDA, we employ classification techniques such as SVM, KNN, and deep learning to categorize cognitive control levels. The experiment results demonstrate that high-arousal emotions significantly enhance users' cognitive control abilities. Utilizing complementary information among multi-modal physiological signal features, we achieve an accuracy of 84.52% in distinguishing between high and low cognitive control. Additionally, time-frequency analysis results confirm the existence of neural patterns related to cognitive control, contributing to a better understanding of the neural mechanisms underlying cognitive control in VR. Our research indicates that physiological signals measured from both the central and autonomic nervous systems can be employed for cognitive control classification, paving the way for novel approaches to improve VR/AR interactions.

Assessing Learning in an Immersive Virtual Reality: A Curriculum-Based Experiment in Chemistry Education

Despite the recent advances in Virtual Reality technology and its use in education, the review of the literature shows several gaps in research on how immersive virtual environments impact the learning process. In particular, the lack of curriculum-specific experiments along with investigations of the effects of different content, activity, and interaction types in the current VR studies has been identified as a significant shortcoming. This has been more significant in STEM fields, where VR has the potential to offer engaging experiential learning opportunities. The study reported here was designed to address this gap by assessing the effect of authentic visualization and interaction types on learning a particular scientific concept. A use case scenario of “orbital hybridization” in chemistry education was selected to create this experiment and to collect data for analysis. We collected data on learning outcomes, task-completion efficiency, accuracy, and subjective usability. A combination of learning content and tasks designed based on the relevant educational theories was presented to three groups: 2D, VR interaction type 1 (hand gestures), and VR interaction type 2 (ray casting). The results showed that VR could improve learning and that interaction type could influence efficiency and accuracy depending on the task.

Impact of interactive learning elements on personal learning performance in immersive virtual reality for construction safety training

To improve construction safety through proactive prevention strategies, it is essential to leverage virtual reality (VR) training programs that foster active learning through dynamic interaction between VR systems and learners. Addressing this need, this study proposed an interactive immersive VR (IVR)-based construction safety training framework, incorporating four interactive learning elements (ILEs): immediate feedback, basic interaction with objects, assembling objects, and knowledge testing. This study divided sixty trainees into two groups: one experienced the proposed novel interactive IVR-based training approach, while the other underwent traditional, non-interactive IVR-based training. To accurately evaluate the impact of the four ILEs on individual learning performance, this study conducted t-test analyses and utilized machine learning-based SHAP (Shapley Additive exPlanations) analysis to compare the results between both groups. The findings indicated that the proposed method significantly enhanced active learning, positively influencing the sensory and knowledge domains of trainees more effectively than the existing methods. Notably, 'ILE-(1). Immediate feedback' and 'ILE-(2). Basic interaction with objects' were identified as key factors in improving personal learning outcomes during training. This study contributes significantly to the field of management in engineering, demonstrating that interactive IVR-based training can offer a superior learning experience and enhance safety performance, thereby reducing accident rates and creating safer construction sites.

Immersive Storytelling in Social Virtual Reality for Human-Centered Learning about Sensitive Historical Events

History is a subject that students often find uninspiring in school education. This paper explores the application of social VR metaverse platforms in combination with interactive, nonlinear web platforms designed for immersive storytelling to support learning about a sensitive historical event, namely the Asia Minor Catastrophe. The goal was to design an alternative method of learning history and investigate if it would engage students and foster their independence. A mixed-methods research design was applied. Thirty-four (n = 34) adult participants engaged in the interactive book and VR space over the course of three weeks. After an online workshop, feedback was collected from participants through a custom questionnaire. The quantitative data from the questionnaire were analyzed statistically utilizing IBM SPSS, while the qualitative responses were coded thematically. This study reveals that these two tools can enhance historical education by increasing student engagement, interaction, and understanding. Participants appreciated the immersive and participatory nature of the material. This study concludes that these technologies have the potential to enhance history education by promoting active participation and engagement.