Immersive Human-computer Interaction Research Articles

Immersive human–computer interaction and digital entertainment new media application in English e-learning mode

With the rapid development of e-learning, more and more learners use electronic devices to learn English. The traditional way of learning is often lack of interest and interaction, therefore, this paper discusses how to improve the effect of English learning through the application of immersive human–computer interaction and digital entertainment new media. Research on the selection of appropriate virtual reality, augmented reality or game platforms, design interactive content and scenes according to learning objectives, and use elements such as virtual scenes, virtual characters and audio-visual effects to improve learners’ immersion. Through careful design and production, learners can be presented with a lively, interesting and challenging learning environment to stimulate their interest and enthusiasm in learning. Develop interactive learning tools and applications that enable learners to interact with virtual environments or games, make appropriate adjustments and feedback based on learner feedback and behavior, and provide personalized learning support and guidance. Through the collection of learner feedback and behavioral data, combined with the evaluation indicators of learning outcomes, the immersive learning environment and content can be evaluated and improved and optimized according to the evaluation results, so as to further enhance learners’ learning motivation and learning outcomes.

Research on the visualization of information of Chinese traditional music with human-computer interaction from the perspective of metaverse

<p>In the metaverse environment, establish an immersive human-computer interaction system for Chinese traditional music based on virtual reality technology. Design the system’s functionality according to the Y model, and construct a four-layered system architecture. Collect high-quality instructional audio and utilize polygon modeling technology to create contextualized scenes of Chinese traditional music, as well as high-fidelity models of characters and instruments. Implement motion capture through inertial sensor technology for performance action data mapping. Utilize a metaverse engine platform to realize interactive functions and conduct performance optimization. The system is capable of eliciting learners’ intrinsic experiences, enabling interactive self-directed learning and creative exploration of Chinese traditional music performance, demonstrating significant practical value.</p>

Virtual-Reality Interpromotion Technology for Metaverse: A Survey

The Metaverse aims to build an immersive virtual reality world to support the daily life, work, and recreation of people. In this survey, the status quo of the metaverse is investigated, and the technical framework of the metaverse is introduced from three aspects: the generation of virtual worlds, the connection of virtual and real objects, and the transmission of data. Specifically, this survey first discusses the development and challenges of the related technologies for virtual world generation methods from three aspects: the 3D world generation, immersive human-computer interaction experience, and ecosystem. Second, we investigate the status quo of extended reality (XR), motion capture, and brain-computer interface technologies, and evaluate the potential and research directions of these entrance technologies for the metaverse. Finally, network and data transmission technologies for the metaverse are reviewed from the internet of things (IoT), 5G/6G wireless, and edge computing aspects, the demand side of the metaverse in virtual-reality inter-promotion, big data processing, and low-latency networking is discussed, and promising research hotspots are identified.

HaptGlove-Untethered Pneumatic Glove for Multimode Haptic Feedback in Reality-Virtuality Continuum.

Novel haptics technologies are urgently needed to bridge the gap between entirely physical world and fully digital environment to render a more realistic and immersive human-computer interaction. Current virtual reality (VR) haptic gloves either deliver limited haptic feedback or are bulky and heavy. The authors develop a haptic glove or HaptGlove, an untethered and lightweight pneumatic glove, that allows users to "physically" interact in a VR environment and enables both kinesthetic and cutaneous sensations naturally and realistically. Integrated with five pairs of haptic feedback modules and fiber sensors, HaptGlove provides variable stiffness force feedback and fingertip force and vibration feedback, allowing users to touch, press, grasp, squeeze, and pull various virtual objects and feel the dynamic haptic changes. Significant improvements in VR realism and immersion areobserved in a user study with participants achieving 78.9% accuracy in sorting six virtual balls of different stiffnesses. Importantly, HaptGlove facilitates VR training, education, entertainment, and socialization in a reality-virtuality continuum.

Background Augmentation Generative Adversarial Networks (BAGANs): Effective Data Generation Based on GAN-Augmented 3D Synthesizing

Augmented Reality (AR) is crucial for immersive Human–Computer Interaction (HCI) and the vision of Artificial Intelligence (AI). Labeled data drives object recognition in AR. However, manually annotating data is expensive, labor-intensive, and data distribution asymmetry. Scantily labeled data limits the application of AR. Aiming at solving the problem of insufficient and asymmetry training data in AR object recognition, an automated vision data synthesis method, i.e., background augmentation generative adversarial networks (BAGANs), is proposed in this paper based on 3D modeling and the Generative Adversarial Network (GAN) algorithm. Our approach has been validated to have better performance than other methods through image recognition tasks with respect to the natural image database ObjectNet3D. This study can shorten the algorithm development time of AR and expand its application scope, which is of great significance for immersive interactive systems.

Integrated 3D display and imaging using dual purpose passive screen and head-mounted projectors and camera.

We propose an integrated 3D display and imaging system using a head-mounted device and a special dual-purpose passive screen that can simultaneously facilitate 3D display and imaging. The screen is mainly composed of two optical layers, the first layer is a projection surface, which are the finely patterned retro-reflective microspheres that provide high optical gain when illuminated with head-mounted projectors. The second layer is an imaging surface made up of an array of curved mirrors, which form the perspective views of the scene captured by a head-mounted camera. The display and imaging operation are separated by performing polarization multiplexing. The demonstrated prototype system consists of a head-worn unit having a pair of 15 lumen pico-projectors and a 24MP camera, and an in-house designed and fabricated 30cm × 24cm screen. The screen provides bright display using 25% filled retro-reflective microspheres and 20 different perspective views of the user/scene using 5 × 4 array of convex mirrors. The real-time implementation is demonstrated by displaying stereo-3D content providing high brightness (up to 240 cd/m2) and low crosstalk (<4%), while 3D image capture is demonstrated by performing the computational reconstruction of the discrete free-viewpoint stereo pair displayed on a desktop or virtual reality display. Furthermore, the capture quality is determined by measuring the imaging MTF of the captured views and the capture light efficiency is calculated by considering the loss in transmitted light at each interface. Further developments in microfabrication and computational optics can present the proposed system as a unique mobile platform for immersive human-computer interaction of the future.

Immersive human–computer interactive virtual environment using large-scale display system

Large-scale display system with immersive human–computer interaction (HCI) is an important solution for virtual reality (VR) systems. In contrast to the traditional human–computer interactive VR system that requires the user to wear heavy VR headsets for visualization and data gloves for HCI, the proposing method utilizes a large-scale display screen (with or without 3D glasses) to visualize the virtual environment and a bare-handed gesture recognition solution to receive user instructions. The entire framework is named as an immersive HCI system. Through a virtual 3D interactive rectangular parallelepipe, we establish the correspondence between the virtual scene and the control information. A bare-handed gesture recognition method is presented based on extended genetic algorithm. An arm motion estimation method is designed based on the fuzzy predictive control theory. Experimental results showed that the proposed method has lower error rates than most existing solutions with acceptable recognition frequency rates.

Immersive Computing Technology to Investigate Tradeoffs Under Uncertainty in Disassembly Sequence Planning

The scientific and industrial communities have begun investigating the possibility of making product recovery economically viable. Disassembly sequence planning may be used to make end-of-life product take-back processes more cost effective. Much of the research involving disassembly sequence planning relies on mathematical optimization models. These models often require input data that is unavailable or can only be approximated with high uncertainty. In addition, there are few mathematical models that include consideration of the potential of product damage during disassembly operations. The emergence of Immersive Computing Technologies (ICT) enables designers to evaluate products without the need for physical prototypes. Utilizing unique 3D user interfaces, designers can investigate a multitude of potential disassembly operations without resorting to disassembly of actual products. The information obtained through immersive simulation can be used to determine the optimum disassembly sequence. The aim of this work is to apply a decision analytical approach in combination with immersive computing technology to optimize the disassembly sequence while considering trade-offs between two conflicting attributes: disassembly cost and damage estimation during disassembly operations. A wooden Burr puzzle is used as an example product test case. Immersive human computer interaction is used to determine input values for key variables in the mathematical model. The results demonstrate that the use of dynamic programming algorithms coupled with virtual disassembly simulation is an effective method for evaluating multiple attributes in disassembly sequence planning. This paper presents a decision analytical approach, combined with immersive computing techniques, to optimize the disassembly sequence. Future work will concentrate on creating better methods of estimating damage in virtual disassembly environments and using the immersive technology to further explore the feasible design space.

Immersive manipulation of virtual objects through glove-based hand gesture interaction

Immersive visualisation is increasingly being used for comprehensive and rapid analysis of objects in 3D and object dynamic behaviour in 4D. Challenges are therefore presented to provide natural user interaction to enable effortless virtual object manipulation. Presented in this paper is the development and evaluation of an immersive human–computer interaction system based on stereoscopic viewing and natural hand gestures. For the development, it is based on the integration of a back-projection stereoscopic system for object and hand display, a hybrid inertial and ultrasonic tracking system to provide the absolute positions and orientations of the user’s head and hands, as well as a pair of high degrees-of-freedom data gloves to provide the relative positions and orientations of digit joints and tips on both hands. For the evaluation, it is based on a two-object scene with a virtual cube and a CT (computed tomography) volume created for demonstration of real-time immersive object manipulation. The system is shown to provide a correct user view of objects and hands in 3D with depth, as well as to enable a user to use a number of simple hand gestures to perform basic object manipulation tasks involving selection, release, translation, rotation and scaling. Also included in the evaluation are some quantitative tests of the system performance in terms of speed and latency.