Real-time Virtual Environment Research Articles

VAEnvGen: A Real-Time Virtual Agent Environment Generation System Based on Large Language Models

Environment plays an important role in non-verbal communication for human-virtual agent interaction. Existing research explores the influence of an agent’s appearance and attributes to enhance human-virtual agent communication. However, there is no common practice for dynamically adjusting the surrounding environments of the virtual agent. In this paper, we introduce a real-time virtual agent environment generation system (VAEnvGen), which contributes to the field by enhancing users’ content perception and improving task performance through dynamic environment adjustment. The system dynamically analyzes both the appropriate communication environment and filters the key information according to the current context. Leveraging Large Language Models, it generates a pseudo-3D background space to create an engaging atmosphere and a dynamic foreground content space for vivid key information display, thereby significantly enhancing content perception. For widespread adoption and flexibility, VAEnvGen is developed as a web application. We further evaluate the impact of VAEnvGen on content perception, user attention, and subjective satisfaction through a mixed-design user study with 50 participants. Quantitative and qualitative results reveal significant improvements in content perception, task completion time, and user satisfaction when using VAEnvGen. The system effectively redistributes user attention from subtitles and the virtual agent itself to the dynamically generated background and key foreground information, leading to a more immersive and less fatiguing user experience.

Immersive Digital Twin under ISO 23247 Applied to Flexible Manufacturing Processes

Digital twin (DT) technology provides a path for implementing cyber–physical systems (CPS) and developing smart manufacturing because they are essential tools for monitoring and controlling manufacturing processes. It is considered a vital technology in smart manufacturing and is being widely researched in academia and industry. Furthermore, the combination of DTs and immersive environments has shown great potential for integrating novel capabilities into the new generation of CPS. This research presents an architecture for implementing immersive digital twins under ISO 23247 in flexible manufacturing processes. The proposed system is based on the integration of DT technologies in conjunction with augmented reality (AR) and gesture tracking, and validation was performed in the sorting station of the MPS 500 to increase the interaction and flexibility between physical and virtual environments in real time, thus enhancing the capabilities of the DT. The methodology used for the design and implementation of the DT includes (1) general principles and requirements; (2) models with functional views based on domains and entities; (3) attributes of the observable manufacturing elements; and (4) protocols for the exchange of information between entities. The results show that the integration of these technologies improves the monitoring, control, and simulation capabilities of processes using 3D resources and immersive environments, achieving a higher level of interactivity. In addition, error detection tests were carried out, where a reduction of time was observed in the resolution of errors that may be caused by internal or external disturbances of the process, thus avoiding production delays.

Building Virtual Scene Construction and Environmental Impact Analysis Based on Image Processing

With the rapid development of computer technology, the virtual scene construction technology of image processing has gradually become a hotspot in computer research. The application of virtual scene construction technology in the construction industry is also expanding, with the intelligentization of architectural design and construction. With the rapid development of the industry, virtual technology can better provide users with diversified services and experiences. The main content of the article is as follows: (1) The article introduces image construction technology and analyzes the three-dimensional presentation of virtual technology in construction design and the diversified application of virtual technology in construction design. (2) The article introduces two virtual scene construction techniques: one is to combine the real scene obtained by taking pictures with the virtual scene by image technology processing to construct a natural virtual scene; the other is the use of geometric construction by architects. The virtual building model is drawn by the model method, and the virtual building model is thus obtained. (3) The article chooses two representative buildings as cases. The virtual platform monitors and records the movement trajectory of volunteers in the virtual environment in real time. The experimental results show that the number and shape characteristics of the escalators in the building all have a little impact on volunteers’ awareness. (4) The article sets up three control groups of normal mode, fixed skin, and variable skin. It analyzes the environmental impact of buildings from three different aspects of wind environment, thermal environment, and light environment and proposes related control measures.

Spatial Resolution of Late Reverberation in Virtual Acoustic Environments.

Late reverberation involves the superposition of many sound reflections, approaching the properties of a diffuse sound field. Since the spatially resolved perception of individual late reflections is impossible, simplifications can potentially be made for modelling late reverberation in room acoustics simulations with reduced spatial resolution. Such simplifications are desired for interactive, real-time virtual acoustic environments with applications in hearing research and for the evaluation of hearing supportive devices. In this context, the number and spatial arrangement of loudspeakers used for playback additionally affect spatial resolution. The current study assessed the minimum number of spatially evenly distributed virtual late reverberation sources required to perceptually approximate spatially highly resolved isotropic and anisotropic late reverberation and to technically approximate a spherically isotropic sound field. The spatial resolution of the rendering was systematically reduced by using subsets of the loudspeakers of an 86-channel spherical loudspeaker array in an anechoic chamber, onto which virtual reverberation sources were mapped using vector base amplitude panning. It was tested whether listeners can distinguish lower spatial resolutions of reproduction of late reverberation from the highest achievable spatial resolution in different simulated rooms. The rendering of early reflections remained unchanged. The coherence of the sound field across a pair of microphones at ear and behind-the-ear hearing device distance was assessed to separate the effects of number of virtual sources and loudspeaker array geometry. Results show that between 12 and 24 reverberation sources are required for the rendering of late reverberation in virtual acoustic environments.

Construction of force haptic reappearance system based on Geomagic Touch haptic device

Force haptic reappearance technology is considered to be one of the top ten technologies that can change human life in the future. It has broad application prospects and market demand. Most of the existing medical robots, especially the remote diagnosis and treatment robots, lack haptic feedback, or the calculation of feedback force is insufficient. Haptic reappearance technology is an effective method to solve the problem of haptic presence and improve the practicability of medical robot.The ultimate goal of the force haptic reappearance system is to let the operator feel the haptic feedback when interacting with the soft tissue model in the virtual environment in real time. Haptic device is the necessary condition to realize force haptic reappearance, and it is an essential part of the system. Its important role is to introduce the external force imposed by the operator into the virtual environment, and let the operator feel the force in the virtual environment, which effectively guarantees the operator's sense of reality and immersion when interacting with the virtual environment.Therefore, starting with the key technology of force and haptic reappearance system, this paper studies the construction of force and haptic reappearance system. Soft tissue surface model is drawn by OpenGL, and hand model is drawn by 3Ds Max. The haptic reappearance and visual feedback of soft tissue model of hand palpation are realized. The quality of feedback is evaluated. The haptic reappearance is stable and realistic, and the visual feedback is smooth. This indicates that the system has a certain application value and is worth to promote to the public.

A Real-time Interactive Tai Chi Learning System Based on VR and Motion Capture Technology

Abstract As a popular traditional Chinese exercise, Tai Chi is of great benefit to physical and mental health. Professional and face-to-face guidance is a traditional and effective way to learn Tai Chi, but it is restricted by factors such as site and time. Although the way to learn Tai Chi by watching videos is not limited by time or place, it is not convenient for learners to observe learning from multiple perspectives since it shows sequences of actions that have been recorded from fixed viewpoints. This paper proposes a real-time interactive Tai Chi learning system based on VR and motion capture technology, in which the learner wears the head-mounted display to follow the virtual coach in the scene, and can watch the demonstration of the virtual coach in all directions; the learner can also drive and watch his own avatar, and can operate the menu by natural interaction in the virtual environment in real time through wearable motion capture device, so that he can know the effects of his training and learning of each action. The actions of the avatar and virtual coach are also compared by using a motion similarity matching algorithm, to evaluate the quality of learning and training and guide the learner to adjust his motion in real time. With the help of our system, learners can practice Tai Chi in the same way as in the real environments and drive their avatars in virtual scenes, so as to achieve effective self-learning and evaluation.

Socio-Emotional Concern Dynamics in a Model of Real-Time Dyadic Interaction: Parent-Child Play in Autism.

We used a validated agent-based model—Socio-Emotional CONcern DynamicS (SECONDS)—to model real-time playful interaction between a child diagnosed with Autism Spectrum Disorders (ASD) and its parent. SECONDS provides a real-time (second-by-second) virtual environment that could be used for clinical trials and testing process-oriented explanations of ASD symptomatology. We conducted numerical experiments with SECONDS (1) for internal model validation comparing two parental behavioral strategies for stimulating social development in ASD (play-centered vs. initiative-centered) and (2) for empirical case-based model validation. We compared 2,000 simulated play sessions of two particular dyads with (second-by-second) time-series observations within 29 play sessions of a real parent-child dyad with ASD on six variables related to maintaining and initiating play. Overall, both simulated dyads provided a better fit to the observed dyad than reference null distributions. Given the idiosyncratic behaviors expected in ASD, the observed correspondence is non-trivial. Our results demonstrate the applicability of SECONDS to parent-child dyads in ASD. In the future, SECONDS could help design interventions for parental care in ASD.

A New Calibration Process for a Homogeneous Cyclorama Illumination in Virtual TV Sets

A virtual TV set combines actors and objects with computer-generated virtual environments in real time. Nowadays, this technology is widely used in television broadcasts and cinema productions. A virtual TV set consists of three main elements: the stage, the computer-system and the chroma-keyer. The stage is composed by a monochrome cyclorama (the background) in front of which actors and objects are located (the foreground). The computer-system generates the virtual elements that will form the virtual environment. The chroma-keyer combines the elements in the foreground with the computer-generated environments by erasing the monochrome background and insetting the synthetic elements using the chroma-keying technique. In order to ease the background removal, the cyclorama illumination must be diffuse and homogeneous, avoiding the hue differences that are introduced by shadows, shines and over-lighted areas. The analysis of this illumination is usually performed manually by an expert using a photometer which makes the process slow, tedious and dependent on the experience of the operator. In this paper, a new calibration process to check and improve the homogeneity of a cyclorama’s illumination by non-experts using a custom software which provides both visual information and statistical data, is presented. This calibration process segments a cyclorama image in regions with similar luminance and calculates the centroid of each of them. The statistical study of the variation in the size of the regions and the position of the centroids are the key tools used to determine the homogeneity of the cyclorama lighting.

A Computational Model to Implement Binaural Synthesis in a Hard Real-Time Auditory Virtual Environment

There is a growing interest in the development and the evaluation of real-time auditory virtual environments (AVE). The implementation of this type of simulation system in general purpose computers is a still a challenge, and there are few studies that evaluated the perceived quality of synthetized sounds of simulated acoustic scenes. To evoke in the listener a correct image of the modeling space, the system must be dynamic and interactive. That is, it must respond to the changes in the acoustic scenario produced by the listener movement, in a perceptually acceptable time and with an update rate that guarantees continuity in the reproduction of sound events. Hard real-time systems ensure that a given task runs within a given time interval, providing deterministic behavior for applications with time restrictions. In the current article, a computational model to implement binaural synthesis in a hard real-time AVE is presented and evaluated. The computer model was implemented in an open-source auralization system. Measurements and real-time simulations on a university classroom were carried out to perform a reverberation time parameters validation and a system performance evaluation. Also, measured and simulated binaural soundtracks (composed from anechoic stimuli) were compared in terms of three selected perceptual attributes for subjective evaluations of static positions. The results showed that real-time performance was acceptable according to values previously reported in the literature and that computer prediction errors for the measured parameters were within the subjective difference limens. The computational model was able to generate an AVE with an acceptable overall perceptual quality.

New distributed virtual TV set architecture for a synergistic operation of sensors and improved interaction between real and virtual worlds

A virtual TV set is a studio that is able to combine recorded actors and objects with computer generated virtual environments in real time. In order to achieve this combination seamlessly, in an ideal configuration, several elements such as cameras, objects and people should be tracked so that all their actions on the stage have a corresponding effect in the virtual world. However, in the actual professional virtual TV sets, the tracking possibilities are quite limited because of the hardware and software architecture used, which has not had a major evolution since the first prototypes presented in the nineties. This traditional architecture uses to be rigid, including just one monolithic tracking system and low levels of interactivity. In this paper, a new distributed, flexible and scalable hardware and software architecture that allows the inclusion of multiple kinds of devices in parallel is introduced. It breaks with the traditional structure of the virtual TV sets, opening the technology to an easier inclusion of new devices without the need of updating the proprietary software of the set, thus facilitating its future evolution. The design, implementation and test of this architecture, through the adaptation of a traditional virtual TV set, is presented. The tests are developed through the inclusion of modern devices (in our case Optitrack infrared cameras, Microsoft Kinect V2 and Leap Motion) that, through a synergistic operation, allow the system to solve some traditional drawbacks of this technology such as free and multiple object and camera tracking, presenter natural interaction and automatic distance keying.

Consistency and fairness in real-time distributed virtual environments: Paradigms and relationships

Distributed real-time virtual environments entail a well-known set of trade-offs resulting in a lack of state consistency across simulation nodes. This lack of consistency gives rise to a number of challenges that must be addressed by system designers to ensure users remain engaged and satisfied. This paper surveys several approaches to defining, measuring, and controlling the consistency of simulation state data and relates them to the concepts of fairness and fitness for purpose. The interplay between consistency and fairness, in relation to fitness for purpose, must be carefully considered by system engineers to ensure appropriate design trade-offs.

Modeling the Scalability of Real-Time Online Interactive Applications on Clouds

We address the scalability of Real-Time Online Interactive Applications (ROIA) on Clouds. Popular examples of ROIA include, e.g., multi-player online computer games, simulation-based e-learning, and training in real-time virtual environments. Cloud computing allows to combine ROIA’s high demands on QoE (Quality of Experience) with the requirement of efficient and economic utilization of computation and network resources. We propose a generic scalability model for ROIA on Clouds that monitors the application performance at runtime and predicts the benefit–cost ratio of load-balancing decisions: by weighting the potential benefits of particular load-balancing actions against their time and resources overhead, our model recommends whether and how often to redistribute workload or add/remove Cloud resources when the number of users changes. We describe how the scalability is modeled w.r.t. two kinds of resources – computation (CPU) and communication (network) – and how we combine these models together. We experimentally evaluate the quality of our combined model using a challenging multi-player shooter game as a use case.

ARK: Augmented Reality for Kilobots

Working with large swarms of robots has challenges in calibration, sensing, tracking, and control due to the associated scalability and time requirements. Kilobots solve this through their ease of maintenance and programming, and are widely used in several research laboratories worldwide where their low cost enables large-scale swarms studies. However, the small, inexpensive nature of the Kilobots limits their range of capabilities as they are only equipped with a single sensor. In some studies, this limitation can be a source of motivation and inspiration, while in others it is an impediment. As such, we designed, implemented, and tested a novel system to communicate personalized location-and-state-based information to each robot, and receive information on each robots' state. In this way, the Kilobots can sense additional information from a virtual environment in real time; for example, a value on a gradient, a direction toward a reference point or a pheromone trail. The augmented reality for Kilobots (ARK) system implements this in flexible base control software which allows users to define varying virtual environments within a single experiment using integrated overhead tracking and control. We showcase the different functionalities of the system through three demos involving hundreds of Kilobots. The ARK provides Kilobots with additional and unique capabilities through an open-source tool which can be implemented with inexpensive, off-the-shelf hardware.

Crowd simulation using DC model and density information

Realistic crowd simulation is an important issue for the production of virtual worlds for games, crowd management, public space design, education, entertainment or architectural and urban planning. In this paper, crowd simulation is considered from two aspects: intra-group simulation and inter-group simulation. We propose a unified framework for crowd simulation in real-time virtual environment. Based on this framework, for intra-group simulation, we propose a novel density-based information crowd simulation to collision-free. For inter-group simulation, we propose a novel discrete choice (DC) model to realistic simulation of crowds and path planning. Meanwhile, we also propose a variable bounding box method for intra-group/inter-groups intersection problem. The simulation results show that the developed framework allows different group structures to be easily modeled. And the proposed framework could be used for real-time navigation of many moving crowd in complicated virtual environments.

Recording Auditory and Visual Evoked Responses With Mobile EEG Systems in a Computer Assisted Rehabilitation Environment (CAREN)

The Computer Assisted Rehabilitation Environment (CAREN) is a novel, real-time virtual reality environment for recording, evaluation, and training of functional human behavior. Advances in the field of neuroscience have demonstrated that recording electroencephalography (EEG) in such a dynamic setting is necessary for studying brain functions and critical in evaluating and rehabilitating human behaviors after brain injuries. Advances in technology over the last few years have made it feasible to record EEG on human subjects performing cognitive tasks in a dynamic setting with real-time actions, although this remains to be tested extensively. The purpose of this study was to test the performance of three mobile EEG systems integrated into the CAREN on subjects performing classic auditory and visual oddball tasks in resting and active states. The recorded EEG signals were processed off-line to remove movement artifact by using algorithms representing state-of-the-art techniques of digital signal processing. Results revealed similar patterns of evoked responses whether the subjects were sitting or walking while performing the oddball tasks, demonstrating the feasibility and viability of recording EEG in a virtual reality environment with real-time actions. Implementation of mobile EEG recording systems into the CAREN system will offer unique opportunities to study cognitive brain function in a dynamic virtual environment and to develop effective methods for behavior and functional rehabilitation after brain injuries.

Simulated Training System for Undersea Oil Spill Emergency Response

This paper describes a simulated training system supported by Virtual Environment in real time for undersea oil spill emergency response, which provides all the functionalities to train emergency workers. This system aims to establish a highly immersed sense of 3D oil spill interactive environment based on “The semi physical simulation system of undersea emergency maintenance”. The research in this paper provides a new solution for undersea oil spill accident treatment. At present the system has been completed the designs of system architecture, hardware structure, software platform and interface.

Non-parametric modeling and editing for captured SVBRDF

Reflectance properties of real-world objects can be represented by spatially varying bidirectional reflectance distribution function (SVBRDF). Non-parametric BRDF becomes the main aspect nowadays because of its verisimilitude and generality. We present a new method for capturing, modeling and editing non-parametric SVBRDF. Our method seeks to achieve high reconstruction accuracy, compactness and editability of representation, and meanwhile to speed up the SVBRDF modeling processes. For a planar surface, we (1) design a capturing device to acquire reflectance samples of the surface; (2) propose a Laplacian-based angular interpolation scheme for a 2D slice of BRDF at a given surface location, and a kernel Nyström method for SVBRDF data matrix reconstruction; (3) propose a practical algorithm to extract linearly independent basis BRDFs, and to calculate blending weights by projecting reconstructed reflectance onto these basis BRDFs; (4) decompose these basis BRDFs into groups of 1D curves for editing intuitively. Our experiment results demonstrate that our approach can model real-world reflectance with both high accuracy and high visual fidelity for real-time virtual environment rendering.

Validation of a Method for Real Time Foot Position and Orientation Tracking With Microsoft Kinect Technology for Use in Virtual Reality and Treadmill Based Gait Training Programs

The use of virtual reality for the provision of motor-cognitive gait training has been shown to be effective for a variety of patient populations. The interaction between the user and the virtual environment is achieved by tracking the motion of the body parts and replicating it in the virtual environment in real time. In this paper, we present the validation of a novel method for tracking foot position and orientation in real time, based on the Microsoft Kinect technology, to be used for gait training combined with virtual reality. The validation of the motion tracking method was performed by comparing the tracking performance of the new system against a stereo-photogrammetric system used as gold standard. Foot position errors were in the order of a few millimeters (average RMSD from 4.9 to 12.1 mm in the medio-lateral and vertical directions, from 19.4 to 26.5 mm in the anterior-posterior direction); the foot orientation errors were also small (average %RMSD from 5.6% to 8.8% in the medio-lateral and vertical directions, from 15.5% to 18.6% in the anterior-posterior direction). The results suggest that the proposed method can be effectively used to track feet motion in virtual reality and treadmill-based gait training programs.

Combining peer-to-peer and cloud computing for large scale on-line games

No abstract available.

Integration of Tree Models into a Real-Time Pollution Dispersion Simulation

For exploring the effect of particle dispersion and wind simulation in an urban surrounding. The tree was constructed with smaller numbers of polygons for saving the rendering time in real-time virtual environment simulation system. It was integrated into a real time pollution dispersion simulation system. Also a way was used for simulating the wind tree. The results show that this method is proper and efficient for simulating tree animation in real-time virtual environment simulation system.