Multimodal Virtual Environment Research Articles

Identifying Complex Emotions in Alexithymia Affected Adolescents Using Machine Learning Techniques.

Many scientific researchers' study focuses on enhancing automated systems to identify emotions and thus relies on brain signals. This study focuses on how brain wave signals can be used to classify many emotional states of humans. Electroencephalography (EEG)-based affective computing predominantly focuses on emotion classification based on facial expression, speech recognition, and text-based recognition through multimodality stimuli. The proposed work aims to implement a methodology to identify and codify discrete complex emotions such as pleasure and grief in a rare psychological disorder known as alexithymia. This type of disorder is highly elicited in unstable, fragile countries such as South Sudan, Lebanon, and Mauritius. These countries are continuously affected by civil wars and disaster and politically unstable, leading to a very poor economy and education system. This study focuses on an adolescent age group dataset by recording physiological data when emotion is exhibited in a multimodal virtual environment. We decocted time frequency analysis and amplitude time series correlates including frontal alpha symmetry using a complex Morlet wavelet. For data visualization, we used the UMAP technique to obtain a clear district view of emotions. We performed 5-fold cross validation along with 1 s window subjective classification on the dataset. We opted for traditional machine learning techniques to identify complex emotion labeling.

Classification of Complex Emotions Using EEG and Virtual Environment: Proof of Concept and Therapeutic Implication.

During the last decades, neurofeedback training for emotional self-regulation has received significant attention from scientific and clinical communities. Most studies have investigated emotions using functional magnetic resonance imaging (fMRI), including the real-time application in neurofeedback training. However, the electroencephalogram (EEG) is a more suitable tool for therapeutic application. Our study aims at establishing a method to classify discrete complex emotions (e.g., tenderness and anguish) elicited through a near-immersive scenario that can be later used for EEG-neurofeedback. EEG-based affective computing studies have mainly focused on emotion classification based on dimensions, commonly using passive elicitation through single-modality stimuli. Here, we integrated both passive and active elicitation methods. We recorded electrophysiological data during emotion-evoking trials, combining emotional self-induction with a multimodal virtual environment. We extracted correlational and time-frequency features, including frontal-alpha asymmetry (FAA), using Complex Morlet Wavelet convolution. Thinking about future real-time applications, we performed within-subject classification using 1-s windows as samples and we applied trial-specific cross-validation. We opted for a traditional machine-learning classifier with low computational complexity and sufficient validation in online settings, the Support Vector Machine. Results of individual-based cross-validation using the whole feature sets showed considerable between-subject variability. The individual accuracies ranged from 59.2 to 92.9% using time-frequency/FAA and 62.4 to 92.4% using correlational features. We found that features of the temporal, occipital, and left-frontal channels were the most discriminative between the two emotions. Our results show that the suggested pipeline is suitable for individual-based classification of discrete emotions, paving the way for future personalized EEG-neurofeedback training.

IMVEST, an immersive multimodal virtual environment stress test for humans that adjusts challenge to individual's performance

Laboratory stressors are essential tools to study the human stress response. However, despite considerable progress in the development of stress induction procedures in recent years, the field is still missing standardization and the methods employed frequently require considerable personnel resources. Virtual reality (VR) offers flexible solutions to these problems, but available VR stress-induction tests still contain important sources of variation that challenge data interpretation. One of the major drawbacks is that tasks based on motivated performance do not adapt to individual abilities. Here, we provide open access to, and present, a novel and standardized immersive multimodal virtual environment stress test (IMVEST) in which participants are simultaneously exposed to mental -arithmetic calculations- and environmental challenges, along with intense visual and auditory stimulation. It contains critical elements of stress elicitation – perceived threat to physical self, social-evaluative threat and negative feedback, uncontrollability and unpredictability – and adjusts mathematical challenge to individual's ongoing performance. It is accompanied by a control VR scenario offering a comparable but not stressful situation. We validate and characterize the stress response to IMVEST in one-hundred-and-eighteen participants. Both cortisol and a wide range of autonomic nervous system (ANS) markers – extracted from the electrocardiogram, electrodermal activity and respiration – are significantly affected. We also show that ANS features can be used to train a stress prediction machine learning model that strongly discriminates between stress and control conditions, and indicates which aspects of IMVEST affect specific ANS components.

Measuring the Behavioral Response to Spatial Audio within a Multi-Modal Virtual Reality Environment in Children with Autism Spectrum Disorder

Virtual Reality (VR) has been an active area of research in the development of interactive interventions for individuals with autism spectrum disorder (ASD) for over two decades. These immersive environments create a safe platform in which therapy can address the core symptoms associated with this condition. Recent advancements in spatial audio rendering techniques for VR now allow for the creation of realistic audio environments that accurately match their visual counterparts. However, reported auditory processing impairments associated with autism may affect how an individual interacts with their virtual therapy application. This study aims to investigate if these difficulties in processing audio information would directly impact how individuals with autism interact with a presented virtual spatial audio environment. Two experiments were conducted with participants diagnosed with ASD (n = 29) that compared: (1) behavioral reaction between spatialized and non-spatialized audio; and (2) the effect of background noise on participant interaction. Participants listening to binaural-based spatial audio showed higher spatial attention towards target auditory events. In addition, the amount of competing background audio was reported to influence spatial attention and interaction. These findings suggest that despite associated sensory processing difficulties, those with ASD can correctly decode the auditory cues simulated in current spatial audio rendering techniques.

Evaluation of Presence in Virtual Environments: Haptic Vest and User’s Haptic Skills

This paper presents the integration of a haptic vest with a multimodal virtual environment, consisting of video, audio, and haptic feedback, with the main objective of determining how users, who interact with the virtual environment, benefit from tactile and thermal stimuli provided by the haptic vest. Some experiments are performed using a game application of a train station after an explosion. The participants of this experiment have to move inside the environment, while receiving several stimuli to check if any improvement in presence or realism in that environment is reflected on the vest. This is done by comparing the experimental results with those similar scenarios, obtained without haptic feedback. These experiments are carried out by three groups of participants who are classified on the basis of their experience in haptics and virtual reality devices. Some differences among the groups have been found, which can be related to the levels of realism and synchronization of all the elements in the multimodal environment that fulfill the expectations and maximum satisfaction level. According to the participants in the experiment, two different levels of requirements are to be defined by the system to comply with the expectations of professional and conventional users.

Evaluation of CAD Model Manipulation in Desktop and Multimodal Immersive Interface

In this paper is presented an evaluation study regarding the manipulation of 3D CAD models using multimodal Virtual Reality (VR) interface and traditional desktop environment. A multimodal interface based on VR technologies that offers direct access to the space model by using several sensorial channels (visual, tactile, verbal) is described. Users can manipulate 3D objects using 6DOF tracking device, gestures and visualize the design status in an immersive CAVE - like system. The results of the evaluation study illustrate that using VR technologies as an alternative to replace WIMP CAD software interface is viable and brings more advantages.

Multimodal training and tele-assistance systems for the maintenance of industrial products

The paper describes an application based on Virtual and Augmented Reality technologies specifically developed to support maintenance operations of industrial products. Two scenarios have been proposed. In the first an operator learns how to perform a maintenance operation in a multimodal Virtual Reality environment that mixes a traditional instruction manual with the simulation, based on visual and haptic technologies, of the maintenance training task. In the second scenario, a skilled user operating in a multimodal Virtual Reality environment can remotely train another operator who sees the instructions about how the operations should be correctly performed, which are superimposed onto the real product. The paper presents the development of the application as well as its testing with users. Furthermore limits and potentialities of the use of Virtual and Augmented Reality technologies for training operators for product maintenance are discussed.

Neural networks underlying affective states in a multimodal virtual environment: contributions to boredom

The interaction of low perceptual stimulation or goal-directed behavior with a negative subjective evaluation may lead to boredom. This contribution to boredom may shed light on its neural correlates, which are poorly characterized so far. A video game served as simulation of free interactive behavior without interruption of the game’s narrative. Thirteen male German volunteers played a first-person shooter game (Tactical Ops: Assault on Terror) during functional magnetic resonance imaging (fMRI). Two independent coders performed the time-based analysis of the audio-visual game content. Boredom was operationalized as interaction of prolonged absence of goal-directed behavior with lowered affect in the Positive and Negative Affect Schedule (PANAS). A decrease of positive affect (PA) correlated with response amplitudes in bilateral insular clusters extending into the amygdala to prolonged inactive phases in a game play and an increase in negative affect (NA) was associated with higher responses in bilateral ventromedial prefrontal cortex (vmPFC). Precuneus and hippocampus responses were negatively correlated with changes in NA. We describe for the first time neural contributions to boredom, using a video game as complex virtual environment. Further our study confirmed that PA and NA are separable constructs, reflected by distinct neural patterns. PA may be associated with afferent limbic activity whereas NA with affective control.

Virtual Rehabilitation Environment Using Principles of Intrinsic Motivation and Game Design

This paper presents a novel multimodal virtual rehabilitation environment. Its design and implementation are based on principles related to intrinsic motivation and game design. The system consists of visual, acoustic, and haptic modalities. Elements contributing to intrinsic motivation are carefully joined in the three modalities to increase patients' motivation during the long process of rehabilitation. The message in a bottle (MIB) virtual scenario is designed to allow interplay between motor and cognitive challenges in the exercising patient. The user first needs to perform a motor action to receive a cognitive challenge that is finally solved by a second motor action. Visual feedback provides the most relevant information related to the task. Acoustic feedback consists of environmental sounds, music, and spoken instructions or encouraging statements for the patient. The haptic modality generates tactile information related to the environment and provides various modes of assistance for the patient's arm movements. The MIB scenario was evaluated with 16 stroke patients, who rated it positively using the Intrinsic Motivation Inventory questionnaire. Additionally, the MIB scenario seems to elicit higher motivation than a simpler pick-and-place training task.

Enhancing virtual environment spatial awareness training and transfer through tactile and vestibular cues

Haptic interaction has been successfully incorporated into a variety of virtual environment (VE) systems, yet designing multimodal VE training systems remains challenging as each cue incorporated during training should maximise learning and training transfer. This study examined the impact of incorporating two independent, spatialised tactile cues and vestibular cues into a military VE training environment with the goal of empirically examining whether such cues could enhance performance within the training environment and also that knowledge and skills gained during training could transfer to another environment. The results showed that tactile cues enhanced spatial awareness and performance during both repeated training and within a transfer environment, yet there were costs associated when two independent tactile cues were presented during training. In addition, results suggest that spatial awareness benefits from a tactile point indicator may be impacted by vestibular cues, as performance benefits were seen when tactile cues were paired with head tracking. To fully realise training potential, it is essential to determine how best to leverage multimodal capacity of VE training systems by identifying how multimodal training cues may advance knowledge, skills and attitudes of trainees. Results from this study provide design guidelines for incorporating tactile cues in VE training environments to enhance spatial awareness.

Physically Based Sound Synthesis for Large-Scale Virtual Environments

By offering a natural, intuitive interface with the virtual world, auditory display can enhance a user's experience in a multimodal virtual environment and further improve the user's sense of presence. However, compared to graphical display, sound synthesis has not been well investigated because of the extremely high computational cost for simulating realistic sounds. The state of the art for sound production in virtual environments is to use recorded sound clips that events in the virtual environment trigger, similar to how recorded animation sequences in the earlier days generated all the character motion in the virtual world. In this article, we describe several techniques for accelerating sound simulation, thereby enabling realistic, physically based sound synthesis for large-scale virtual environments

Prototyping and design for assembly analysis using multimodal virtual environments

The goal of this work is to investigate whether estimates of ease of part handling and part insertion can be provided by multimodal simulation using virtual environment (VE) technology, rather than by using conventional table-based methods such as Boothroyd and Dewhurst Charts. The long term goal is to extend cad systems to evaluate and compare alternative designs using Design for Assembly Analysis. A unified physically based model has been developed for modeling dynamic interactions among virtual objects and haptic interactions between the human designer and the virtual objects. This model is augmented with auditory events in a multimodal VE system called the Virtual Environment for Design for Assembly (VEDA). The designer sees a visual representation of the objects, hears collision sounds when objects hit each other and can feel and manipulate the objects through haptic interface devices with force feedback. Currently these models are 2D in order to preserve interactive update rates. Experiments were conducted with human subjects using two-dimensional peg-in-hole apparatus and a VEDA simulation of the same apparatus. The simulation duplicated as well as possible the weight, shape, size, peg-hole clearance, and frictional characteristics of the physical apparatus. The experiments showed that the Multimodal VE is able to replicate experimental results in which increased task completion times correlated with increasing task difficulty (measured as increased friction, increased handling distance combined with decreased peg-hole clearance). However, the Multimodal VE task completion times are approximately two times the physical apparatus completion times. A number of possible factors for this temporal discrepancy have been identified but their effect has not been quantified.

Experiments Using Multimodal Virtual Environments in Design for Assembly Analysis

The goal of this work is to investigate whether estimates of ease of part handling and part insertion can be provided by multimodal simulation using virtual environment (VE) technology. The long-term goal is to use this data to extend computer-aided design (CAD) systems in order to evaluate and compare alternate designs using design for assembly analysis. A unified, physically-based model has been developed for modeling dynamic interactions and has been built into a multimodal VE system called the Virtual Environment for Design for Assembly (VEDA). The designer sees a visual representation of objects, hears collision sounds when objects hit each other, and can feel and manipulate the objects through haptic interface devices with force feedback. Currently these models are 2D in order to preserve interactive update rates. Experiments were conducted with human subjects using a two-dimensional peg-in-hole apparatus and a VEDA simulation of the same apparatus. The simulation duplicated as well as possible the weight, shape, size, peg-hole clearance, and fictional characteristics of the physical apparatus. The experiments showed that the multimodal VE is able to replicate experimental results in which increased task completion times correlated with increasing task difficulty (measured as increased friction, increased handling distance, and decreased peg-hole clearance). However, the multimodal VE task completion times are approximately twice those of the physical apparatus completion process. A number of possible factors have been identified, but the effect of these factors has not been quantified.