Impact of interaction techniques on performance in virtual reality-based drone training for construction

User Experience and Engagement in the Reality–Virtuality Continuum: A Special Issue Guest Editorial

Developing Future Wearable Interfaces for Human-Drone Teams through a Virtual Drone Search Game

Hand-tracking enables controller-free interaction with virtual environments, which can make virtual reality (VR) experiences more natural and immersive. As naturalness hinges on both technological and human influence factors, fine-tuning the former while assessing the latter can be used to increase overall experience. This paper investigates a reach-grab-place task inside VR using two input modalities (hand-tracking vs. handheld-controller). Subjects (N = 33) compared the two input methods available on a consumer grade VR headset for their effects on objective user performance and subjective experience of the perceived sense of presence, cognitive workload, and ease-of-use. We found that virtual hands (with hand-tracking) did not influence the subjective feelings of perceived presence, naturalness, & engagement; neither did it inspire the overall ease-of-use while performing the task. In fact, subjects completed the task faster and felt a lower mental workload and higher overall usability with handheld-controllers. The result found that in this particular case, hand-tracking did not improve the psychological and emotional determinants of immersive VR experiences. The study helps expand on our understanding of the two input modalities in terms of their viability for naturalistic experiences in VR akin to real-world scenarios.

The effect of information availability in a user interface (UI) on in-vehicle task performance: A pilot study

Effective training is crucial for mitigating construction accidents and enhancing worker performance during high-risk complex tasks. Virtual reality (VR) environments are increasingly recognized as offering safer, more effective, and more efficient training methods for preparing individuals for hazardous construction settings. This study examines the impact of VR training on learning and motivation in such tasks, focusing specifically on post-tensioning slab construction. This study compares three training methods: VR training, VR observer training, and traditional paper-based teaching. Data collection involved post-test quiz scores and ratings from the Intrinsic Motivation Inventory (IMI). Statistical analyses, including one-way analysis of variance, post hoc tests, and effect size calculations, were applied to the data. The results indicate that the VR user group achieved significantly higher post-test scores and demonstrated greater intrinsic motivation across most dimensions than did the traditional group. Effect-size analyses underscore the practical significance of these findings, suggesting that VR technology and observation methods can significantly enhance learning effectiveness and motivation in challenging and hazardous construction tasks. These findings have implications for educational strategies and instructional designs in occupational training contexts.

Manipulating virtual objects using bare hands has been an attractive interaction paradigm in virtual and augmented reality due to its intuitive nature. However, one limitation of freehand input lies in the ambiguous resulting effect of the interaction. The same gesture performed on a virtual object could invoke different operations on the object depending on the context, object properties, and user intention. We present an experimental analysis of a set of disambiguation techniques in a virtual reality environment, comparing three input modalities (head gaze, speech, and foot tap) paired with three different timings in which options appear to resolve ambiguity (before, during, and after an interaction). The results indicate that using head gaze for disambiguation during an interaction with the object achieved the best performance.

INTRODUCTION: There are many potential human-machine interfaces for controlling complex robotics. However, restrictions in hardware, software, or human capability may pose limits on the input device degrees-of-freedom (DOF). This study examined effects on operational performance and strategy when interface DOF were limited, hypothesizing that different limitations on interface DOF would affect operator performance and technique.METHODS: Experiments used a Canadarm2 simulator with a dual-joystick interface adapted to operate under limited DOF conditions. Four interfaces were compared: full multiaxis (FM), limited translation (TL), limited rotation (RL), and without simultaneous translation/rotation or "non-bimanual" (NB). Subjects were tasked with operating the Canadarm2 in a simulated ISS control scenario to approach and grapple a moving cargo vehicle within a 90-s time limit.RESULTS: No significant difference was seen between FM and RL in task time or grapple success, and both were significantly different from TL. NB exhibited significantly increased task time from FM and RL, but no significant difference in grapple success rate. When rotating, subjects decreased time spent using multirotation for NB over FM.DISCUSSION: Similar performance between FM and RL suggests that restricting rotation may be preferred for interfaces with DOF design limits. For the NB condition, there was increased task time combined with decreased multirotation, highlighting potential use for NB in training for rotation efficiency. Two different strategies were observed during TL to overcome inability to visually track, align with, and move toward the target simultaneously. Examination of these techniques provides insight on which strategic elements were most critical for success.Hall SA, Stirling L. Human-machine interface degree of freedom effects on performance in space telerobotics. Aerosp Med Hum Perform. 2018; 89(12):1022-1030.

This study explores the serial position effect (SPE) in virtual reality (VR) user interface (UI) design, a concept extensively used in traditional UIs to enhance user recall by positioning critical information at the beginning or end of a sequence. While SPE is well-documented in non-VR contexts, its effectiveness in VR environments remains underexplored. We conducted an experiment involving 55 students who interacted with various UI designs in a VR setting. The interfaces displayed sequential information through different layouts, including lists and trees. Our findings indicate that while SPE influences recall in conventional settings, its impact in VR is mitigated by factors such as item familiarity and brand recognition. Notably, familiar items positioned mid-sequence were recalled as effectively as those at the start or end, suggesting that prior exposure can override SPE in VR. These results contribute to understanding UI design in immersive environments and suggest that traditional design principles, like SPE, may need adaptation for VR contexts.

Given the increased consumption of software solutions with virtual reality, there is also the growing availability of digital games. One of the quality attributes that can significantly influence the acceptance of virtual reality digital games on the market is playability. In this context, some new heuristics sets have been proposed to assess this quality attribute. These new specific sets approach explicitly the challenges posed by virtual reality, such as interface design, placement, and space among virtual and real reality. However, there is a lack of empirical evidence about the efficiency of these new heuristic sets. Therefore, this work presents an exploratory study where we conduct playability evaluations in five virtual reality games on first-person shooter gender using two heuristic sets: VR Play Guidelines and VR Heuristic Evaluation Tool. After analyzing the results of these evaluations, we noticed that the choice of the proper heuristic set to evaluate playability depends on the development stage, the objective, and the technical knowledge of the evaluator.

Target selection is a fundamental interaction in virtual reality (VR). But the act of confirming a selection, such as a button press or pinch, can disturb the tracked pose and shift the intended target, which is referred to as the Heisenberg Effect. Prior research has mainly investigated controller input. However, it remains unclear how the effect manifests in the bare-hand input and how score-based techniques may mitigate the effect in different spatial variations. To fill the gap, we conduct a within-subject study to examine the Heisenberg Effect across two input modalities (i.e., controller and hand) and two selection mechanisms (i.e., direct and score-based). Our results show that hand input is more susceptible to the Heisenberg Effect, with direct selection more influenced by target width and score-based selection more sensitive to target density. Based on previous vote-oriented technique and our temporal analysis, we introduce weighted VOTE, a history-based intention accuracy model for target voting, that reweights recent interaction intent to counteract input disturbances. Our evaluation shows the method improves selection accuracy compared to baseline techniques. Finally, we discuss future directions for adaptive selection methods.

Conventional CAD modelling software demands substantial utilisation of input modalities like the keyboard and mouse for creating 3-dimensional (3D) models. The dexterity measures involved in controlling input modalities could pose challenges to users with motor disabilities—including the inability to move their limbs, particularly their upper and lower arms, and fingers, due to traumatic damage or congenital problems. In order to meet these challenges, this paper proposes a virtual reality (VR)-based medium to help users with motor disabilities build simple 3D models for architectural design. The concept of operating buttons using head-gaze in the VR environment has been utilised to perform scaling—a 3D object manipulation method—to create simplified building models. Moreover, navigation in the VR space using tilting of the head has been employed with the user seated on a revolving chair, thus eliminating the need for any limbic movement. Unity game engine was used to develop two variations of the VR model with a different button layout for creating simple cuboidal volumes mimicking buildings in the virtual environment. Both variations have been tested with 32 individuals against a specific performance indicator (i.e., task completion time) and self-reported metrics, such as the perception of effort applied and degree of visual clutter, followed by retrospective participant feedback sessions. One of the VR application's variants (i.e., variant 1) produced promising results regarding overall usability and effort demand. This paper also proposes a methodological framework for an AI-based, intelligent, and adaptive VR application interface that caters to the user's abilities and pain points in real-time. In the future, this framework could be instrumental in creating a comprehensive gaze-based VR tool for 3D modelling having multiple functions to help users with motor disabilities.

Despite the popularity of VR, there is a lack of research in interaction methods for VR HMDs. In their confined setting, conventional VR headsets provide only limited input modalities. Commonly-supported interaction methods include tracking the user's head orientation, external controllers and buttons, and more recently, gaze interaction by eye tracking. This research aims to widen the currently available range of input modalities in virtual reality, by developing novel interaction designs and exploring multimodal interaction in mobile VR. The first work in this research aims to add a novel interaction method for a DIY cardboard VR headset that enfolds a commodity smartphone. A set of novel tap-gesture-based inputs have been developed that take advantage of the motion sensing capabilities available in smartphones. Experimental results have shown promising potential for this system to extend the interactivity of future cardboard-based VR headset applications. The second work in this research explores eye gaze tracking as a feasible interaction modality in VR, and understands some of its challenges. It presents DualGaze, a novel gaze-based interaction method developed to address the Midas Touch problem for gaze mediated VR interaction. In this, users perform a distinctive two-step gaze gesture for object selection. A user study was conducted to compare the accuracy and selection speed of DualGaze and the popular gaze fixation method on a simple gaze-typing task. The results show that DualGaze is significantly more accurate while maintaining a comparable selection speed that was observed to improve with familiarity of use. The next part of the research explores hand and finger gesture-based interaction in VR using the smartphone camera inside a VR HMD, and vision-based tracking methods. The research also explores the effects of combining these modalities together with head gaze-based user input in VR, and proposes a few novel interaction designs. The first interaction design developed in this modality is a novel two-handed gesture interaction design that is invariant to head motion, called Twin-Fingers. The technique provides an interactive 2D cursor control in VR, where the user slides two overlapping splayed fingers over two from the other hand to perform the gesture. In addition, a conventional single-hand gesture design is also considered and its performance is compared with Twin-Fingers on an onscreen cursor-based pointing task that involves both head movement and finger-gesture interaction. Our user study shows that most users prefer the simplicity and convenience of a single-hand gesture for tasks without requiring head movement. However, with limited practice, users performed better with Twin-Fingers when tackling tasks that required both head movement and finger gestures. When it was found that Twin-Fingers required a considerable learning curve, another interaction design was developed in a similar vein, called X-Fingers, which involved only one (index) finger from the dominant hand sliding over the index finger of the non-dominant hand. X-Fingers required less effort to perform the gesture by the user, while retaining Twin-Fingers' properties of invariance to hand and head motion; this led to the idea that this interaction design can be coordinated with the movement of the user's arms or head to provide an additional input modality. The incorporation of the arms or the head provides physically-coupled and physically-decoupled multimodal interactions respectively. Given these two design options, user studies were conducted to understand how the nature of the physical coupling of interactions influences the user's performance with tasks of varying degrees of coordination between the modalities. The results show that physically-decoupled interactions designs are preferred when the degree of coordination is high within the multimodal interaction. Moreover, experiments were also conducted to investigate user performances in object positioning tasks of the coupled interaction modality with arms together versus a decoupled arms-apart modality. Results show that user performances and preferences depend on the target size and task type for each interaction modality. Combining the findings from these user evaluations, the research presents guidelines for VR interaction design for task types and coordination levels based on users' performances and preferences. By presenting designs for hand and head interactions along with the previous work on tap and eye-based interactions, this research broadens the existing input modalities available in VR HMDs, thus allowing for richer user experiences in mobile VR.

Synchronous computer-mediated communication (SCMC) is a topic of great interest in CALL literature where research has investigated the effectiveness of SCMC compared to traditional face-to-face instruction. However, there are few studies that investigate the intrinsic differences in SCMC modes, particular in terms of their effect on oral communication. At the JALTCALL 2019 conference, we introduced research which assessed the anxiety-reducing affordances of VR. This year we presented results of a follow-up study which focused on the effect of SCMC modality on learners’ speaking performance. 30 participants (15 pairs) completed a spot-the-difference task within three different SCMC modes: voice, video, and virtual reality (VR). Using the complexity, accuracy, and fluency (CAF) model, participants’ oral task performances were analysed. Results suggest that the voice mode promoted the highest structural complexity, however, the VR mode promoted the highest lexical complexity. Findings therefore suggest that different modes of communication may be used to focus on different skill development. Additionally, practitioners should consider how modality affects learner anxiety and choose the most appropriate system for their students and needs. This paper introduces the VR system, a detailed analysis of results, pedagogical implications, and future research directions for the use of VR in language teaching contexts.

Human-Computer Interaction (HCI) in Virtual Reality (VR) environments is a rapidly evolving field that seeks to enhance user experience through immersive and intuitive design principles. As VR technology advances, the interaction between humans and virtual systems becomes increasingly complex, requiring innovative approaches to ensure usability, accessibility, and engagement. This paper explores the fundamental principles of HCI in VR, focusing on interaction techniques, input modalities, feedback mechanisms, and the psychological impact of virtual experiences. Various interaction techniques, such as hand tracking, motion controllers, eye-tracking, and voice commands, are examined, highlighting their advantages and limitations in different applications. Additionally, feedback mechanisms, including haptic feedback, spatial audio, and visual cues, play a crucial role in enhancing realism and user immersion. The study also addresses cognitive and ergonomic challenges, such as motion sickness, cognitive load, and the importance of adaptive interfaces that accommodate diverse user needs. Furthermore, the concept of presence—the feeling of “being there” in a virtual space—is explored, emphasizing how design choices influence immersion and engagement. Accessibility considerations, including designing for users with disabilities and optimizing VR experiences for different demographics, are also discussed. By analyzing current trends, user experience research, and best practices, this study provides insights for designers, developers, and researchers aiming to create effective, user-friendly, and inclusive VR applications. Ultimately, the goal is to improve the seamless integration of humans and virtual environments, enhancing usability and effectiveness across various domains such as gaming, education, healthcare, and remote collaboration.

The number of small drones and drone operations is expanding and proliferating tremendously. However, there is a problem. Drones crash. Often. When they do, international studies show that over 70% of the drone crashes can be related to human factors. Combining these two facts, it is clear that - if we want to avoid a massive number of drone incidents in the future – it is required to develop a strategy to incorporate human factors in the drone deployment process and the training of drone pilots. Pilots for regular aircraft or for larger (typically military) drones generally follow extensive simulator training before engaging in any real flight. However, for small rotorcraft, this is much less the case, because it is very difficult to convey a realistic representation to the human sensory system. Both for fixed wing and rotary wing drones, the main problem with current simulator-based pilot training programs is that they are limited to simplistic scenarios (typically flying predefined patterns and practicing take-off and landing operations), without providing much qualitative feedback to the trainee or the supervising entity.In response to these identified shortcomings, we present in this paper a drone operator performance assessment tool, which uses a realistic environment and realistic operational conditions to measure the performance of the drone operator, both in a qualitative and quantitative manner. These metrics can then be used by training responsibles to adapt / adjust the theoretical and practical training courses for drone pilots, such that the curriculum (both the practical and the theoretical courses) can be iteratively optimized to best fit the needs. An important aspect of any qualification assessment procedure is the definition of the test methodologies and of the test scenarios. Within the subject of drone pilot training, these test scenarios are currently most often very limited to simple take-off & landing operations and of following simple patterns in the air. For pilots working in the security sector (military, police, firefighters, civil protection, ...) in tough operating conditions, these highly simplistic scenarios are hardly relevant. Therefore, we also propose a set of standard test methods specifically geared towards the training of drone operators in the security sector.In this paper, we present the architecture of the developed assessment tool, which runs inside a simulation environment, enabling repetitive (statistically relevant) testing in a controlled environment. The simulation engine is based upon the AirSim simulator and uses the UnReal game engine for realistic environmental rendering. Together with the end-users, twenty-two scenarios have been defined within two main mission environments: a mountainous and an urban environment. These scenarios are especially conceived to cover most of the hazards and environmental challenges that end users wanted to test for, the pilot performance capabilities that they wanted to see measured and the human factors that they identified as potentially important influencing factors for pilot performance.