Mid-air Interaction Research Articles

TouchView: Mid-Air Touch on Zoomable 2D View for Distant Freehand Selection on a Virtual Reality User Interface.

Selection is a fundamental interaction element in virtual reality (VR) and 3D user interfaces (UIs). Raycasting, one of the most common object selection techniques, is known to have difficulties in selecting small or distant objects. Meanwhile, recent advancements in computer vision technology have enabled seamless vision-based hand tracking in consumer VR headsets, enhancing accessibility to freehand mid-air interaction and highlighting the need for further research in this area. This study proposes a new technique called TouchView, which utilizes a virtual panel with a modern adaptation of the Through-the-Lens metaphor to improve freehand selection for VR UIs. TouchView enables faster and less demanding target selection by allowing direct touch interaction with the magnified object proxies reflected on the panel view. A repeated-measures ANOVA on the results of a follow-up experiment on multitarget selection with 23 participants showed that TouchView outperformed the current market-dominating freehand raycasting technique, Hybrid Ray, in terms of task performance, perceived workload, and preference. User behavior was also analyzed to understand the underlying reasons for these improvements. The proposed technique can be used in VR UI applications to enhance the selection of distant objects, especially for cases with frequent view shifts.

Comparison of Unencumbered Interaction Technique for Head-Mounted Displays

Head Mounted Displays (HMDs) are gaining more public attention. With the advancement of tracking technologies, they are incorporating unencumbered interaction techniques to address the need for user-friendly and efficient interaction techniques for day-to-day activities. While there is a good understanding of the different interaction techniques individually, very little research has been done to compare them directly. This would be vital to understanding their strengths and weaknesses in different contexts and building better synergies among them. This paper uses a target selection task to compare the performance and user preferences for four interaction techniques: gaze-pinch, ray pointer, hand-proximate user interface, and direct mid-air interactions. Results indicate that the gaze-pinch interaction technique required significantly more time to complete the task than the others, whose time to complete was similar. However, in terms of preferences and errors, the interaction techniques mostly performed similar.

Ergonomic assessment of mid-air interaction and device-assisted interactions under vibration environments based on task performance, muscle activity and user perceptions

Mid-air interaction has been increasingly introduced for human-computer interaction (HCI) tasks in vibration environments, but it has seldom been assessed from ergonomic aspects, especially in comparison with device-assisted interactions. This study aimed to provide a comprehensive ergonomic assessment of mid-air interaction and device-assisted interactions under vibration environments based on task performance, muscle activity in the upper limb and shoulder, and user perceptions. A within-subjects design was implemented in this study, where participants were required to perform basic pointing and dragging tasks with four interaction modes (i.e., one mid-air interaction and three device-assisted interactions) under static, low and high vibration environments, respectively. Both small and large target sizes were examined. Muscle activity was recorded with surface electromyography for five muscles from participants’ dominant arm. Results showed that mid-air interaction yielded longer task completion time, more errors, higher perceived workload, lower usability ratings, and larger muscle activities in the forearm, upper arm and shoulder compared with device-assisted interactions. There were significant interaction effects between vibration and interaction mode. Specifically, compared with device-assisted interactions, mid-air interaction was associated with greater susceptibility to the detrimental effects of vibration (poorer task performance and larger muscle activities). Target size significantly affected task performance, but the effects varied by tasks. Overall, our results suggest that mid-air interaction presents a higher ergonomic risk compared with device-assisted interactions, especially in vibration environments. These findings provide implications for better use, configuration and ergonomic assessment of interaction tools in vibration environments, and are useful in developing evidence-based interventions to control ergonomic risk in HCI tasks in vibration environments.

To Stick or Not to Stick? Studying the Impact of Offset Recovery Techniques During Mid-Air Interactions.

During mid-air interactions, common approaches (such as the god-object method) typically rely on visually constraining the user's avatar to avoid visual interpenetrations with the virtual environment in the absence of kinesthetic feedback. This paper explores two methods which influence how the position mismatch (positional offset) between users' real and virtual hands is recovered when releasing the contact with virtual objects. The first method (sticky) constrains the user's virtual hand until the mismatch is recovered, while the second method (unsticky) employs an adaptive offset recovery method. In the first study, we explored the effect of positional offset and of motion alteration on users' behavioral adjustments and users' perception. In a second study, we evaluated variations in the sense of embodiment and the preference between the two control laws. Overall, both methods presented similar results in terms of performance and accuracy, yet, positional offsets strongly impacted motion profiles and users' performance. Both methods also resulted in comparable levels of embodiment. Finally, participants usually expressed strong preferences toward one of the two methods, but these choices were individual-specific and did not appear to be correlated solely with characteristics external to the individuals. Taken together, these results highlight the relevance of exploring the customization of motion control algorithms for avatars.

NICER: A New and Improved Consumed Endurance and Recovery Metric to Quantify Muscle Fatigue of Mid-Air Interactions

Natural gestures are crucial for mid-air interaction, but predicting and managing muscle fatigue is challenging. Existing torque-based models are limited in their ability to model above-shoulder interactions and to account for fatigue recovery. We introduce a new hybrid model, NICER , which combines a torque-based approach with a new term derived from the empirical measurement of muscle contraction and a recovery factor to account for decreasing fatigue during rest. We evaluated NICER in a mid-air selection task using two interaction methods with different degrees of perceived fatigue. Results show that NICER can accurately model above-shoulder interactions as well as reflect fatigue recovery during rest periods. Moreover, both interaction methods show a stronger correlation with subjective fatigue measurement ( ρ = 0.978/0.976) than a previous model, Cumulative Fatigue ( ρ = 0.966/0.923), confirming that NICER is a powerful analytical tool to predict fatigue across a variety of gesture-based interactive applications.

Tangible and Mid-Air Interactions in Hand-Held Augmented Reality for Upper Limb Rehabilitation: An Evaluation of User Experience and Motor Performance

Hand-held augmented reality (AR) offers accessible, interactive rehabilitation options for patients with upper limb motor deficits. Incorporating hand-involved interactions (e.g., tangible and mid-air interactions) into hand-held AR provides patients with intuitive manners to perform rehabilitation exercises mimicking real-world activities. Previous work has shown the importance of user experience and motor performance in rehabilitation systems, but little was known in the literature regarding the impact of hand-involved interactions in hand-held AR on user experience and motor performance in rehabilitation exercises. Hence, this study aims to evaluate user experience and motor performance when using three types of hand-involved interactions in hand-held AR rehabilitation: (1) tangible cube (i.e., a space-multiplexed tangible interaction with a physical cube acting as a real proxy to manipulate a virtual object in the same form); (2) tangible controller (i.e., a time-multiplexed tangible interaction with a physical controller applied to manipulate a virtual object); and (3) hand motion (i.e., a form of mid-air interaction to move a virtual object with hands). Based on the findings from self-report, electroencephalography (EEG), and performance measures, this study reveals the advantages of the tangible cube over the tangible controller, both superior to the hand motion in hand-held AR rehabilitation regarding user experience and motor performance. This study offers new understanding of the advantages and disadvantages of various interaction techniques in hand-held AR rehabilitation, emphasizing crucial design considerations for these systems, with a focus on user experience and motor performance in upper limb rehabilitation.

Breadth and orientation of pie menus for mid-air interaction: effects on upper extremity biomechanics, performance, and subjective assessment

ABSTRACT The number and layout of options within pie menus are crucial for determining the direction of radial swiping in mid-air interactions, influencing the physical demands imposed on users. This research examined six distinct option quantities, referred to as menu breadth, and investigated two layouts, designated as menu orientations. These orientations encompass landscape and portrait formats, with the former positioning a top option centred parallel to the horizontal axis, while the latter situates two top options on either side of the vertical axis. The study employed comprehensive metrics, including muscle activity, wrist movement, user performance, and subjective assessments. Our results indicated that the breadth should be controlled to no more than six to ensure high efficiency and align with users’ interactive experience, while breadth 2 should be avoided to eliminate the potential risk of wrist musculoskeletal disorders. Besides, the portrait orientation for breadth 3 and the landscape orientation for breadth 6 were recommended for their lower physiological loads and higher efficiency. Also, the landscape orientation for breadth 4 should be favoured as it outperformed portrait orientation in both wrist movement dimensions. These findings could provide scientific design principles for using pie menus in mid-air interaction within future HCI applications.

Comparison of Panel and Pie Menus for Mid-Air Interaction in a Photo Booth: Muscle Activity, Performance, and User Experience

Research on visual menus, specifically panel and pie menus, has gained widespread attention in recent times. However, limited research focuses on the mid-air interaction when comparing these two menu types. This study aimed to conduct a comprehensive comparison between panel and pie menus in the context of cursor-based mid-air interaction within a photo booth setting. The evaluation involved analyzing muscle activities, performance metrics, and user experience to quantify the differences between the two menus across three menu operations. Our results indicated that the pie menu offered notable advantages by reducing muscular loads, achieving shorter task completion times, and delivering an overall more enjoyable user experience, making it a preferable choice for mid-air interaction in photo booth settings. Conversely, the panel menu exhibited superior scores of learnability and demonstrated enhanced accuracy in the switching operation, suggesting that it might be a better choice for complex and precision-demanding tasks.

Controllers or Bare Hands? A Controlled Evaluation of Input Techniques on Interaction Performance and Exertion in Virtual Reality.

Virtual Reality (VR) systems have traditionally required users to operate the user interface with controllers in mid-air. More recent VR systems, however, integrate cameras to track the headset's position inside the environment as well as the user's hands when possible. This allows users to directly interact with virtual content in mid-air just by reaching out, thus discarding the need for hand-held physical controllers. However, it is unclear which of these two modalities-controller-based or free-hand interaction-is more suitable for efficient input, accurate interaction, and long-term use under reliable tracking conditions. While interacting with hand-held controllers introduces weight, it also requires less finger movement to invoke actions (e.g., pressing a button) and allows users to hold on to a physical object during virtual interaction. In this paper, we investigate the effect of VR input modality (controller vs. free-hand interaction) on physical exertion, agency, task performance, and motor behavior across two mid-air interaction techniques (touch, raycast) and tasks (selection, trajectory-tracing). Participants reported less physical exertion, felt more in control, and were faster and more accurate when using VR controllers compared to free-hand interaction in the raycast setting. Regarding personal preference, participants chose VR controllers for raycast but free-hand interaction for mid-air touch. Our correlation analysis revealed that participants' physical exertion increased with selection speed, quantity of arm motion, variation in motion speed, and bad postures, following ergonomics metrics such as consumed endurance and rapid upper limb assessment. We also found a negative correlation between physical exertion and the participant's sense of agency, and between physical exertion and task accuracy.

UbiSurface: A Robotic Touch Surface for Supporting Mid-air Planar Interactions in Room-Scale VR

Room-scale VR has been considered an alternative to physical office workspaces. For office activities, users frequently require planar input methods, such as typing or handwriting, to quickly record annotations to virtual content. However, current off-the-shelf VR HMD setups rely on mid-air interactions, which can cause arm fatigue and decrease input accuracy. To address this issue, we propose UbiSurface, a robotic touch surface that can automatically reposition itself to physically present a virtual planar input surface (VR whiteboard, VR canvas, etc.) to users and to permit them to achieve accurate and fatigue-less input while walking around a virtual room. We design and implement a prototype of UbiSurface that can dynamically change a canvas-sized touch surface's position, height, and pitch and yaw angles to adapt to virtual surfaces spatially arranged at various locations and angles around a virtual room. We then conduct studies to validate its technical performance and examine how UbiSurface facilitates the user's primary mid-air planar interactions, such as painting and writing in a room-scale VR setup. Our results indicate that this system reduces arm fatigue and increases input accuracy, especially for writing tasks. We then discuss the potential benefits and challenges of robotic touch devices for future room-scale VR setups.

“I Can See Your Password”: A Case Study About Cybersecurity Risks in Mid-Air Interactions of Mixed Reality-Based Smart Manufacturing Applications

Abstract This paper aims to present a potential cybersecurity risk existing in mixed reality (MR)-based smart manufacturing applications that decipher digital passwords through a single RGB camera to capture the user’s mid-air gestures. We first created a test bed, which is an MR-based smart factory management system consisting of mid-air gesture-based user interfaces (UIs) on a video see-through MR head-mounted display. To interact with UIs and input information, the user’s hand movements and gestures are tracked by the MR system. We setup the experiment to be the estimation of the password input by users through mid-air hand gestures on a virtual numeric keypad. To achieve this goal, we developed a lightweight machine learning-based hand position tracking and gesture recognition method. This method takes either video streaming or recorded video clips (taken by a single RGB camera in front of the user) as input, where the videos record the users’ hand movements and gestures but not the virtual UIs. With the assumption of the known size, position, and layout of the keypad, the machine learning method estimates the password through hand gesture recognition and finger position detection. The evaluation result indicates the effectiveness of the proposed method, with a high accuracy of 97.03%, 94.06%, and 83.83% for 2-digit, 4-digit, and 6-digit passwords, respectively, using real-time video streaming as input with known length condition. Under the unknown length condition, the proposed method reaches 85.50%, 76.15%, and 77.89% accuracy for 2-digit, 4-digit, and 6-digit passwords, respectively.

Surveying the Social Comfort of Body, Device, and Environment-Based Augmented Reality Interactions in Confined Passenger Spaces Using Mixed Reality Composite Videos

Augmented Reality (AR) headsets could significantly improve the passenger experience, freeing users from the restrictions of physical smartphones, tablets and seatback displays. However, the confined space of public transport and the varying proximity to other passengers may restrict what interaction techniques are deemed socially acceptable for AR users - particularly considering current reliance on mid-air interactions in consumer headsets. We contribute and utilize a novel approach to social acceptability video surveys, employing mixed reality composited videos to present a real user performing interactions across different virtual transport environments. This approach allows for controlled evaluation of perceived social acceptability whilst freeing researchers to present interactions in any simulated context. Our resulting survey (N=131) explores the social comfort of body, device, and environment-based interactions across seven transit seating arrangements. We reflect on the advantages of discreet inputs over mid-air and the unique challenges of face-to-face seating for passenger AR.

Abacus Gestures

Designing an extensive set of mid-air gestures that are both easy to learn and perform quickly presents a significant challenge. Further complicating this challenge is achieving high-accuracy detection of such gestures using commonly available hardware, like a 2D commodity camera. Previous work often proposed smaller, application-specific gesture sets, requiring specialized hardware and struggling with adaptability across diverse environments. Addressing these limitations, this paper introduces Abacus Gestures, a comprehensive collection of 100 mid-air gestures. Drawing on the metaphor of Finger Abacus counting, gestures are formed from various combinations of open and closed fingers, each assigned different values. We developed an algorithm using an off-the-shelf computer vision library capable of detecting these gestures from a 2D commodity camera feed with an accuracy exceeding 98% for palms facing the camera and 95% for palms facing the body. We assessed the detection accuracy, ease of learning, and usability of these gestures in a user study involving 20 participants. The study found that participants could learn Abacus Gestures within five minutes after executing just 15 gestures and could recall them after a four-month interval. Additionally, most participants developed motor memory for these gestures after performing 100 gestures. Most of the gestures were easy to execute with the designated finger combinations, and the flexibility in executing the gestures using multiple finger combinations further enhanced the usability. Based on these findings, we created a taxonomy that categorizes Abacus Gestures into five groups based on motor memory development and three difficulty levels according to their ease of execution. Finally, we provided design guidelines and proposed potential use cases for Abacus Gestures in the realm of mid-air interaction.

Mid-Air Gestural Interaction with a Large Fogscreen

Projected walk-through fogscreens have been created, but there is little research on the evaluation of the interaction performance with fogscreens. The present study investigated mid-air hand gestures for interaction with a large fogscreen. Participants (N = 20) selected objects from a fogscreen using tapping and dwell-based gestural techniques, with and without vibrotactile/haptic feedback. In terms of Fitts’ law, the throughput was about 1.4 bps to 2.6 bps, suggesting that gestural interaction with a large fogscreen is a suitable and effective input method. Our results also suggest that tapping without haptic feedback has good performance and potential for interaction with a fogscreen, and that tactile feedback is not necessary for effective mid-air interaction. These findings have implications for the design of gestural interfaces suitable for interaction with fogscreens.

The effect of hands synchronicity on users perceived arms Fatigue in Virtual reality environment

The use of virtual reality (VR) system has become more common with the arrival of a new generation of headsets. This increase in accessibility has in turn led to VR being used for applications which require longer sessions, such as video games, artistic expression, rehabilitation, and so on. Taking muscular fatigue in account for the VR applications is therefore becoming essential to ensure a comfort of usage and also to avoid injuries. In addition, usual interactions in these diverse applications are very varied and may require the use of one or two hands, synchronized or not. However, little research has focused on fatigue produced by one and two handed interactions especially during mid-air interaction in VR environment. In this paper, we examine the effect of hands synchronicity and movement direction on the user perceived arm fatigue in VR environment during a simple (one line of targets) and a composite (two lines of targets) pointing tasks performed using both controlled and free hands synchronicity. Our findings indicate that to optimize the relationship between fatigue and efficiency, it seems necessary to select the synchronicity of hands depending on the performed task. Furthermore, it is desirable to permit the users to use the hands synchronicity they prefer. In addition, distance covered by the hand and changes in user posture could be used in real-time as indicators of fatigue to trigger changes in the interface, or notifications. Finally, our findings reveal that the directions of movement along the vertical axis and some diagonals are more tiring than those of the horizontal plane, which suggests that it would be better to favor the use of horizontal directions.

Design, evaluation and calibration of wearable electrotactile interfaces for enhancing contact information in virtual reality

Electrotactile feedback is a convenient tactile rendering method thanks to reduced form factor and power consumption. Yet, its usage in immersive virtual reality has been scarcely addressed. This paper explores how electrotactile feedback could be used to enhance contact information for mid-air interactions in virtual reality. We propose an electrotactile rendering method which modulates the perceived intensity of the electrotactile stimuli according to the interpenetration distance between the user’s finger and the virtual surface. In a first user study (N=21), we assessed the performance of our method against visual interpenetration feedback and no-feedback. Contact precision and accuracy were significantly improved when using interpenetration feedback. The results also showed that the calibration of the electrotactile stimuli was key, which motivated a second study exploring how the calibration procedure could be improved. In a second study (N=16), we compared two calibration techniques: a non-VR keyboard and a VR direct interaction method. While the two methods provided comparable usability and calibration accuracy, the VR method was significantly faster. These results pave the way for the usage of electrotactile feedback as an efficient alternative to visual feedback for enhancing contact information in virtual reality.

Playing Gamelan Bonang in the Air: User Requirements for Designing a Digital Musical Instrument for the Malay Bonang

In recent years, computer technologies have been impactful in the design and development of Digital Musical Instruments (DMIs). As music interaction became prominent in the Human-Computer Interaction (HCI) field, emphasis on user requirement upon the design of musical interfaces has also grown since the last decade. Although designing new DMIs is becoming very popular, it is often determined by the designers and often not reflective of users’ needs. In this study, we explored user requirements for the design of a virtual musical instrument of the Malay bonang, an instrument found in the Malay gamelan ensemble. The requirements were elicited from a group of gamelan experts to establish the bonang playing techniques to be mapped to the virtual instrument which we called Air Bonang. Findings revealed that in designing the Air Bonang that is natural and expressive, the fundamental playing techniques of the bonang should be integrated into the system using mid-air interaction. In addition, exploratory techniques might also be integrated into the Air Bonang to leverage musical expression. The outcome of the study proposes design criteria that encompass three aspects of a natural Air Bonang, namely, embodiment, expressiveness, and feedback.

Effects of spatial constraints and ages on children’s upper limb performance in mid-air gesture interaction

Through two controlled experiments, including a pie menu study and a target acquisition study, this paper investigates children’s performance on mid-air gesture interactions in different spatial constraints (i.e., in different orientations/distances), as well as the age effect on such interaction scenarios. The first experiment recorded children’s speeds and accuracies following certain directions under menus with different numbers of items, while the second evaluated the speed–accuracy trade-off (SAT) of children’s arm movements. We also compared performance differences between two age-related groups (i.e., 6–8 years old and 9–12 years old). Based on these experiments, we propose an improved design for UI menus based on mid-air gesture interaction for children. The improved design provides suggestions for setting appropriate directions and difficulty indexes, which makes it much easier and quicker for children to use the menus with mid-air interaction.

Advanced modeling method for quantifying cumulative subjective fatigue in mid-air interaction

Interaction in mid-air can be fatiguing. A model-based method to quantify cumulative subjective fatigue for such interaction was recently introduced in HCI research. This model separates muscle units into three states: active (MA) fatigued (MF) or rested (MR) and defines transition rules between states. This method demonstrated promising accuracy in predicting subjective fatigue accumulated in mid-air pointing tasks. In this paper, we introduce an improved model that additionally captures the variations of the maximum arm strength based on arm postures and adds linearly-varying model parameters based on current muscle strength. To validate the applicability and capabilities of the new model, we tested its performance in various mid-air interaction conditions, including mid-air pointing/docking tasks, with shorter and longer rest and task periods, and a long-term evaluation with individual participants. We present results from multiple cross-validations and comparisons against the previous model and identify that our new model predicts fatigue more accurately. Our modeling approach showed a 42.5% reduction in fatigue estimation error when the longitudinal experiment data is used for an individual participant’s fatigue. Finally, we discuss the applicability and capabilities of our new approach.

Pseudo-haptic button for improving user experience of mid-air interaction in VR

Mid-air interaction is one of the promising interaction modalities in virtual reality (VR) due to its merits in naturalness and intuitiveness, but the interaction suffers from the lack of haptic feedback as no force or vibrotactile feedback can be provided in mid-air without extra devices. As a breakthrough to compensate for this insufficiency, the application of pseudo-haptic features which create the visuo-haptic illusion without actual physical haptic stimulus can be explored. Therefore, this study aimed to investigate the effect of four pseudo-haptic features: proximity feedback, protrusion, hit effect, and penetration blocking on user experience for free-hand mid-air button interaction in VR. We conducted a user study on 21 subjects to collect user ratings on various aspects of user experience while users were freely interacting with 16 buttons with different combinations of four features. Results indicated that all investigated features significantly improved user experience in terms of haptic illusion, embodiment, sense of reality, spatiotemporal perception, satisfaction, and hedonic quality. In addition, protrusion and hit effect were more beneficial in comparison with the other two features. It is recommended to utilize the four proposed pseudo-haptic features in 3D user interfaces (UIs) to make users feel more pleased and amused, but caution is needed when using proximity feedback together with other features because it can be distracting. The findings of this study could be helpful for VR developers and UI designers in providing better interactive buttons in the 3D interfaces.