Gesture-based Input Research Articles

Design of 3D Microgestures for Commands in Virtual Reality or Augmented Reality

Virtual and augmented reality (VR, AR) systems present 3D images that users can interact with using controllers or gestures. The design of the user input process is crucial and determines the interactive efficiency, comfort, and adoption. Gesture-based input provides a device-free interaction that may improve safety and creativity compared to using a hand controller while allowing the hands to perform other tasks. Microgestures with small finger and hand motions may have an advantage over the larger forearm and upper arm gestures by reducing distraction, reducing fatigue, and increasing privacy during the interaction. The design of microgestures should consider user experience, ergonomic principles, and interface design to optimize productivity and comfort while minimizing errors. Forty VR/AR or smart device users evaluated a set of 33 microgestures, designed by ergonomists, and linked them to 20 common AR/VR commands based on usability, comfort, and preference. Based primarily on preference, a set of microgestures linked to specific commands is proposed for VR or AR systems. The proposed microgesture set will likely minimize fatigue and optimize usability. Furthermore, the methodology presented for selecting microgestures and assigning them to commands can be applied to the design of other gesture sets.

Investigating the Performance of Gesture-Based Input for Mid-Air Text Entry in a Virtual Environment: A Comparison of Hand-Up versus Hand-Down Postures.

Although the interaction technology for virtual reality (VR) systems has evolved significantly over the past years, the text input efficiency in the virtual environment is still an ongoing problem. We deployed a word-gesture text entry technology based on gesture recognition in the virtual environment. This study aimed to investigate the performance of the word-gesture text entry technology with different input postures and VR experiences in the virtual environment. The study revealed that the VR experience (how long or how often using VR) had little effect on input performance. The hand-up posture has a better input performance when using word-gesture text entry technology in a virtual environment. In addition, the study found that the perceived exertion to complete the text input with word-gesture text entry technology was relatively high. Furthermore, the typing accuracy and perceived usability for using the hand-up posture were obviously higher than that for the hand-down posture. The hand-up posture also had less task workload than the hand-down posture. This paper supports that the word-gesture text entry technology with hand-up posture has greater application potential than hand-down posture.

Blind Leading the Sighted

Voice-activated personal assistants (VAPAs) are becoming smaller, cheaper, and more accurate, such that they are now prevalent in homes (e.g., Amazon Echo, Sonos One) and on mobile devices (e.g., Google Assistant, Apple Siri) around the world. VAPAs offer considerable potential to individuals who are blind, offering efficiencies over gesture-based input on touchscreen devices. However, research is just beginning to reveal the ways in which these technologies are used by people who are blind. In the first of two studies, we interviewed 14 blind adults with experience of home and/or mobile-based VAPAs, surfacing myriad accessibility, usability, and privacy issues for this community. A second study analyzing podcast content from 28 episodes relating to blind interactions with VAPAs was then undertaken to validate and extend findings from the first study. In addition to verifying prior findings, we learned that blind users wanted to leverage VAPAs for more productivity-oriented tasks and increased efficiency over other interaction modalities. We conclude that (1) VAPAs need to support a greater variety of AI personas, each specializing in a specific type of task; (2) VAPAs need to maintain continuity of voice interaction for both usability and accessibility; and (3) blind VAPA users, and especially blind technology podcasters, are expert voice interface users who should be incorporated into design processes from the beginning. We argue that when the blind lead the sighted through voice interface design, both blind and sighted users can benefit.

A Gesture-based Recognition System for Augmented Reality

With the geometrical improvement in Information Technology, current conventional input devices are becoming increasingly obsolete and lacking. Experts in Human Computer Interaction (HCI) are convinced that input devices remain the bottleneck of information acquisition specifically in when using Augmented Reality (AR) technology. Current input mechanisms are unable to compete with this trend towards naturalness and expressivity which allows users to perform natural gestures or operations and convert them as input. Hence, a more natural and intuitive input device is imperative, specifically gestural inputs that have been widely perceived by HCI experts as the next big input device. To address this gap, this project is set to develop a prototype of hand gesture recognition system based on computer vision in modeling basic human-computer interactions. The main motivation in this work is a technology that requires no outfitting of additional equipment whatsoever by the users. The gesture-based had recognition system was implemented using the Rapid Application Development (RAD) methodology and was evaluated in terms of its usability and performance through five levels of testing, which are unit testing, integration testing, system testing, recognition accuracy testing, and user acceptance testing. The test results of unit, integration, system testing as well as user acceptance testing produced favorable results. In conclusion, current conventional input devices will continue to bottleneck this advancement in technology; therefore, a better alternative input technique should be looked into, in particularly, gesture-based input technique which offers user a more natural and intuitive control.

Using Gesture, Gaze, and Combination Input Schemes as Alternatives to the Computer Mouse

Novel input devices can increase the bandwidth between users and their devices. Traditional desktop computing uses windows, icons, menus, and pointers – an interface built for the computer mouse and very effective for pointing-and-clicking. Alternative devices provide a variety of interactions including touch-free, gesture-based input and gaze-tracking to determine the user’s on-screen gaze location, but these input channels are not well-suited to a point-and-click interface. This study evaluates five new schemes, some multi-modal. These experimental schemes perform worse than mouse-based input for a picture sorting task, and motion-based gesture control creates more errors. Some gaze-based input has similar performance to the mouse while not creating additional workload.

Collaborative Haptic Exploration of Dynamic Remote Environments

Haptic perception is an important modality in reinforcing the presence of virtual or remote targets, but providing haptic feedback of dynamic environments still remains a challenging task. We address the issue of haptic telepresence systems by improving and integrating two independent approaches: real-time haptic rendering of unstructured spatial data and collaborative interaction with a remote partner. Contact with physical constraints is directly estimated from streaming point-cloud data without surface reconstruction, and haptic guidance cues that restrict or promote the users motion can be provided by both predefined triggers and gesture-based input from a helper. Through a user study with a proof-of-concept prototype, the authors show that the proposed approach significantly improves the performance of remote exploration tasks while enabling stable haptic interaction with real-world spatial data.

Mobile Mixed-Reality Interfaces That Enhance Human–Robot Interaction in Shared Spaces

Although user interfaces with gesture-based input and augmented graphics have promoted intuitive human-robot interactions (HRI), they are often implemented in remote applications on research-grade platforms requiring significant training and limiting operator mobility. This paper proposes a mobile mixed-reality interface approach to enhance HRI in shared spaces. As a user points a mobile device at the robot's workspace, a mixed-reality environment is rendered providing a common frame of reference for the user and robot to effectively communicate spatial information for performing object manipulation tasks, improving the user's situational awareness while interacting with augmented graphics to intuitively command the robot. An evaluation with participants is conducted to examine task performance and user experience associated with the proposed interface strategy in comparison to conventional approaches that utilize egocentric or exocentric views from cameras mounted on the robot or in the environment, respectively. Results indicate that, despite the suitability of the conventional approaches in remote applications, the proposed interface approach provides comparable task performance and user experiences in shared spaces without the need to install operator stations or vision systems on or around the robot. Moreover, the proposed interface approach provides users the flexibility to direct robots from their own visual perspective (at the expense of some physical workload) and leverages the sensing capabilities of the tablet to expand the robot's perceptual range.

Abstract No. 425 - Radiofrequency hyperthermia-enhanced local chemotherapy of esophageal squamous cancers

No. 425 Radiofrequency hyperthermia-enhanced local chemotherapy of esophageal squamous cancers Y. shi, F. Zhang, Z. Bai, J. Wang, L. Qiu, X. Yang; Radiology, Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University, Hangzhou, China; Radiology, University of Washington School of Medicine, Seattle, WA Purpose: To determine whether radiofrequency (RF) hyperthermia could enhance the therapeutic effect of cisplatin and 5-fluorouracil (5-FU) on esophageal squamous cancers (ESC). Materials and Methods: Human ESC cells (ESCC) were first labeled with red fluorescent protein (RFP) via a lentivirus transfection approach. For both in vitro confirmation and in vivo validation studies, RFP-ESCCs and 24 RFP-ESCengrafted mice were divided into four study groups with various treatments of (i) combination therapy with chemotherapy (cisplatin and 5-FU) plus MR imaging-heating-guidewire (MRIHG)-mediated local RF hyperthermia; (ii) chemotherapy only; (iii) RF hyperthermia only; and (iv) phosphate-buffered saline (PBS). In vitro cell proliferation was quantified by MTS assay, while in vivo validation with size changes of ESC masses and RFP-ESC signals among different treatment groups were monitored by ultrasound imaging and optical imaging over time with subsequent pathology correlation. Results: Of in vitro experiments, MTS assay demonstrated lowest cell proliferation of combination therapy compared to those of three control groups (23.4 7% VS 44.6 7.5% VS 95.8 2% VS 100 3%, po0.05). Of in vivo experiments, ultrasound imaging showed smaller tumor volumes with combination therapy than those with three control treatments (0.65 0.08 VS 1.27 0.04 VS 2.63 0.16 VS 2.71 0.19, Po0.05). Optical imaging demonstrated a decrease of RFPESC signals for the combination therapy group in comparison to those for three control groups (0.57 0.09 VS 1.37 0.17 VS 2.37 0.35 VS 2.54 0.35, Po0.05), which were well correlated with histologic confirmation. Conclusion: Local RF hyperthermia can enhance chemotherapeutic effect on human ESCs, which may open a new avenue for efficient management of esophageal malignancies. Educational Exhibit Abstract No. 426 Use of hands free gesture-based imaging control for vessel identification during hepatic transarterial chemoembolization and selective internal radiotherapy procedures Z.L. Bercu, V.V. Patil, R.S. Patel, E. Kim, S.F. Nowakowski, R.A. Lookstein, A.M. Fischman; Division of Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, NY Learning Objectives: 1. Understand the technical implementation of an inexpensive gesture-based input device for intra-procedural control of picture archiving and communications systems (PACS). 2. Understand the importance of vessel identification for transarterial chemoembolization (TACE) and selective internal radiotherapy (SIRT) procedures. 3. Understand how gesture-based device integration can lead to lower radiation doses to both the patient and operator through reduced need of angiograms. 4. Understand how gesture-based device integration can lead to faster, safer, and more accurate TACE and SIRT procedures. Background: TACE and MAA both require detailed appreciation for normal and variant hepatic anatomy. Identification of the cystic, gastric, gastroduodenal, falciform, and accessory arteries is critical to reduce the risk of nontarget chemoembolization or radiotherapy delivery. Patients may have variant anatomy difficult to deduce on angiography alone. Vessel identification is essential during subselective TACE and SIRT for complex disease. Use of CT/MRI is invaluable but often not available readily and delays case progression with the need to “scrub out.” The ability to review prior imaging in real-time in the angiosuite facilitates proper vessel identification, reduced need for radiation, and increased safety. Clinical Findings/Procedure Details: The Leap Motion Controller (Leap Motion, Inc., San Francisco, CA) allows for gesture mapping to input commands. It is inexpensive and Posters and Exhibits ’ JVIR S186 P os te rs an d Ex hi bi ts

Applying Fitts’ Law to Gesture Based Computer Interactions

As gesture interfaces become more main-stream, it is increasingly important to investigate the behavioral characteristics of these interactions – particularly in three-dimensional (3D) space. In this study, Fitts’ method was extended to such input technologies, and the applicability of Fitts’ law to gesture-based interactions was examined. The experiment included three gesture-based input devices that utilize different techniques to capture user movement, and compared them to conventional input technologies like touchscreen and mouse. Participants completed a target-acquisition test and were instructed to move a cursor from a home location to a spherical target as quickly and accurately as possible. Three distances and three target sizes were tested six times in a randomized order for all input devices. A total of 81 participants completed all tasks. Movement time, error rate, and throughput were calculated for each input technology. Results showed that the mean movement time was highly correlated with the target's index of difficulty for all devices, providing evidence that Fitts’ law can be extended and applied to gesture-based devices. Throughputs were found to be significantly lower for the gesture-based devices compared to mouse and touchscreen, and as the index of difficulty increased, the movement time increased significantly more for these gesture technologies. Error counts were statistically higher for all gesture-based input technologies compared to mouse. In addition, error counts for all inputs were highly correlated with target width, but little impact was shown by movement distance. Overall, the findings suggest that gesture-based devices can be characterized by Fitts’ law in a similar fashion to conventional 1D or 2D devices.

The design and evaluation of a tool to support software designers at the whiteboard

Despite the availability of advanced software design tools, software developers are known to frequently turn to the whiteboard when faced with a specific design problem. Free from any restrictions, they engage in an informal form of software design that relies heavily on sketching. However, whereas whiteboards afford flexibility and fluidity, they at the same time limit a designer in only being able to draw and erase content. This paper presents Calico, a novel software design tool that leverages electronic whiteboards to enhance the design experience with explicit support for the creative, exploratory aspects of design. Specifically, Calico offers a grid to partition the drawing space, scraps to organize and manipulate the content, and a reusable palette together with gesture-based input to address several natural behaviors exhibited by software designers, including frequent shifts in focus, use of low-detail models, use of a variety of notations, and a tendency to refine representations. To evaluate Calico, we performed a laboratory experiment involving the design of an educational traffic simulator. Through a three-pronged analysis examining the use of Calico’s features, design behaviors exhibited by the participants, and the nature of the design conversations, we show how Calico benefits the design process as it takes place at the whiteboard.

The easy ADL home: A physical-virtual approach to domestic living

Smart environments worthy of the name need to capture, interpret, and support human activities that take place within their realms. Most existing efforts tend to focus on either real world activities or activities taking place in the virtual world accessed through digital devices. However, as digital computation continues to permeate our everyday real world environments, and as the border between physical and digital continues to blur for the human agents acting in these environments, we need system design approaches that can cope with human activities that span the physical-virtual gap. In this paper, we present such an approach and use it for designing a smart home intended to support Activities of Daily Living (ADL). The easy ADL home is designed based on a wearable personal server that runs a personal ADL support middleware and a set of computationally augmented everyday objects within the easy ADL home. An initial qualitative study of the system involving 20 subjects revealed a highly positive attitude (score 4.37 out of 5) towards the system's capability of co-locating and synchronizing physical and virtual events throughout the everyday activity scenarios, while classical usability aspects in particular related to the gesture-based input (score 2.89 out of 5) leaves room for improvement.

Gesture based Input Device: An All Inertial Approach

In this paper, we develop a gesture-based input device equipped with accelerometers and gyroscopes. The sensors measure the inertial measurements, i.e., accelerations and angular velocities produced by the movement of the system when a user is inputting gestures on a plane surface or in a 3D space. The gyroscope measurements are integrated to give orientation of the device and consequently used to compensate the accelerations. The compensated accelerations are doubly integrated to yield the position of the device. With this approach, a user's gesture input trajectories can be recovered without any external sensors. Three versions of motion tracking algorithms are provided to cope with wide spectrum of applications. Then, a Bayesian network based recognition system processes the recovered trajectories to identify the gesture class. Experimental results convincingly show the feasibility and effectiveness of the proposed gesture input device. In order to show practical use of the proposed input method, we implemented a prototype system, which is a gesture-based remote controller (Magic Wand).

Model–based video tracking for gestural interaction

Among many techniques to interact with 3D environments, gesture-based input appears promising. However, due to insufficient computing hardware capabilities, such interfaces have to be built either upon standard tracking devices or using limited image-based video tracking algorithms. As today computing power tends to be more and more powerful, more complex video analysis such as real-time model-based tracking is at hand. Considering the use of a model-based approach to allow unencumbered input gives us the advantage of extracting a low-level hand description useful to build natural interfaces. The algorithm we developed relies on a 3D polygonal hand model. Its pose parametrization is iteratively refined so that its 2D projection matches more closely the input 2D image. Relying on the graphics hardware to handle fast 2D projection is critical, while adding more cameras is useful to cope with the occlusion issue.

Gestures with speech for graphic manipulation

This paper reports on the utility of gestures and speech to manipulate graphic objects. In the experiment described herein, three different populations of subjects were asked to communicate with a computer using either speech alone, gestures alone, or both. The task was the manipulation of a three-dimensional cube on the screen. They were asked to assume that the computer could see their hands, hear their voices, and understand their gestures and speech as well as a human could. A gesture classification scheme was developed to analyse the gestures of the subjects. A primary objective of the classification scheme was to determine whether common features would be found among the gestures of different users and classes of users. The collected data show a surprising degree of commonality among subjects in the use of gestures as well as speech. In addition to the uniformity of the observed manipulations, subjects expressed a preference for a combined gesture/speech interface. Furthermore, all subjects easily completed the simulated object manipulation tasks. The results of this research, and of future experiments of this type, can be applied to develop a gesture-based or gesture/speech-based system which enables computer users to manipulate graphic objects using easily learned and intuitive gestures to perform spatial tasks. Such tasks might include editing a three-dimensional rendering, controlling the operation of vehicles or operating virtual tools in three dimensions, or assembling an object from components. Knowledge about how people intuitively use gestures to communicate with computers provides the basis for future development of gesture-based input devices.