Freehand Gestural Interaction Research Articles

Freehand Gestural Selection with Haptic Feedback in Wearable Optical See-Through Augmented Reality

Augmented reality (AR) technologies can blend digital and physical space and serve a variety of applications intuitively and effectively. Specifically, wearable AR enabled by optical see-through (OST) AR head-mounted displays (HMDs) might provide users with a direct view of the physical environment containing digital objects. Besides, users could directly interact with three-dimensional (3D) digital artefacts using freehand gestures captured by OST HMD sensors. However, as an emerging user interaction paradigm, freehand interaction with OST AR still requires further investigation to improve user performance and satisfaction. Thus, we conducted two studies to investigate various freehand selection design aspects in OST AR, including target placement, size, distance, position, and haptic feedback on the hand and body. The user evaluation results indicated that 40 cm might be an appropriate target distance for freehand gestural selection. A large target size might lower the selection time and error rate, and a small target size could minimise selection effort. The targets positioned in the centre are the easiest to select, while those in the corners require extra time and effort. Furthermore, we discovered that haptic feedback on the body could lead to high user preference and satisfaction. Based on the research findings, we conclude with design recommendations for effective and comfortable freehand gestural interaction in OST AR.

Virtual Block Augmented Reality Game Using Freehand Gesture Interaction

Augmented Reality (AR) manages to bring a virtual environment into a real-world environment seamlessly. As AR has been recognised as advancing technology, AR brings future changes to the learning process. The goal of this study is to use freehand gestures to create a virtual block game in AR. First of all, the stages of this study are to explore block games and freehand movements by using Leap Motion. Secondly, the design and development of Leap Motion virtual block games, and thirdly, the implementation of free-hand gesture interaction virtual block games. The paper explains about virtual blocks AR game using freehand gesture. AR tracking system with real hand gesture recognition system is merged to execute the freehand gesture. A prototype virtual block has been described in this paper. The paper ends with the conclusion and future works.

Understanding freehand gestures: a study of freehand gestural interaction for immersive VR shopping applications

Unlike retail stores, in which the user is forced to be physically present and active during restricted opening hours, online shops may be more convenient, functional and efficient. However, traditional online shops often have a narrow bandwidth for product visualizations and interactive techniques and lack a compelling shopping context. In this paper, we report a study on eliciting user-defined gestures for shopping tasks in an immersive VR (virtual reality) environment. We made a methodological contribution by providing a varied practice for producing more usable freehand gestures than traditional elicitation studies. Using our method, we developed a gesture taxonomy and generated a user-defined gesture set. To validate the usability of the derived gesture set, we conducted a comparative study and answered questions related to the performance, error count, user preference and effort required from end-users to use freehand gestures compared with traditional immersive VR interaction techniques, such as the virtual handle controller and ray-casting techniques. Experimental results show that the freehand-gesture-based interaction technique was rated to be the best in terms of task load, user experience, and presence without the loss of performance (i.e., speed and error count). Based on our findings, we also developed several design guidelines for gestural interaction.

The Gesture Disagreement Problem in Free-hand Gesture Interaction

ABSTRACTAccurately understanding a user’s intention is often essential to the success of any interactive system. An information retrieval system, for example, should address the vocabulary problem (Furnas et al., 1987) to accommodate different query terms users may choose. A system that supports natural user interaction (e.g., full-body game and immersive virtual reality) must recognize gestures that are chosen by users for an action. This article reports an experimental study on the gesture choice for tasks in three application domains. We found that the chance for users to produce the same gesture for a given task is below 0.355 on average, and offering a set of gesture candidates can improve the agreement score. We discuss the characteristics of those tasks that exhibit the gesture disagreement problem and those tasks that do not. Based on our findings, we propose some design guidelines for free-hand gesture-based interfaces.

Freehand Gesture and Tactile Interaction for Shape Design

This paper presents a novel system that allows product designers to design, experience, and modify new shapes of objects, starting from existing ones. The system allows designers to acquire and reconstruct the 3D model of a real object and to visualize and physically interact with this model. In addition, the system allows designer to modify the shape through physical manipulation of the 3D model and to eventually print it using a 3D printing technology. The system is developed by integrating state-of-the-art technologies in the sectors of reverse engineering, virtual reality, and haptic technology. The 3D model of an object is reconstructed by scanning its shape by means of a 3D scanning device. Then, the 3D model is imported into the virtual reality environment, which is used to render the 3D model of the object through an immersive head mounted display (HMD). The user can physically interact with the 3D model by using the desktop haptic strip for shape design (DHSSD), a 6 degrees of freedom servo-actuated developable metallic strip, which reproduces cross-sectional curves of 3D virtual objects. The DHSSD device is controlled by means of hand gestures recognized by a leap motion sensor.

Towards the design of effective freehand gestural interaction for interactive TV

As interactive devices become pervasive, people are beginning to looking for more advanced interaction with televisions in the living room. Interactive television has the potential to offer a very engaging experience. But most common user tasks are still challenging with such systems, such as menu selection or text input. And little work has been done on understanding and sup-porting the effective design of freehand interaction with an TV in the living room. In this paper, we perform two studies investi-gating freehand gestural interaction with a consumer level sensor, which is suitable for TV scenarios. In the first study, we inves-tigate a range of design factors for tiled layout menu selection, including wearable feedback, push gesture depth, target size and position in motor space. The results show that tactile and audio feedback have no significant effect on performance and prefer-ence, and these results inform potential designs for high selection performance. In the second study, we investigate a common TV user task of text input using freehand gesture. We design and evaluate two virtual keyboard layouts and three freehand selec-tion methods. Results show that ease of use and error tolerance can be both achieved using a text entry method utilizing a dual circle layout and an expanding target selection technique. Finally, we propose design guidelines for effective, usable and com-fortable freehand gestural interaction for interactive TV based on the findings.

Computer-Assisted Culture Learning in an Online Augmented Reality Environment Based on Free-Hand Gesture Interaction

The physical-virtual immersion and real-time interaction play an essential role in cultural and language learning. Augmented reality (AR) technology can be used to seamlessly merge virtual objects with real-world images to realize immersions. Additionally, computer vision (CV) technology can recognize free-hand gestures from live images to enable intuitive interactions. Therefore, we incorporate the latest AR and CV algorithms into a Virtual English Classroom, called VECAR, to promote immersive and interactive language learning. By wearing a pair of mobile computing glasses, users can interact with virtual contents in a three-dimensional space by using intuitive free-hand gestures. We design three cultural learning activities that introduce students to authentic cultural products and new cultural practices, and allow them to examine various cultural perspectives. The objectives of the VECAR are to make cultural and language learning appealing, improve cultural learning effectiveness, and enhance interpersonal communication between teachers and students.

3D freehand gestural navigation for interactive public displays.

Users increasingly expect more-interactive experiences with public displays for applications including learning, gaming, urban visualization, and planning. However, user interaction with applications on public displays is challenging and often doesn't lend itself to established interaction techniques. Navigation using only freehand gestures captured with low-cost devices might often be an appropriate solution. A formal, quantitative lab experiment and an informal, qualitative field study compared freehand gestural interaction with keyboard-and-mouse interaction. No significant difference in navigation performance existed between the two interaction types. However, gestural interaction provided a more natural experience in both personal and public environments.

3D selection with freehand gesture

The use of 3D computer graphics is important in a very wide range of applications. However, user interaction with 3D applications is still challenging and often does not lend itself to established techniques that have been developed primarily for 2D desktop interaction. Meanwhile, 3D user interfaces that rely on tracking hand-held devices or fiducial markers attached to the user are cumbersome or entirely inappropriate in some situations. These challenges may be addressed by refining and building on the increasing use of freehand gestural input, i.e. without markers or hand-held devices, to extend the fluidity and immediacy of today's 2D touch-based interactions. In this paper, we analyze the characteristics of freehand gestural 3D interaction, and report a set of 3 related evaluation studies focused on the fundamental user interface task of object selection. We found that interaction design requiring a high accuracy single action are not appropriate for freehand gestural selection, while separating it into several connected low demand operations could be a potential solution; that our Reach technique is accurate and potentially useful for option selection tasks with freehand gesture; and that strong directional effects influence performance and usability of both 2D and 3D option selection. We propose guidelines for designers of 3D freehand gestural interaction based on our evaluation results.

Eyes-free interaction with free-hand gestures and auditory menus

Auditory interfaces can overcome visual interfaces when a primary task, such as driving, competes for the attention of a user controlling a device, such as radio. In emerging interfaces enabled by camera tracking, auditory displays may also provide viable alternatives to visual displays. This paper presents a user study of interoperable auditory and visual menus, in which control gestures remain the same in the visual and the auditory domain. Tested control methods included a novel free-hand gesture interaction with camera-based tracking, and touch screen interaction with a tablet. The task of the participants was to select numbers from a visual or an auditory menu including a circular layout and a numeric keypad layout. Results show, that even with participant's full attention to the task, the performance and accuracy of the auditory interface are the same or even slightly better than the visual when controlled with free-hand gestures. The auditory menu was measured to be slower in touch screen interaction, but questionnaire revealed that over half of the participants felt that the circular auditory menu was faster than the visual menu. Furthermore, visual and auditory feedback in touch screen interaction with numeric layout was measured fastest, touch screen with circular menu second fastest, and the free-hand gesture interface was slowest. The results suggest that auditory menus can potentially provide a fast and desirable interface to control devices with free-hand gestures.

Design and evaluation of freehand menu selection interfaces using tilt and pinch gestures

Freehand gestural interaction, in which the user’s hands move in mid-air to provide input, has been of interest to researchers, but freehand menu selection interfaces have been under-investigated so far. Freehand menu selection is inherently difficult, especially with increasing menu breadth (i.e., the number of items), largely because moving hands in free space cannot achieve precision as high as physical input devices such as mouse and stylus. We have designed a novel menu selection interface called the rapMenu ( Ni et al., 2008), which is controlled by wrist tilt and multiple pinch gestures, and takes advantage of the multiple discrete gesture inputs to reduce the required precision of the user hand movements. In this article, we first review the visual design and behavior of the rapMenu technique, as well as related design issues and its potential advantages. In the second part, we present two studies of the rapMenu in order to further investigate the strengths and limitations of the design principle. In the first study, we compared the rapMenu to the extensively studied tilt menu technique ( Rahman et al., 2009). Our results revealed that the rapMenu outperforms the tilt menu as menu breadth increases. In the second study, we investigated how the rapMenu affords the opportunity of eyes-free selection and users’ transition from novice to expert. We found that within 10 min of practice, eyes-free selection with rapMenu has competitive speed and accuracy with the visual rapMenu and the tilt menu. Finally, we discuss design variations that use other axes of wrist movement and adopt alternative auditory feedback.