Gesture Interface Research Articles

Omnis Prædictio: Estimating the full spectrum of human performance with stroke gestures

Designing effective, usable, and widely adoptable stroke gesture commands for graphical user interfaces is a challenging task that traditionally involves multiple iterative rounds of prototyping, implementation, and follow-up user studies and controlled experiments for evaluation, verification, and validation. An alternative approach is to employ theoretical models of human performance, which can deliver practitioners with insightful information right from the earliest stages of user interface design. However, very few aspects of the large spectrum of human performance with stroke gesture input have been investigated and modeled so far, leaving researchers and practitioners of gesture-based user interface design with a very narrow range of predictable measures of human performance, mostly focused on estimating production time, of which extremely few cases delivered accompanying software tools to assist modeling. We address this problem by introducing "Omnis Praedictio" (Omnis for short), a generic technique and companion web tool that provides accurate user-independent estimations of any numerical stroke gesture feature, including custom features specified in code. Our experimental results on three public datasets show that our model estimations correlate on average r > .9 with groundtruth data. Omnis also enables researchers and practitioners to understand human performance with stroke gestures on many levels and, consequently, raises the bar for human performance models and estimation techniques for stroke gesture input.

On attaining user-friendly hand gesture interfaces to control existing GUIs

BackgroundHand gesture interfaces are dedicated programs that principally perform hand tracking and hand gesture prediction to provide alternative controls and interaction methods. They take advantage of one of the most natural ways of interaction and communication, proposing novel input and showing great potential in the field of the human-computer interaction. Developing a flexible and rich hand gesture interface is known to be a time-consuming and arduous task. Previously published studies have demonstrated the significance of the finite-state-machine (FSM) approach when mapping detected gestures to GUI actions. MethodsIn our hand gesture interface, we broadened the FSM approach by utilizing gesture-specific attributes, such as distance between hands, distance from the camera, and time of occurrences, to enable users to perform unique GUI actions. These attributes are obtained from hand gestures detected by the RealSense SDK employed in our hand gesture interface. By means of these gesture-specific attributes, users can activate static gestures and perform them as dynamic gestures. We also provided supplementary features to enhance the efficiency, convenience, and user-friendliness of our hand gesture interface. Moreover, we developed a complementary application for recording hand gestures by capturing hand keypoints in depth and color images to facilitate the generation of hand gesture datasets. ResultsWe conducted a small-scale user survey with fifteen subjects to test and evaluate our hand gesture interface. Anonymous feedback obtained from the users indicates that our hand gesture interface is adequately facile and self-explanatory to use. In addition, we received constructive feedback about minor flaws regarding the responsiveness of the interface. ConclusionsWe proposed a hand gesture interface along with key concepts to attain user-friendliness and effectiveness in the control of existing GUIs.

Using infrared depth-sensing technology to improve the brachytherapy operating room experience

PurposeThe purpose of this study was to discuss the merits of using depth-sensing infrared camera technology in the brachytherapy operating room during interstitial brachytherapy for gynecologic malignancies. Materials and MethodsThe infrared depth-sensing camera from a Microsoft Kinect that had been adapted for surgical use was introduced into a high-volume interstitial brachytherapy operating room. Brachytherapists then used the touchless, gestural interface to review preoperative MRI in real time to guide needle insertion. ResultsThe interface was used for 10 consecutive procedures by 4 separate brachytherapists. The initial training and adjustment to the technology was variable among brachytherapists. All brachytherapists found the controls intuitive and were able to successfully navigate MRI on the system after 1, 30, 30, and 45 min. Qualitatively, brachytherapists found the system helpful for interpretation of intraoperative ultrasound imaging. Furthermore, it ensured adequate needle positioning and deposition was maintained for large tumors. Surgeons involved in its use agreed on potential for considerable benefit when performing interstitial brachytherapy. ConclusionsAdapting this technology for use in the brachytherapy suite provided a higher level of comfort with interstitial catheter placement. This novel tool or similar technology might be considered within other brachytherapy suites.

Challenges of Developing a Mobile Game for Children with Down Syndrome to Test Gestural Interface

After a literature review published by Nascimento et. al. (2017), the research team noticed the lack of studies focused on game controllers’ accessibility during use by children with Down syndrome. In view of that, this research describes a mobile game development and its usability analyses, which were created to evaluate the accessibility of touchscreen gestural interfaces. The methodology was organized into three steps: bibliographic research and the definition of the project guidelines, the game development, and its evaluation. The guidelines used were based on a study made by Nascimento et. al. (2019) of the impairments that children can have, their game preferences found on Prena’s article (2014), games accessibility guidelines for people with intellectual deficiency from the Includification Book (2012), a manual of touchscreen gestural interfaces from Android and iOS and a game development framework from Schuytema (2008). Then, for the usability analyses, the team decided to first submit the game to a group of experts in order to make some improvements before submitting it to the audience. In this way, two evaluations were done, a heuristic test with usability specialists and a cognitive walkthrough with health professionals. The list of heuristics used on the tests was created by a mash up of the Breyer evaluation (2008) and the recommendations of the Able Games Association (2012) and the cognitive one followed the Preece, Sharp and Rogers (2007) recommendations. The results found reveal some challenges in the field and adjustments, mainly in the narrative, game goals and interface feedback, that should be addressed as soon as possible.

Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network

Using hand gestures is a natural method of interaction between humans and computers. We use gestures to express meaning and thoughts in our everyday conversations. Gesture-based interfaces are used in many applications in a variety of fields, such as smartphones, televisions (TVs), video gaming, and so on. With advancements in technology, hand gesture recognition is becoming an increasingly promising and attractive technique in human–computer interaction. In this paper, we propose a novel method for fingertip detection and hand gesture recognition in real-time using an RGB-D camera and a 3D convolution neural network (3DCNN). This system can accurately and robustly extract fingertip locations and recognize gestures in real-time. We demonstrate the accurateness and robustness of the interface by evaluating hand gesture recognition across a variety of gestures. In addition, we develop a tool to manipulate computer programs to show the possibility of using hand gesture recognition. The experimental results showed that our system has a high level of accuracy of hand gesture recognition. This is thus considered to be a good approach to a gesture-based interface for human–computer interaction by hand in the future.

Skeleton-Based Dynamic Hand Gesture Recognition Using a Part-Based GRU-RNN for Gesture-Based Interface

Recent improvements in imaging sensors and computing units have led to the development of a range of image-based human-machine interfaces (HMIs). An important approach in this direction is the use of dynamic hand gestures for a gesture-based interface, and some methods have been developed to provide real-time hand skeleton generation from depth images for dynamic hand gesture recognition. Towards this end, we propose a skeleton-based dynamic hand gesture recognition method that divides geometric features into multiple parts and uses a gated recurrent unit-recurrent neural network (GRU-RNN) for each feature part. Because each divided feature part has fewer dimensions than an entire feature, the number of hidden units required for optimization is reduced. As a result, we achieved similar recognition performance as the latest methods with fewer parameters.

Evaluation of Pattern Recognition Methods for Head Gesture-Based Interface of a Virtual Reality Helmet Equipped with a Single IMU Sensor.

The motivation of this paper is to examine the effectiveness of state-of-the-art and newly proposed motion capture pattern recognition methods in the task of head gesture classifications. The head gestures are designed for a user interface that utilizes a virtual reality helmet equipped with an internal measurement unit (IMU) sensor that has 6-axis accelerometer and gyroscope. We will validate a classifier that uses Principal Components Analysis (PCA)-based features with various numbers of dimensions, a two-stage PCA-based method, a feedforward artificial neural network, and random forest. Moreover, we will also propose a Dynamic Time Warping (DTW) classifier trained with extension of DTW Barycenter Averaging (DBA) algorithm that utilizes quaternion averaging and a bagged variation of previous method (DTWb) that utilizes many DTW classifiers that perform voting. The evaluation has been performed on 975 head gesture recordings in seven classes acquired from 12 persons. The highest value of recognition rate in a leave-one-out test has been obtained for DTWb and it equals 0.975 (0.026 better than the best of state-of-the-art methods to which we have compared our approach). Among the most important applications of the proposed method is improving life quality for people who are disabled below the neck by supporting, for example, an assistive autonomous power chair with a head gesture interface or remote controlled interfaces in robotics.

Comparing head gesture, hand gesture and gamepad interfaces for answering Yes/No questions in virtual environments

A potential application of gesture recognition algorithms is to use them as interfaces to interact with virtual environments. However, the performance and the user preference of such interfaces in the context of virtual reality (VR) have been rarely studied. In the present paper, we focused on a typical VR interaction scenario—answering Yes/No questions in VR systems to compare the performance and the user preference of three types of interfaces. These interfaces included a head gesture interface, a hand gesture interface and a conventional gamepad interface. We designed a memorization task, in which participants were asked to memorize several everyday objects presented in a virtual room and later respond to questions on whether they saw a specific object through the given interfaces when these objects were absent. The performance of the interfaces was evaluated in terms of the real-time accuracy and the response time. A user interface questionnaire was also used to reveal the user preference for these interfaces. The results showed that head gesture is a very promising interface, which can be easily added to existing VR systems for answering Yes/No questions and other binary responses in virtual environments.

Relative Pointing Interface: A gesture interaction method based on the ability to divide space

A new type of 3D gesture interface called Relative Pointing Interface (RelPoInt) is suggested in this study. RelPoInt allows users to select a virtual button by pointing to a relative position in a virtual 3D menu. This approach makes use of the distance or angle between the relative point and the reference point. Users can set the reference point freely and operate all the functions on the menu with a simple gesture. Microsoft Kinect was used to implement the RelPoInt in this study. To suggest a radial menu suitable for the RelPoInt, we conducted an experiment in which 20 participants were asked to point to virtual buttons, which were divided into four to eight regions. Errors and subjective assessment of easiness were measured at each treatment condition. RelPoInt can make users to manipulate functions without memorizing complicated gesture vocabularies, which is among the most serious pain points of using a gesture interface. Furthermore, RelPoInt does not require expensive gesture recognition equipment compared with finger recognition interfaces because the required resolution of interaction recognition is less sophisticated. It can be useful in applying 3D gesture interfaces to a variety of devices for smart home, virtual or augmented reality. Relevance to industry The proposed interface can be applied to various environments with low cost and easy to implement. Therefore, this interface is highly worthy of use in areas where gestures are useful to interact, such as IoT (Internet of Things) and VR (Virtual Reality)/AR (Augmented Reality).

3D User Interface Design and Usability for Immersive VR

This work presents a novel design of a new 3D user interface for an immersive virtual reality desktop and a new empirical analysis of the proposed interface using three interaction modes. The proposed novel dual-layer 3D user interface allows for user interactions with multiple screens portrayed within a curved 360-degree effective field of view available for the user. Downward gaze allows the user to raise the interaction layer that facilitates several traditional desktop tasks. The 3D user interface is analyzed using three different interaction modes, point-and-click, controller-based direct manipulation, and a gesture-based user interface. A comprehensive user study is performed within a mixed-methods approach for the usability and user experience analysis of all three user interaction modes. Each user interaction is quantitatively and qualitatively analyzed for simple and compound tasks in both standing and seated positions. The crafted mixed approach for this study allows to collect, evaluate, and validate the viability of the new 3D user interface. The results are used to draw conclusions about the suitability of the interaction modes for a variety of tasks in an immersive Virtual Reality 3D desktop environment.

A User-Centered Framework for Designing Midair Gesture Interfaces

Due to the recent advances in technologies for gesture recognition, midair gestures can be considered the interface of the future in a large number of applications. However, designing effective interfaces with midair gestures is not an easy task because the design is application dependent and it must fulfill many requirements at the same time. Despite the availability of general guidelines in the literature, clear and well-established procedures for the optimal design of midair gesture-based interfaces are, to date, not available and remain an open issue. The main contribution of this paper is a user-centered modular framework, which integrates existing and novel methods. It supports the designer considering multiple aspects including ergonomics, memorability, and specific user requirements tailored to the application scenario. The framework involves three design steps and a final validation step, also supported by dedicated software. We tested with success the proposed framework in an industrial case study, where technicians must easily access technical information by browsing digital manuals during maintenance operations.

A Self-powered Always-on Vision-based Wake-up Detector for Wearable Gesture User Interfaces

A Self-powered Always-on Vision-based Wake-up Detector for Wearable Gesture User Interfaces

A Low-power Real-time Hidden Markov Model Accelerator for Gesture User Interface on Wearable Devices

A Low-power Real-time Hidden Markov Model Accelerator for Gesture User Interface on Wearable Devices

Feel the noise: Mid-air ultrasound haptics as a novel human-vehicle interaction paradigm

Focussed ultrasound can be used to create the sensation of touch in mid-air. Combined with gestures, this can provide haptic feedback to guide users, thereby overcoming the lack of agency associated with pure gestural interfaces, and reducing the need for vision – it is therefore particularly apropos of the driving domain. In a counter-balanced 2 × 2 driving simulator study, a traditional in-vehicle touchscreen was compared with a virtual mid-air gestural interface, both with and without ultrasound haptics. Forty-eight experienced drivers (28 male, 20 female) undertook representative in-vehicle tasks – discrete target selections and continuous slider-bar manipulations – whilst driving. Results show that haptifying gestures with ultrasound was particularly effective in reducing visual demand (number of long glances and mean off-road glance time), and increasing performance (shortest interaction times, highest number of correct responses and least ‘overshoots’) associated with continuous tasks. In contrast, for discrete, target-selections, the touchscreen enabled the highest accuracy and quickest responses, particularly when combined with haptic feedback to guide interactions, although this also increased visual demand. Subjectively, the gesture interfaces invited higher ratings of arousal compared to the more familiar touch-surface technology, and participants indicated the lowest levels of workload (highest performance, lowest frustration) associated with the gesture-haptics interface. In addition, gestures were preferred by participants for continuous tasks. The study shows practical utility and clear potential for the use of haptified gestures in the automotive domain.

Mouth Gesture Interface for VLC Media Player

This proposed work presents a frame work of mouth gesture recognition for Human Computer Interface (HCI). It replaces the traditional input devices such as mouse and keyboard which allows a user to work on a computer using his/her mouth gestures. This work is aimed at helping severely disabled and paralyzed people. The entire work includes mouth detection, region extraction, gesture classification, and interface creation with computer applications. Initially face and mouth regions are detected using Haar-cascaded classifier. Secondly, the gesture recognition is done using the concept of Deep learning through Convolutional Neural Network (CNN). The mouth gestures are recognized and classified as mouth close, mouth open, tongue left and tongue right. Finally an HCI is created by mapping the mouth gestures into VLC player operations such as play, pause, forward jump and backward jump. The performance of the proposed method is measured and compared with other existing methods. This work is found to perform better than the other methods.

Robot Learning of Assistive Manipulation Tasks by Demonstration via Head Gesture-based Interface.

Assistive robotic manipulators have the potential to support the lives of people suffering from severe motor impairments. They can support individuals with disabilities to independently perform daily living activities, such as drinking, eating, manipulation tasks, and opening doors. An attractive solution is to enable motor impaired users to teach a robot by providing demonstrations of daily living tasks. The user controls the robot 'manually' with an intuitive human-robot interface to provide demonstration, which is followed by the robot learning of the performed task. However, the control of robotic manipulators by motor impaired individuals is a challenging topic. In this paper, a novel head gesture-based interface for hands-free robot control and a framework for robot learning from demonstration are presented. The head gesture-based interface consists of a camera mounted on the user's hat, which records the changes in the viewed scene due to the head motion. The head gesture recognition is performed using the optical flow for feature extraction and support vector machine for gesture classification. The recognized head gestures are further mapped into robot control commands to perform object manipulation task. The robot learns the demonstrated task by generating the sequence of actions and Gaussian Mixture Model method is used to segment the demonstrated path of the robot's end-effector. During the robotic reproduction of the task, the modified Gaussian Mixture Model and Gaussian Mixture Regression are used to adapt to environmental changes. The proposed framework was evaluated in a real-world assistive robotic scenario in a small study involving 13 participants; 12 able-bodied and one tetraplegic. The presented results demonstrate a potential of the proposed framework to enable severe motor impaired individuals to demonstrate daily living tasks to robotic manipulators.

Effects of Varying Noise Levels and Lighting Levels on Multimodal Speech and Visual Gesture Interaction with Aerobots

This paper investigated the effects of varying noise levels and varying lighting levels on speech and gesture control command interfaces for aerobots. The aim was to determine the practical suitability of the multimodal combination of speech and visual gesture in human aerobotic interaction, by investigating the limits and feasibility of use of the individual components. In order to determine this, a custom multimodal speech and visual gesture interface was developed using CMU (Carnegie Mellon University) sphinx and OpenCV (Open source Computer Vision) libraries, respectively. An experiment study was designed to measure the individual effects of each of the two main components of speech and gesture, and 37 participants were recruited to participate in the experiment. The ambient noise level was varied from 55 dB to 85 dB. The ambient lighting level was varied from 10 Lux to 1400 Lux, under different lighting colour temperature mixtures of yellow (3500 K) and white (5500 K), and different background for capturing the finger gestures. The results of the experiment, which consisted of around 3108 speech utterance and 999 gesture quality observations, were presented and discussed. It was observed that speech recognition accuracy/success rate falls as noise levels rise, with 75 dB noise level being the aerobot’s practical application limit, as the speech control interaction becomes very unreliable due to poor recognition beyond this. It was concluded that multi-word speech commands were considered more reliable and effective than single-word speech commands. In addition, some speech command words (e.g., land) were more noise resistant than others (e.g., hover) at higher noise levels, due to their articulation. From the results of the gesture-lighting experiment, the effects of both lighting conditions and the environment background on the quality of gesture recognition, was almost insignificant, less than 0.5%. The implication of this is that other factors such as the gesture capture system design and technology (camera and computer hardware), type of gesture being captured (upper body, whole body, hand, fingers, or facial gestures), and the image processing technique (gesture classification algorithms), are more important in developing a successful gesture recognition system. Some further works were suggested based on the conclusions drawn from this findings which included using alternative ASR (Automatic Speech Recognition) speech models and developing more robust gesture recognition algorithm.

Digital play in young children’s numeracy learning

Advances in technology have seen a proliferation of touch-screen interfaces available to young children. These screens have changed the way in which young children engage with digital technology; with increased exposure and use, it raises debates about the suitability of integrating digital technology within early childhood settings. There are limited empirical studies that investigate the appropriateness of emergent digital technology within the field of early childhood. Based upon the TPACK model, this timely paper thus discussed the Digital Play Model (Numeracy) and reported on a project investigating the integration of digital technologies into childcare settings. In particular, the focus of this paper was to report on the use of games using a gestural interface device have on 3- to 4-year-old children’s ability to learn numeracy concepts. This study found that young children’s numeracy learning outcomes were improved. This paper also provided empirical evidence of the use of Apps via iPad technologies on young children’s numeracy learning, with an implication to a need for a pedagogical model for successful integration of digital technology within early childhood settings.

Speech analysis for conceptual CAD modeling using multi-modal interfaces: An investigation into Architects’ and Engineers’ speech preferences

AbstractSpeech- and gesture-based interfaces for computer-aided design (CAD) modeling must employ vocabulary suitable for target professional groups. We conducted an experiment with 40 participants from architecture and engineering backgrounds to elicit their speech preferences for four CAD manipulation tasks: Scale, Rotate, Copy, and Move. We compiled speech command terms used by participants and analyzed verbalizations based on three analytic themes: the exactness of descriptions, the granularity of descriptions, and the use of CAD legacy terms. We found that participants from both groups used precise and vague expressions in their verbalizations and used a median of three parameters in their verbalizations. Architects used CAD legacy terms more than Engineers in the tasks Scale and Rotate. Based on these findings, we give recommendations for the design of speech- and gesture-based interface for conceptual CAD modeling.

Let’s Play a Game! Kin-LDD: A Tool for Assisting in the Diagnosis of Children with Learning Difficulties

This paper presents an alternative approach for the diagnosis of learning difficulties in children. A game-based evaluation study, using Kinaesthetic Learning Difficulties Diagnosis (Kin-LDD), was performed during the actual diagnosis procedure for the identification of learning difficulties. Kin-LDD is a serious game that provides a gesture-based interface and incorporates spatial and time orientation activities. These activities assess children’s cognitive attributes while they are using their motor skills to interact with the game. The aim of this work was to introduce the fun parameter to the diagnostic process, provide a useful tool for the special educators and investigate potential correlations between in-game metrics and the diagnosis outcome. An experiment was conducted in which 30 children played the game during their official assessment for the diagnosis of learning difficulties at the Center for Differential Diagnosis, Diagnosis and Support. Performance metrics were collected automatically while the children were playing the game. These metrics, along with questionnaires appropriate for children and post-session interviews were later analyzed and the findings are presented in the paper. According to the results: (a) children evaluated the game as a fun experience, (b) special educators claimed it was helpful to the diagnostic procedure, and (c) there were statistically significant correlations between in-game metrics and the category of learning difficulty the child was characterized with.