Natural Interaction Modalities Research Articles

A novel static and dynamic hand gesture recognition using self organizing map with deep convolutional neural network

Gesture recognition has gained a lot of popularity as it allows humans to communicate with real or virtual systems through gestures, offering new and natural interaction modalities. Recent technologies, such as augmented reality (AR) and the Internet of Things (IoT), have witnessed enormous growth in computer applications that focus on human–computer interaction (HCI). However, a few of these tactics make use of a combination of methods, such as image segmentation, pre-processing, and classification. The hessian-based multiscale filtering and YCbCr colour space are used to separate the gesture region to be recognized. A modified marker-controlled watershed method is employed to segment the gesture contour along with the eight-connector graph to increase recognition precision. The proposed hand gesture recognition methodology uses Self Organizing Map (SOM) with Deep Convolutional Neural Network (DCNN) provides better results with fast convergence speed. Experiments were carried out on a dataset of 30 static and 6 dynamic gestures and also evaluated on a publicly available IIITA-ROBITA ISL Gesture Database to show the effectiveness. The results show that the suggested method can recognize gesture classes with 95.63% accuracy rate without significantly affecting the recognition time. The proposed algorithm was then implemented to control household appliances.

SoundLip

As a natural and convenient interaction modality, voice input has now become indispensable to smart devices (e.g. mobile phones and smart appliances). However, voice input is strongly constrained by surroundings and may raise privacy leakage in public areas. In this paper, we present SoundLip, an end-to-end interaction system enabling users to interact with smart devices via silent voice input. The key insight is to use inaudible acoustic signals to capture the lip movements of users when they issue commands. Previous works have considered lip reading as a naive classification task and thus can only recognize individual words. In contrast, our proposed system enables lip reading at both word and sentence levels, which are more suitable for daily-life use. We exploit the built-in speakers and microphones of smart devices to emit acoustic signals and listen to their reflections, respectively. In order to better abstract representations from multi-frequency and multi-modality acoustic signals, we elaborate a hierarchical convolutional neural network (HCNN) to serve as the front-end as well as recognize individual word commands. Then, for the sentence-level recognition, we exploit a multi-task encoder-decoder network to get around temporal segmentation and output sentences in an end-to-end way. We evaluate SoundLip on 20 individual words and 70 sentences from 12 participants. Our system achieves an accuracy of 91.2% at word-level and a word error rate of 7.1% at sentence-level in both user-independent and environment-independent settings. Given its innovative solution and promising performance, we believe that SoundLip has made a significant contribution to the advancement of silent voice input technology.

A Minimally Invasive Low-Power Platform for Real-Time Brain Computer Interaction Based on Canonical Correlation Analysis

A growing trend in human–computer interaction is to integrate computational capabilities into wearable devices, to enable sophisticated and natural interaction modalities. Acting directly by decoding neural activity is a very natural way of interaction and one of the fundamental paradigms of brain computer interfaces (BCIs) as well. In this paper, we present a wearable Internet of Things node designed for BCI spelling. The system is based on visual evoked potentials detection and runs the canonical correlation analysis on a low power microcontroller. Neural data is acquired by an array of electroencephalography active dry electrodes, suitable for a minimally intrusive interface. To evaluate our solution, we optimized the system on eight subjects and tested it on five different subjects for four and eight stimuli, reaching a peak transfer rate of 1.57 b/s, comparable with those achieved by state-of-the-art nonembedded systems. The power consumption of the device is less than 30 mW, resulting in 122 h of operation with a standard 1000-mAh battery.

Collective expression: how robotic swarms convey information with group motion

Abstract When faced with the need of implementing a decentralized behavior for a group of collaborating robots, strategies inspired from swarm intelligence often avoid considering the human operator, granting the swarm with full autonomy. However, field missions require at least to share the output of the swarm to the operator. Unfortunately, little is known about the users’ perception of group behavior and dynamics, and there is no clear optimal interaction modality for swarms. In this paper, we focus on the movement of the swarm to convey information to a user: we believe that the interpretation of artificial states based on groups motion can lead to promising natural interaction modalities. We implement a grammar of decentralized control algorithms to explore their expressivity. We define the expressivity of a movement as a metric to measure how natural, readable, or easily understandable it may appear. We then correlate expressivity with the control parameters for the distributed behavior of the swarm. A first user study confirms the relationship between inter-robot distance, temporal and spatial synchronicity, and the perceived expressivity of the robotic system. We follow up with a small group of users tasked with the design of expressive motion sequences to convey internal states using our grammar of algorithms. We comment on their design choices and we assess the interpretation performance by a larger group of users. We show that some of the internal states were perceived as designed and discuss the parameters influencing the performance.

A Graphical Authentication System Controlled by Eye Movement

Graphical password is a potential alternative to the traditional alphanumerical password due to its advantages of recognition and memory in the security system. Eye movement, a natural interaction modality, is a promising tool to replace the keyboard and the mouse in authentication. This paper proposes a graphical authentication system controlled by eye movement. In this keyboard-less authentication system, the user’s eye movements were recorded by eye-tracker as the control signal to generate recognition-based graphical passwords. The user test study has been conducted to verify the availability and effectiveness of this keyboard-less graphical authentication system. It’s also shown that the efficiency could be improved as the user getting familiar with this system.

GraSp: Combining Spatially‐aware Mobile Devices and a Display Wall for Graph Visualization and Interaction

AbstractGoing beyond established desktop interfaces, researchers have begun re‐thinking visualization approaches to make use of alternative display environments and more natural interaction modalities. In this paper, we investigate how spatially‐aware mobile displays and a large display wall can be coupled to support graph visualization and interaction. For that purpose, we distribute typical visualization views of classic node‐link and matrix representations between displays. The focus of our work lies in novel interaction techniques that enable users to work with personal mobile devices in combination with the wall. We devised and implemented a comprehensive interaction repertoire that supports basic and advanced graph exploration and manipulation tasks, including selection, details‐on‐demand, focus transitions, interactive lenses, and data editing. A qualitative study has been conducted to identify strengths and weaknesses of our techniques. Feedback showed that combining mobile devices and a wall‐sized display is useful for diverse graph‐related tasks. We also gained valuable insights regarding the distribution of visualization views and interactive tools among the combined displays.

Towards the improvement of the cognitive, motoric and academic skills of students with special educational needs using Kinect learning games

There is an ongoing interest in developing motion Kinect-based learning games that are in accordance with intervention protocols for helping children with special educational needs. These games, which offer natural user interaction modalities, seem to be very beneficial for this population because of the combination of physical activities with cognitive-training tasks. There is still an open research issue about how to integrate such games into schools and how to organize systematic evaluation studies for showing their added value. This paper presents the positive findings of a pilot research study in inclusive classroom settings that were carried out at two primary schools with 20 children who have special educational needs and who used the Kinems suite that contains movement-based educational games for such children. Analysis of data gathered via pre- and post-test questionnaires, interviews and an in-depth study of kinetic and learning analytics showed that these games have a positive impact on childrens academic performance and improvement of their cognitive, motor and academic skills.

Comparison of Interaction Modalities for Mobile Indoor Robot Guidance: Direct Physical Interaction, Person Following, and Pointing Control

Three advanced natural interaction modalities for mobile robot guidance in an indoor environment were developed and compared using two tasks and quantitative metrics to measure performance and workload. The first interaction modality is based on direct physical interaction requiring the human user to push the robot in order to displace it. The second and third interaction modalities exploit a 3-D vision-based human-skeleton tracking allowing the user to guide the robot by either walking in front of it or by pointing toward a desired location. In the first task, the participants were asked to guide the robot between different rooms in a simulated physical apartment requiring rough movement of the robot through designated areas. The second task evaluated robot guidance in the same environment through a set of waypoints, which required accurate movements. The three interaction modalities were implemented on a generic differential drive mobile platform equipped with a pan-tilt system and a Kinect camera. Task completion time and accuracy were used as metrics to assess the users’ performance, while the NASA-TLX questionnaire was used to evaluate the users’ workload. A study with 24 participants indicated that choice of interaction modality had significant effect on completion time ( $F(2,61) = 84.874$ , $p ), accuracy ( $F(2,29) = 4.937$ , $p = 0.016$ ), and workload ( $F(2,68) = 11.948$ , $p ). The direct physical interaction required less time, provided more accuracy and less workload than the two contactless interaction modalities. Between the two contactless interaction modalities, the person-following interaction modality was systematically better than the pointing-control one: The participants completed the tasks faster with less workload.

Adding pluggable and personalized natural control capabilities to existing applications.

Advancements in input device and sensor technologies led to the evolution of the traditional human-machine interaction paradigm based on the mouse and keyboard. Touch-, gesture- and voice-based interfaces are integrated today in a variety of applications running on consumer devices (e.g., gaming consoles and smartphones). However, to allow existing applications running on desktop computers to utilize natural interaction, significant re-design and re-coding efforts may be required. In this paper, a framework designed to transparently add multi-modal interaction capabilities to applications to which users are accustomed is presented. Experimental observations confirmed the effectiveness of the proposed framework and led to a classification of those applications that could benefit more from the availability of natural interaction modalities.

Affective Stack — A Model for Affective Computing Application Development

Affective computing applications hold promise to revolutionize human-computer interaction by enabling more natural and intuitive interaction modalities. These may include the communication of vocal expressions, physiological signals or other non-verbal indicators of underlying affective state. Although the field has experienced substantial expansion in recent years, the tools and techniques utilized are not yet mature or well established. A notable issue is the one-off nature of currently implemented affective computing systems. There is as yet no notion of standardized program architecture and there is no straightforward way to extend the functionality of existing software to include affective components. This paper introduces a new model which describes the affective computing application in terms of a set of loosely coupled functional components. This model encourages a uniform and replicable approach to affective application development in which functional components can be improved independently and subsequently re-used. The model also incorporates existing third party software as a functional component highlighting the potential to build upon existing, well established, software packages. It is hoped that this model and discussion spurs further, focused development in this growing field.

ImageSpirit

Humans describe images in terms of nouns and adjectives while algorithms operate on images represented as sets of pixels. Bridging this gap between how humans would like to access images versus their typical representation is the goal of image parsing, which involves assigning object and attribute labels to pixels. In this article we propose treating nouns as object labels and adjectives as visual attribute labels. This allows us to formulate the image parsing problem as one of jointly estimating per-pixel object and attribute labels from a set of training images. We propose an efficient (interactive time) solution. Using the extracted labels as handles, our system empowers a user to verbally refine the results. This enables hands-free parsing of an image into pixel-wise object/attribute labels that correspond to human semantics. Verbally selecting objects of interest enables a novel and natural interaction modality that can possibly be used to interact with new generation devices (e.g., smartphones, Google Glass, livingroom devices). We demonstrate our system on a large number of real-world images with varying complexity. To help understand the trade-offs compared to traditional mouse-based interactions, results are reported for both a large-scale quantitative evaluation and a user study.