Intuitive Interaction Methods Research Articles

High-Accuracy Hand Gesture Recognition on the Wrist Tendon Group Using Pneumatic Mechanomyography (pMMG)

Hand gesture recognition has received considerable attention as an intuitive interaction method in recent years. This research introduces a new wearable hand gesture recognition system that employs pneumatic mechanomyography (pMMG) to directly monitor the wrist tendon group, which transmits muscle force to the fingers. The experimental findings demonstrate that the proposed method provides raw observations proportional to the finger flexion force, with high R-squared values exceeding 0.94. The performance of the proposed system was evaluated by conducting a hand gesture experiment consisting of 28 hand gestures. The proposed method achieved an average accuracy of 98.12%, surpassing the surface electromyography (sEMG) system's accuracy of 93.89%. Furthermore, the fusion of pMMG and sEMG sensors yielded an accuracy of 99.18%. The results suggest that the proposed approach exhibits the potential to enhance the accuracy and efficiency of hand gesture recognition systems.

A Review on Dynamic Hand Gesture Recognition Techniques

Abstract: Body language is one of the nonverbal methodsof communication, and it comprises hand gestures, arm movements, posturing, and gestures and facial expressions. One way to communicate information through the movement of the body is through gestures. HGR is a smart, intuitive, and easy method of human-computer interaction (HCI). HGR systems have two key applications: SLR and GBC. To help the deaf communicate with the hearing community, SLR tries to automaticallyinterpret SLs via a computer. The idea that SL is a highly ordered and primarily symbolic collection of human gestures is what led to the development of universal gesture-based HCI.

Video Hand Gestures Recognition Using Depth Camera and Lightweight CNN

Hand gestures are a well-known and intuitive method of human-computer interaction. The majority of the research has concentrated on hand gesture recognition from the RGB images, however, little work has been done on recognition from videos. In addition, RGB cameras are not robust in varying lighting conditions. Motivated by this, we present the video based hand gestures recognition using the depth camera and a light weight convolutional neural network (CNN) model. We constructed a dataset and then used a light weight CNN model to detect and classify hand movements efficiently. We also examined the classification accuracy with a limited number of frames in a video gesture. We compare the depth camera’s video gesture recognition performance to that of the RGB camera. We evaluate the proposed model’s performance on edge computing devices and compare to benchmark models in terms of accuracy and inference time. The proposed model results in an accuracy of 99.48% on the RGB version of test dataset and 99.18% on the depth version of test dataset. Finally, we compare the accuracy of the proposed light weight CNN model with the state-of-the hand gesture classification models.

Watch & Do: A Smart IoT Interaction System with Object Detection and Gaze Estimation

The Internet of Things (IoT) attempts to help people access Internet-connected devices, applications, and services anytime and anywhere. However, how providing an efficient and intuitive method of interaction between people and IoT devices is still an open challenge. In this paper, we propose a novel interaction system called Watch & Do , where users can control an IoT device by gazing at it and doing simple gestures. The proposed system mainly consists of: 1) object detection module; 2) gaze estimation module; 3) hand gesture recognition module; and 4) IoT controller module. The target device is identified by various deep learning-based gaze estimation and object detection techniques. Afterwards, hand gesture recognition is applied to generate an IoT device control command which is transmitted to the IoT platform. The experimental results and case studies demonstrate the feasibility of the proposed system and imply the future research directions.

Visualization and interaction of finite element analysis in augmented reality

In this paper, augmented reality (AR) is used to enhance the visualization and interaction of finite element analysis (FEA) of structures. An integrated simulation system is proposed which acquires input data using sensors and uses AR technology to visualize FEA results in the real world. A number of intuitive interaction methods have been devised and implemented in this system. The user can perform real-time FEA simulation to investigate structural behavior under different loading conditions either through manipulating virtual loads or creating different loading conditions. Exploration of FEA results is enhanced through natural interfaces for manipulating, slicing and clipping the result data. Moreover, the user can modify the FE models of the structures through simplified operations for different purposes, e.g., adding structural members for stiffening and performing local mesh refinement. The modified model can be re-analyzed automatically. A prototype system has been built and a case study has been implemented to demonstrate the innovative interaction methods and evaluate the system performance.

Digital and tactile drawing-system for blind users

In my PhD-Task I want to allow blind users to create digital drawings on their own and, in particular, support tactile drawing by fingers. As the common mechanisms for creating tactile graphics by blind people are analog or text-based methods, editing graphics is a hard or nearly impossible task. I want to build a system that allows a blind user to create tactile graphics with intuitive interaction methods, such as drawing with the fingers, drawing with a pen as well as using stamps and tangibles. Technically this can be realized with a digitalization pen, a standard consumer time of flight (TOF) camera system for input and a two-dimensional touch-sensitive tactile pin-matrix device for output. With this system blind users will get for the first time the freedom of their own digital pictorial expression with all the benefits of digital drawing tools that visual users have for several decades.

An intuitive user interaction method using multi-sensors for pencil drawing filter of NPR rendering in mobile devices

The user interface is one of the most important factors to use various functions of mobile device easily. Recently, the importance of the user interface has been increased after Apple emphasized emotional UX(User eXperience) by applying multi-touches to iPhone. Mobile device Manufacturers have tried to apply embedded sensors to apps as user interface. Microphone was applied to execute functions combined with voice recognition and GPS sensor was used to find a location in navigation apps or game apps. Like these services, several sensors have roles of interface between user and device. In this paper, we propose an intuitive user interaction method using multi-sensors in NPR(Non-Photorealistic) rendering. The proposed method renders pencil drawing filter for a given photo image with the direction and strength of line strokes which are applied in vector fields. At this point, the method changes direction of stroke with x-coordinate values and y-coordinate values and control the strength of the stroke with z-coordinate values after catching three coordinate values in real-time from the orientation sensor on a mobile device. The method also adjusts brightness of image by transforming intensity of light from the light sensor. The light sensor updates intensity of light source continuously. If the user brings the mobile device closer to the external light source, the light sensor value will increase and the intensity of brightness becomes stronger. In addition, the user can make an attenuation effect with orientation sensor combined with light sensor for specific scope in a photo image and change the attenuation scope by moving action of his holding mobile device. This is a user interaction method for emotional image processing using sensors in mobile devices. We developed an app which generated pencil drawing filter effect for a photo image. We experimented sensor based emotional user interaction and compared various results of image processing effects using two combinatorial sensors. Finally, this paper proved efficiency through the experimental results and evaluated the effectiveness of the proposed method.

GPFlow: an intuitive environment for web‐based scientific workflow

AbstractIncreasingly scientists are using collections of software tools in their research. These tools are typically used in concert, often necessitating laborious and error‐prone manual data reformatting and transfer. We present an intuitive workflow environment to support scientists with their research. The workflow, GPFlow, wraps legacy tools, presenting a high level, interactive web‐based front end to scientists. The workflow backend is realized by a commercial grade workflow engine (Windows Workflow Foundation). The workflow model is inspired by spreadsheets and is novel in its support for an intuitive method of interaction enabling experimentation as required by many scientists, e.g. bioinformaticians. We apply GPFlow to two bioinformatics experiments and demonstrate its flexibility and simplicity. Copyright © 2007 John Wiley & Sons, Ltd.

Perceptive three dimensional interface via stereo observation

This paper describes an intuitive approach for interacting with a computer or computer-driven applications. Interaction is achieved by observing, through a stereo camera set-up, the motion of a user’s hands. This motion is then translated into 3-dimensional (3-D) coordinates to enable interaction with either a traditional 2-dimensional (2-D) desktop or a novel 3-D user interface. The aim of this work is to provide an intuitive method of interaction to computer based applications for individuals whose condition might restrict their ability to use a standard keyboard/mouse.

Interfacing with the Real Space and its Performance

This paper presents an immersive gesture-recognition-based system to visualize the indoor thermal environment using Computational Fluid Dynamics (CFD). To enable efficient visualization of CFD in actual space, an Augmented Reality system was integrated with a CFD simulation engine. To facilitate efficient data manipulation of the simulated post-processed CFD data and to increase user control of the immersive environment, an intuitive method of Human-Computer Interaction (HCI) has been incorporated using gesture and speech recognition. While gesture recognition aids in transforming hand postures into command functions through forward kinematics and computation of hand segment positions and their joint angles, speech recognition allows better control of the data manipulation. This enabled real-time interactions between the users and simulated CFD results in actual space.

Visual gesture interfaces for virtual environments

Virtual environments provide a whole new way of viewing and manipulating 3D data. Current technology moves the images out of desktop monitors and into the space immediately surrounding the user. Users can literally put their hands on the virtual objects. Unfortunately, techniques for interacting with such environments are yet to mature. Gloves and sensor-based trackers are unwieldy, constraining and uncomfortable to use. A natural, more intuitive method of interaction would be to allow the user to grasp objects with their hands and manipulate them as if they were real objects. We are investigating the use of computer vision in implementing a natural interface based on hand gestures. A framework for a gesture recognition system is introduced along with results of experiments in colour segmentation, feature extraction and template matching for finger and hand tracking, and simple hand pose recognition. Implementation of a gesture interface for navigation and object manipulation in virtual environments is presented.