Hand-Mouse Interface Using Virtual Monitor Concept for Natural Interaction

Abstract

The growing interest in human–computer interaction has prompted research in this area. In addition, research has been conducted on a natural user interface/natural user experience (NUI/NUX), which utilizes a user’s gestures and voice. In the case of NUI/NUX, a recognition algorithm is needed for the gestures or voice. However, such recognition algorithms have weaknesses because their implementation is complex, and they require a large amount of time for training. Therefore, steps that include pre-processing, normalization, and feature extraction are needed. In this paper, we designed and implemented a hand-mouse interface that introduces a new concept called a “virtual monitor”, to extract a user’s physical features through Kinect in real time. This virtual monitor allows a virtual space to be controlled by the hand mouse. It is possible to map the coordinates on the virtual monitor to the coordinates on the real monitor accurately. A hand-mouse interface based on the virtual monitor concept maintains the outstanding intuitiveness that is the strength of the previous study and enhances the accuracy of mouse functions. In order to evaluate the intuitiveness and accuracy of the interface, we conducted an experiment with 50 volunteers ranging from teenagers to those in their 50s. The results of this intuitiveness experiment showed that 84% of the subjects learned how to use the mouse within 1 min. In addition, the accuracy experiment showed the high accuracy level of the mouse functions [drag (80.9%), click (80%), double-click (76.7%)]. This is a good example of an interface for controlling a system by hand in the future.