User's Hand Motion Research Articles

2A2-W08 圧覚提示と電気触覚刺激の組み合わせによる装着型触覚ディスプレイの試作

We propose a wearable tactile display using transcutaneous electric nerve stimulation and tactile force. This tactile display is suitable for virtual reality applications, for users can wear this device and it doesn't disturb user's hand movement. This device has a high space resolution and users can differentiate edge and flat surface of virtual objects.

Accelerometer Based Joint Step Detection and Adaptive Step Length Estimation Algorithm Using Handheld Devices

The pedestrian inertial navigation systems are generally based on Pedestrian Dead Reckoning (PDR) algorithm. Considering the physiological characteristics of pedestrian movement, we use the cyclical characteristics and statistics of acceleration waveform and features which are associated with the walking speed to estimate the stride length. Due to the randomness of the pedestrian hand-held habit, the step events cannot always be detected by using the periods of zero velocity updates (ZUPTs). Furthermore, the signal patterns of the sensor could differ significantly depending on the carrying modes and the user's hand motion. Hence, the step detection and the associated adaptive step length model using a handheld device equipped with accelerometer is required to obtain high-accurate measurements. To achieve this goal, a compositional algorithm of empirical formula and Back-propagation neural network by using handheld devices is proposed to estimate the step length, as well as achieve the accuracy of step detection higher than 98%. Furthermore, the proposed joint step detection and adaptive step length estimation algorithm can help much in the development of Pedestrian Navigation Devices (PNDs) based on the handheld inertial sensors. Index Terms—Pedestrian inertial navigation; handheld devices; step detection; step length estimation; accelerometer

Sensing and Force-Feedback Exoskeleton (SAFE) Robotic Glove.

This paper presents the design, implementation and experimental validation of a novel robotic haptic exoskeleton device to measure the user's hand motion and assist hand motion while remaining portable and lightweight. The device consists of a five-finger mechanism actuated with miniature DC motors through antagonistically routed cables at each finger, which act as both active and passive force actuators. The SAFE Glove is a wireless and self-contained mechatronic system that mounts over the dorsum of a bare hand and provides haptic force feedback to each finger. The glove is adaptable to a wide variety of finger sizes without constraining the range of motion. This makes it possible to accurately and comfortably track the complex motion of the finger and thumb joints associated with common movements of hand functions, including grip and release patterns. The glove can be wirelessly linked to a computer for displaying and recording the hand status through 3D Graphical User Interface (GUI) in real-time. The experimental results demonstrate that the SAFE Glove is capable of reliably modeling hand kinematics, measuring finger motion and assisting hand grasping motion. Simulation and experimental results show the potential of the proposed system in rehabilitation therapy and virtual reality applications.

A penetrable interactive 3D display based on motion recognition (Invited Paper)

Based on light field reconstruction and motion recognition technique, a penetrable interactive floating 3D display system is proposed. The system consists of a high-frame-rate projector, a flat directional diffusing screen, a high-speed data transmission module, and a Kinect somatosensory device. The floating occlusion-correct 3D image could rotate around some axis at different speeds according to user's hand motion. Eight motion directions and speed are detected accurately, and the prototype system operates efficiently with a recognition accuracy of 90% on average.

Airtouch 기술을 활용한 스마트융합 DID 시스템 설계

본 연구는 터치스크린 방식의 정보전달 기기인 DID 시스템에 Airtouch 기술을 융합하여 새로운 방식의 정보전달 시스템을 개발하는 것이다. 대학의 학내 공지사항, 교육정보 및 취업정보 등을 열람하고 활용 할 수 있는 시스템을 설계하고 Airtouch 기술을 구현하여 원격조작 및 공유 콘텐츠 스토어, 클라우드 서비스 개발을 통한 콘텐츠 동기화 기술 구현으로 스마트융합 DID 시스템을 개발하고자 한다. 키넥트는 USB 인터페이스를 가지고 있기 때문에 정보기기와 연결하여 사용 할 수 있으며, 저렴한 가격의 제품으로 키넥트 센서를 활용하여 Airtouch 기술을 구현한다. 논문에서는 유형의 입력장치 없이 사용자의 손동작만으로 정보기기와 상호작용이 가능한 시스템인 Airtouch 기술을 활용한 스마트융합 DID 시스템을 개발하여 사용자의 손 움직임을 추적하여 이를 바탕으로 마우스 포인터를 조작할 수 있고, 사용자의 손동작을 통해 정보기기에 명령을 내릴 수 있도록 하였다. Airtouch 기술을 활용한 스마트융합 DID 시스템 기술 활용으로 온라인 교육산업, 광고홍보, 정보화 산업 등 여타 산업에 파급효과를 증대시킨다. 또한, 폭넓은 범용성을 가지고 기존의 인터페이스 장치들을 대체기술 활용성이 무한 확장이 가능하다. Airtouch technology to integrate the system in the way of information delivery devices, touch screen DID this study is to develop new ways of information delivery systems. Airtouch technology to design and implement a system that can be utilized to view the college campus announcements, education, information, and employment information, and store the remote operation and sharing content, the development of cloud services to sync content via smart technology implementation fusion DID systemto develop. Packs USB interface kinek because you may be used in connection with the information appliances, and low-cost product by leveraging the Kinect sensor, Airtouch technology implementation. Types of input devices paper Airtouch technology systems, the user's hand gestures alone can interact with information appliances, smart fusion system developed by DID by tracking the user's hand movements to manipulate the mouse pointer, and information through the user's hand gestures to command the unit so that you can make. Airtouch technology smart fusion DID system technology utilizing a ripple effect on other industries, such as the online education industry, advertising, information industry increases. Also, replace the existing interface device with the versatility of a wide range of technologies, usability is an infinite expansion.

Visual manipulations for motor rehabilitation

Mixed reality rehabilitation systems and games are demonstrating potential as innovative adjunctive therapies for health professionals in their treatment of various hand and upper limb motor impairments. Unilateral motor deficits of the arm, for example, are commonly experienced poststroke. Our Augmented Reflection Technology (ART) system provides an augmented reality environment that contributes to this increasingly rich area of research. We present the current set of technological capabilities of the ART system exemplified in two experimental laboratory studies as well as a prototype system built on top of ART which “fools the brain” by visually amplifying user's hand movements—small actual hand movements lead to perceived larger movements. We validate the usability of our system in an empirical study with forty-five non-clinical participants. In addition, we present the first qualitative evidence for the utility of our approach and system for stroke recovery and motor rehabilitation. Future uses of the system are considered by way of conclusion.

손동작을 이용한 상호작용 증강현실 시스템

본 논문에서는 컴퓨터 비전 기술 기반으로 사용자의 손 움직임과 가상객체 사이의 자연스러운 상호작용을 위한 증강현실 시스템을 제안한다. 기존의 증강현실 시스템은 마커를 이용하거나 트랙커 같은 센서를 직접 조작해야 했기 때문에 사용자에게 불편함을 제공한다. 우리는 이러한 문제점을 해결하기 위해서 손 움직임을 증강현실 시스템에 적용한다. 또한 가상객체의 움직임에 물리적 현상을 적용하여 현실감을 부여한다. 제안한 시스템은 마커를 이용하여 기하학적 정보를 얻어 가상의 공이 움직이는 가상공간, 벽돌 등의 시스템 환경을 구성하였으며, 사용자의 손 움직임은 테이핑을 통하여 특징점을 추출하여 정보를 얻는다. 이 특징점을 이용하여 손 위에 가상 평면을 안정적으로 정합한다. 가상의 공의 움직임은 포물선 방정식을 이용하여 기본적으로 포물선 움직임을 보이고, 평면 또는 벽돌에 충돌이 발생할 경우에는 가상 공의 위치와 평면의 법선벡터를 이용하여 가상 공의 움직임을 보였다. 실험을 통해 충돌 시 발생되는 잘못된 공의 위치에 대해 보정되는 과정을 보였고, 증강된 가상객체의 떨림과 수행속도에 대해 마커를 이용한 시스템과 비교하여 제안한 시스템으로 대체할 수 있다는 것을 확인하였다. In this paper, We propose Augmented Reality (AR) System for the interaction between user's hand motion and virtual object motion based on computer vision. The previous AR system provides inconvenience to user because the users have to control the marker and the sensor like a tracker. We solved the problem through hand motion and provide the convenience to the user. Also the motion of virtual object using a physical phenomenon gives a reality. The proposed system obtains geometrical information by the marker and hand. The system environments like virtual space of moving virtual ball and bricks are made by using the geometrical information and user's hand motion is obtained from the hand's information with extracted feature point through the taping hand. And it registers a virtual plane stably by getting movement of the feature points. The movement of the virtual ball basically is parabolic motion with a parabolic equation. When the collision occurs either the planes or the bricks, we show movement of the virtual ball with ball position and normal vector of plane and the ball position is faulted. So we showed corrected ball position through experiment. and we proved that this system can replaced the marker system to compare to jitter of augmented virtual object and progress speed with it.

Virtual Active Touch: Perception of Virtual Gratings Wavelength through Pointing-Stick Interface.

Tactile feedback enhances the usability and enjoyment of human-computer interfaces. Many feedback techniques have been devised to present tactile stimuli corresponding to a user's hand movements taking account of the concept of active touch. However, hand movements may not necessarily be required for achieving natural tactile feedback. Here, we propose a virtual-active-touch method that achieves haptic perception without actual/direct hand movements. In this method, a cursor manipulated by a force-input device is regarded as a virtual finger of the operator on the screen. Tactile feedback is provided to the operator in accordance with cursor movements. To validate the translation of virtual roughness gratings, we compare the virtual-active-touch interface with an interface that involves actual hand movements. By using the appropriate force-to-velocity gain for the pointing-stick interface, we show that the virtual-active-touch method presents the surface wavelengths of the gratings, which is a fundamental property for texture roughness, and that the gain significantly influences the textures experienced by the operators. Furthermore, we find that the perceived wavelengths of objects scaled and viewed on a small screen are skewed. We conclude that although some unique problems remain to be solved, we may be able to perceive the surface wavelengths solely with the intentions of active touch through virtual-active-touch interfaces.

Design and evaluation of freehand menu selection interfaces using tilt and pinch gestures

Freehand gestural interaction, in which the user’s hands move in mid-air to provide input, has been of interest to researchers, but freehand menu selection interfaces have been under-investigated so far. Freehand menu selection is inherently difficult, especially with increasing menu breadth (i.e., the number of items), largely because moving hands in free space cannot achieve precision as high as physical input devices such as mouse and stylus. We have designed a novel menu selection interface called the rapMenu ( Ni et al., 2008), which is controlled by wrist tilt and multiple pinch gestures, and takes advantage of the multiple discrete gesture inputs to reduce the required precision of the user hand movements. In this article, we first review the visual design and behavior of the rapMenu technique, as well as related design issues and its potential advantages. In the second part, we present two studies of the rapMenu in order to further investigate the strengths and limitations of the design principle. In the first study, we compared the rapMenu to the extensively studied tilt menu technique ( Rahman et al., 2009). Our results revealed that the rapMenu outperforms the tilt menu as menu breadth increases. In the second study, we investigated how the rapMenu affords the opportunity of eyes-free selection and users’ transition from novice to expert. We found that within 10 min of practice, eyes-free selection with rapMenu has competitive speed and accuracy with the visual rapMenu and the tilt menu. Finally, we discuss design variations that use other axes of wrist movement and adopt alternative auditory feedback.

Finger-Mounted Tactile Sensor for Evaluating Surfaces

This paper presents a finger-mounted tactile sensor for extracting information on fine surface properties of objects such as textile fabrics. The prototype sensor has a thin structure composed of a sheet of PVDF (Polyvinylidene Fluoride) film sensor and some metal parts for converting compressive forces into area expansion of the PVDF film. By using a signal processing program based on FFT (fast Fourier transform), voltage signal sequences from nine different fabrics were distinguished, in spite of the variation in the pressing force and the speed of rubbing motion induced by the fluctuation of the user's hand motion. In addition, the signal sequences from abraded fabrics were sorted in the order of the level of abrasion, by extracting a signal component correlated with the abrasion level.

Design of Dynamic Vibrotactile Textures.

This paper describes a method for creating virtual textures without force feedback by using a simple motion sensor and a single vibrotactile actuator. It is based on wavetable synthesis driven by the user's hand movements. The output of the synthesis is rendered with the tactile actuator attached in a hand-held box together with the motion sensor. The method provides a solution for creating tangible properties for virtual objects which can be explored by pointing at them with the sensor-actuator device. The study introduces 12 virtual textures which were based on three different envelope ridge lengths, two spatial densities, and were either regularly or irregularly organized. To evaluate the role of each design parameter in the perception of the texture, a series of experiments was conducted. The perceived similarity was assessed in a pairwise comparison test and the outcome was analyzed by using multidimensional scaling. The analysis revealed that envelope ridge length and spatial density were distinguishable design parameters while regularity was not. The textures were also rated according to five attribute scales previously determined in the pilot experiment. The results show that ridge length and spatial density influence perceived roughness and flatness similarly as with real textures.

Actual remote control: a universal remote control using hand motions on a virtual menu

Consumer electronic devices are becoming more complex and diverse. In order to provide users with an easy and convenient way of controlling various consumer electronics, it is necessary to use a universal way to control various devices. This requires using a user-friendly and intuitive interface. We introduce a wristwatch-type of remote that offers a unified way to control various devices. Total of seven gestures that are based on hand motions are defined. These seven gestures are suitable to the wearable wristwatch-type device. For a variety of systems, these gestures can be used in the same way. Gestures are not designed and analyzed in a discrete manner; instead, they are designed in continuous hand motions and analyzed in gesture commands. Also, a method of modeling a virtual menu from the menu on a consumer electronic device is introduced. With the virtual menu, the user is able to control various devices by using gestures. The virtual menu has to reflect hand motion characteristics and represents functions of electronic appliances. The virtual menu is implemented in XML to represent the basic menu construction and its properties. In order to use virtual menu, we introduce how a user's hand motions can be used in a fast and effective way. Finally we compare our wearable remote control to the conventional remote control through user tests in terms of user convenience and efficiency.

Pen-Based Interface Using Hand Motions in the Air

A system which employs a stylus as an input device is suitable for creative activities like writing and painting. However, such a system does not always provide the user with a GUI that is easy to operate using the stylus. In addition, system usability is diminished because the stylus is not always integrated into the system in a way that takes into consideration the features of a pen. The purpose of our research is to improve the usability of a system which uses a stylus as an input device. We propose shortcut actions, which are interaction techniques for operation with a stylus that are controlled through a user's hand motions made in the air. We developed the Context Sensitive Stylus as a device to implement the shortcut actions. The Context Sensitive Stylus consists of an accelerometer and a conventional stylus. We also developed application programs to which we applied the shortcut actions; e.g., a drawing tool, a scroll supporting tool, and so on. Results from our evaluation of the shortcut actions indicate that users can concentrate better on their work when using the shortcut actions than when using conventional menu operations.

HEMP—hand-displacement-based pseudo-haptics: A study of a force field application and a behavioural analysis

This paper introduces a novel pseudo-haptic approach called HEMP— Hand-displac EMent-based Pseudo-haptics. The main idea behind HEMP is to provide haptic-like sensations by dynamically displacing the visual representation of the user's hand. This paper studies the possible application of HEMP to the simulation of force fields (FFs). The proposed hardware solution for simulating the hand displacement is based on an augmented reality configuration, the video see-through head-mounted display. A response model is proposed for controlling the hand displacement. This model adapts to the user's hand movements. It also accounts for a number of perceptual and system constraints. An experiment has been carried out to investigate the potential of the proposed technique. Subjects had to perform an FF strength comparison task and to fill in an illusion evaluation questionnaire. Comparison response results show that different FF strength levels are discriminable and the questionnaire indicates that subjects perceive flow pressure-like sensations. The analysis of arm muscular activity seems to confirm these results.

Perception-based lossy haptic compression considerations for velocity-based interactions

The ability of technology to transmit multi-media is very dependent on compression techniques. In particular lossy compression has been used in image compression (jpeg) audio compression (mp3) and video compression (mpg) to allow the transmission of audio and video over broadband network connections. Recently the sense of touch or haptics is becoming more important with its addition in computer games or in cruder applications such as vibrations in a cell phone. As haptic technology improves the ability to transmit compressed force sensations becomes more critical. Most lossy audio and visual compression techniques rely on the lack of sensitivity in humans to pick up detailed information in certain scenarios. Similarly limitations in the sensitivity of human touch could be exploited to create haptic models with much less detail and thus requiring smaller bandwidth. The focus of this paper is on the force thresholds of the human haptic system that can be used in a psychophysically motivated lossy haptic (force) compression technique. Most of the research in this field has measured the just noticeable difference (JND) of the human haptic system with a human user in static interaction with a stationary rigid object. In this paper our focus involves cases where the human user or the object are in relative motion. An example of such an application would be the haptic rendering of the user's hand in contact with of a high-viscous material or interacting with a highly deformable object. Thus an approach is presented to measure the force threshold based on the velocity of the user's hand motion. Two experiments are conducted to detect the absolute force threshold (AFT) of the human haptic system using methodologies from the field of psychophysics. The AFTs are detected for three different ranges of velocity of the user's hand motion. This study implies that when a user's hand is in motion fewer haptic details are required to be stored calculated or transmitted. Finally the implications of this study on a more complete future study will be discussed.

A letter input system based on handwriting gestures

This paper considers an input method for the wearable computer environment using a wearable video camera. An input system is proposed in which the user performs the motion of writing an alphanumeric character in the air, for input to the computer. In the proposed method, the user's hand motion is imaged by a wearable video camera, and letter input is performed by analysis of the monochrome gray-level picture in the computer. When a letter is written in the air, it is difficult to identify the start and end of the letter input by the user, or the start and end points of the segments composing the letter. Furthermore, in a wearable computer environment it is desirable that the hand motion be detected in the background, which is continually changing both in the daytime and at night. Consequently, this paper proposes the following three procedures: (1) extraction of the user's hand region in the picture frame by visible ray or infrared illumination; (2) determination of the center of gravity of the user's hand motion in the air, using the intensity difference between picture frames as a clue; (3) continuous DP matching to identify the letter. In order to solve the essential problem of letter recognition in the air, a writing format for alphanumeric characters is proposed. The proposed system is implemented on a video camera and a laptop PC, and an experiment on writing of letters in the air is performed with five subjects and 360 alphanumeric characters. A recognition rate of approximately 75% is obtained. © 2006 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 89(5): 53–64, 2006; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20239

Two-dimensional position detection system with MEMS accelerometers, readout circuitry, and microprocessor for padless mouse applications

A hybrid two-dimensional position sensing system is designed with microelectromechanical systems (MEMS) for padless mouse applications. The X/Y-axis acceleration of the user's hand movements is measured by two MEMS accelerometer devices. These acceleration values are pulsewidth modulated and converted into (X, Y) coordinates on the screen by integral operations on a microprocessor. The overall system consists of four major components: 1) MEMS accelerometers; 2) CMOS analog readout circuitry; 3) an acceleration magnitude extraction module; and 4) a 16-b RISC microprocessor. Mechanical and analog simulation shows that the designed mouse system can detect acceleration as small as 5.3 mg (g=9.8 m/s/sup 2/) with 100-kHz sampling frequency for low power consumption.

Using Kinematics to Assess Software Usability and Ergonomic Risk Factors

Over fifty percent of the workforce utilize the computer and suffer health problems that include musculoskeletal disorders and physical discomfort (Newburger, 1999 and Work Loss Data Institute, 2001). Professionals from disciplines of human factors and ergonomics address computer issues independently. Usability and user interface issues are identified in the software development stage focusing on the users ability to perceive and understand the designed functions. Ergonomic professionals are involved in usability after implementation and upon complaints and user injury. The focus of ergonomics is to evaluate extreme postures, resulting from design that cause stress to the body. By using motion analysis, usability studies are enriched by identifying extreme movements in completing system tasks. In this paper we will present a novel method that utilizes the kinematics of the user's hand and wrist movements to simultaneously evaluate physical strain and usability.

A multimodal learning interface for grounding spoken language in sensory perceptions

We present a multimodal interface that learns words from natural interactions with users. In light of studies of human language development, the learning system is trained in an unsupervised mode in which users perform everyday tasks while providing natural language descriptions of their behaviors. The system collects acoustic signals in concert with user-centric multisensory information from nonspeech modalities, such as user's perspective video, gaze positions, head directions, and hand movements. A multimodal learning algorithm uses this data to first spot words from continuous speech and then associate action verbs and object names with their perceptually grounded meanings. The central ideas are to make use of nonspeech contextual information to facilitate word spotting, and utilize body movements as deictic references to associate temporally cooccurring data from different modalities and build hypothesized lexical items. From those items, an EM-based method is developed to select correct word--meaning pairs. Successful learning is demonstrated in the experiments of three natural tasks: "unscrewing a jar," "stapling a letter," and "pouring water."

3次元空間内の複数の仮想物体を提示するための遭遇型ハプティックデバイスの軌道計画

A path-planning algorithm for encountered-type haptic devices that render multiple virtual objects in 3D space is proposed. To render multiple virtual objects with a single haptic device, the following two behaviors must be realized: (i) the device should move to the target object location in advance of the user's hand so that the user can encounter it; (ii) otherwise, the device should keep away from the user's hand, avoiding an unexpected collision. To realize such a consistent encounter, the user's hand motion must be tracked. The proposed algorithm determines the device location based on the minimum distance problem between a point (the user's hand) and a convex polyhedron constructed from the reference points of multiple objects. An automobile virtual control panel was constructed using an encountered-type haptic device in which the proposed path-planning algorithm was implemented. The validity of the proposed algorithm was confirmed and the required performances, such as the device speed and the accuracy of hand tracking were evaluated.