Natural Hand Gestures Research Articles

Gesture and Voice Based PC Control Using Real Time Camera

Human Computer Interaction with hand gesture has become increasingly popular today. The physical mouse device whether it is our basic desktop mouse or laptop touch pad, it require physical contact of user to convey input. Many advanced technologies have came but they all require physical contact. Hence we need a ubiquitous interface to do this. Our method is to use a camera device and computer gesture and voice based technology, like image segmentation as well as gesture recognition, to control mouse actions such as left click, double click, right click and dragging. We can show how it can do the things that present mouse devices can do. Our project Gesture and Voice Based PC Control is a real time system capable of understanding mouse commands given by hand gestures and voice commands. The end user is able to communicate with computer with the commands given by hand gestures and voice commands. This will avoid the need of physical contact to the computer to control mouse inputs. In this way the interface becomes ubiquitous. Introduction As computer technology continues to improve, people are having smaller and smaller electronic. Devices. They want to use these devices ubiquitously. There is need of a new interfaces designed specifically for use with these electronics devices. We are increasingly recognizing the importance of HCI, and in particularly gesture based gesture recognition. Simple interfaces already exist, such as embedded keyboard, folder keyboard and mini keyboard [2] . However, these interfaces need some amount of space to use as well as can’t be used in motion. Touch screens devices are also providing a good control interface and nowadays it is used globally in many applications [3] . But the cost and other hardware limitations makes it limited. With application of gesture technology and controlling the mouse actions by natural hand gestures, we can reduce the space required. We propose a good approach that uses a camera to control the mouse actions. Problem Definition To develop a software solution to a problem, the first step is to understand the problem. Mouse is a physical device, subject to mechanical wear and tear also demands the user to make physical contact to convey his Input, which is no longer appropriate. Natural actions in human to human communication, such as speak and gesture, seem more appropriate. Interfaces based on computational perception and computer gesture should be available for accomplishing the goals of ubiquitous computing [5] . The problem here is to develop a way so that humans can interact with a computer without having any physical contact with the computer. Need for the New System Mouse is a physical device. This device is subject to mechanical wear and tear. It requires the user to make physical contact to convey his Input, which is not appropriate. Instead, natural actions in human-to-human communication, such as speak and gesture, seem more appropriate [5] . This

가상현실 환경에서 3D 가상객체 조작을 위한 인터페이스와 인터랙션 비교 연구

Received 22 September 2015; received in revised form 3 December 2015; accepted 16 December 2015ABSTRACT Recently immersive virtual reality (VR) becomes popular due to the advanced development of I/O interfaces and related SWs for effectively constructing VR environments. In particular, natu-ral and intuitive manipulation of 3D virtual objects is still considered as one of the most import-ant user interaction issues. This paper presents a comparative study on the manipulation andinteraction of 3D virtual objects using different interfaces and interactions in three VR environ-ments. The comparative study includes both quantitative and qualitative aspects. Three differentexperimental setups are 1) typical desktop-based VR using mouse and keyboard, 2) hand ges-ture-supported desktop VR using a Leap Motion sensor, and 3) immersive VR by wearing anHMD with hand gesture interaction using a Leap Motion sensor. In the desktop VR with handgestures, the Leap Motion sensor is put on the desk. On the other hand, in the immersive VR,the sensor is mounted on the HMD so that the user can manipulate virtual objects in the frontof the HMD. For the quantitative analysis, a task completion time and success rate were mea-sured. Experimental tasks require complex 3D transformation such as simultaneous 3D transla-tion and 3D rotation. For the qualitative analysis, various factors relating to user experiencesuch as ease of use, natural interaction, and stressfulness were evaluated. The qualitative andquantitative analyses show that the immersive VR with the natural hand gesture provides moreintuitive and natural interactions, supports fast and effective performance on task completion,but causes stressful condition. Key Words: Virtual reality, Interface and interaction, User experience, Hand gesture, Leap Motion

주거 공간에서의 3차원 핸드 제스처 인터페이스에 대한 사용자 요구사항

The aim of this study is to find out the user’s needs of a 3D hand gesture interface in the smart home environment. To find out the users’ needs, we investigated which object the users want to use with a 3D hand gesture interface and why they want to use a 3D hand gesture interface. 3D hand gesture interfaces are studied to be applied to various devices in the smart environment. 3D hand gesture interfaces enable the users to control the smart environment with natural and intuitive hand gestures. With these advantages, finding out the user’s needs of a 3D hand gesture interface would improve the user experience of a product. This study was conducted using a diary method to find out the user’s needs with 20 participants. They wrote the needs of a 3D hand gesture interface during one week filling in the forms of a diary. The form of the diary is comprised of who, when, where, what and how to use a 3D hand gesture interface with each consisting of a usefulness score. A total of 322 data (209 normal data and 113 error data) were collected from users. There were some common objects which the users wanted to control with a 3D hand gesture interface and reasons why they want to use a 3D hand gesture interface. Among them, the users wanted to use a 3D hand gesture interface mostly to control the light, and to use a 3D hand gesture interface mostly to overcome hand restrictions. The results of this study would help develop effective and efficient studies of a 3D hand gesture interface giving valuable insights for the researchers and designers. In addition, this could be used for creating guidelines for 3D hand gesture interfaces. †

THE SIXTH SENSE TECHNOLOGY

“We have five senses in which we glory and which we recognise and celebrate, senses that constitute the sensible world for us. But there are other senses -- secret senses, sixth senses, if you will -- equally vital, but unrecognised, and unlauded. These senses, unconscious, automatic, had to be discovered.” ― Oliver Sacks. This paper discusses a technology which is amazing everyone worldwide in today‟s time, which is “Sixth Sense Technology”. All human beings are aware of the five basic senses i.e. Sight, Hearing, Taste, Touch, Smell. Sixth Sense technology was developed by a man named Pranav Mistry, a Ph D student. „Sixth Sense Technology‟ is used to build bridge between the digital world and the physical world, bringing impalpable, digital information out into the palpable world, and allows us to interact with this information using natural hand gestures. The paper showcases the possible applications and opportunities of Sixth Sense. Furthermore, the implementation approaches are compared and the pros and cons of each approach are explored. Most importantly, technical challenges and open issues regarding each implementation approach are brought out. Furthermore, predictions are made on which approach is expected to succeed in the coming years. Finally, solutions are discussed to improve the Sixth Sense Technology with regards to its implementation approaches to insure new and better ways of human-computer interaction.

Sixth Sense Technology

Sixth Sense technology is an interface of wearable gestures which upgrade the physical world and the digital world, bringing intangible, digital datas out into the tangible world, and make us to interact with this data through natural hand gestures. The advent of this technology over the last few decade has established a new dimension in the field of Human computer Interaction (HCI) Using gesture recognition, augmented reality, computer vision and radio frequency identification. Sixth Sense technology is implemented in WUW (wear your world)

A Hybrid Grammar-Based Approach for Learning and Recognizing Natural Hand Gestures

In this paper, we present a hybrid grammar formalism designed to learn structured models of natural iconic gesture performances that allow for compressed representation and robust recognition. We analyze a dataset of iconic gestures and show how the proposed Feature-based Stochastic Context-Free Grammar (FSCFG) can generalize over both structural and feature-based variations among different gesture performances.

Hand gesture guided robot-assisted surgery based on a direct augmented reality interface

Radiofrequency (RF) ablation is a good alternative to hepatic resection for treatment of liver tumors. However, accurate needle insertion requires precise hand-eye coordination and is also affected by the difficulty of RF needle navigation. This paper proposes a cooperative surgical robot system, guided by hand gestures and supported by an augmented reality (AR)-based surgical field, for robot-assisted percutaneous treatment. It establishes a robot-assisted natural AR guidance mechanism that incorporates the advantages of the following three aspects: AR visual guidance information, surgeon's experiences and accuracy of robotic surgery. A projector-based AR environment is directly overlaid on a patient to display preoperative and intraoperative information, while a mobile surgical robot system implements specified RF needle insertion plans. Natural hand gestures are used as an intuitive and robust method to interact with both the AR system and surgical robot. The proposed system was evaluated on a mannequin model. Experimental results demonstrated that hand gesture guidance was able to effectively guide the surgical robot, and the robot-assisted implementation was found to improve the accuracy of needle insertion. This human–robot cooperative mechanism is a promising approach for precise transcutaneous ablation therapy.

Intelligent Sign Language Recognition Using Image Processing

Computer recognition of sign language is an important research problem for enabling communication with hearing impaired people. This project introduces an efficient and fast algorithm for identification of the number of fingers opened in a gesture representing an alphabet of the Binary Sign Language. The system does not require the hand to be perfectly aligned to the camera. The project uses image processing system to identify, especially English alphabetic sign language used by the deaf people to communicate. The basic objective of this project is to develop a computer based intelligent system that will enable dumb people significantly to communicate with all other people using their natural hand gestures. The idea consisted of designing and building up an intelligent system using image processing, machine learning and artificial intelligence concepts to take visual inputs of sign language's hand gestures and generate easily recognizable form of outputs. Hence the objective of this project is to develop an intelligent system which can act as a translator between the sign language and the spoken language dynamically and can make the communication between people with hearing impairment and normal people both effective and efficient. The system is we are implementing for Binary sign language but it can detect any sign language with prior image processing

Hand Motion-Based Remote Control Interface with Vibrotactile Feedback for Home Robots

This paper presents the design and implementation of a hand-held interface system for the locomotion control of home robots. A handheld controller is proposed to implement hand motion recognition and hand motion-based robot control. The handheld controller can provide a ‘connect-and-play’ service for the users to control the home robot with visual and vibrotactile feedback. Six natural hand gestures are defined for navigating the home robots. A three-axis accelerometer is used to detect the hand motions of the user. The recorded acceleration data are analysed and classified to corresponding control commands according to their characteristic curves. A vibration motor is used to provide vibrotactile feedback to the user when an improper operation is performed. The performances of the proposed hand motion-based interface and the traditional keyboard and mouse interface have been compared in robot navigation experiments. The experimental results of home robot navigation show that the success rate of the handheld controller is 13.33% higher than the PC based controller. The precision of the handheld controller is 15.4% more than that of the PC and the execution time is 24.7% less than the PC based controller. This means that the proposed hand motion-based interface is more efficient and flexible.

Virtual Class room using six sense Technology

Sixth Sense is a revolutionary technology that can be used to access digital information in the world with wearable gestures. This paper focuses on various applications of six sense technology in educational purpose and how a virtual classroom, can be conducted anywhere, anytime with the help of six sense technology. Sometimes, It may happen professor or students has the limitations of not carrying the laptop. Therefore, with the help of six sense technologies, It will allow students, professors, researchers to interact with our world like never before. Six sense technology introduces the concept of online multimedia experience as various journals, articles can be accessed anywhere without having computer and without being connected to Internet .It can also recognize articles and newspaper, retrieve the latest related stories or video from the internet and then display them on pages for user. User may not necessary use dictionary or encyclopedia. We can get information on anything we want from anywhere within a few moments. Human figures can be made as keypad where students can perform all the calculations. Researchers' can capture the images without having camera .Human fingers works like the keyboard as well as the mouse. All the typed documents can be converted into digital notes It is a hi-tech device that allows users to be part of the physical world and still access the digital information existing on the World Wide Web by using natural hand gestures .Using Sixth Sense technology in class rooms will creates a more stimulating learning environment .The Sixth Sense prototype is comprised of a pocket projector, a mirror, colored marker and a camera. These components are coupled in a pendant like mobile wearable device. The camera, mirror and projector is connected wirelessly to a blue tooth smart phone device that can easily fit into the user's pocket.. The key here is that sixth sense recognizes the objects around you and displaying the information automatically and letting you access it in any way you want.

Immersive manipulation of virtual objects through glove-based hand gesture interaction

Immersive visualisation is increasingly being used for comprehensive and rapid analysis of objects in 3D and object dynamic behaviour in 4D. Challenges are therefore presented to provide natural user interaction to enable effortless virtual object manipulation. Presented in this paper is the development and evaluation of an immersive human–computer interaction system based on stereoscopic viewing and natural hand gestures. For the development, it is based on the integration of a back-projection stereoscopic system for object and hand display, a hybrid inertial and ultrasonic tracking system to provide the absolute positions and orientations of the user’s head and hands, as well as a pair of high degrees-of-freedom data gloves to provide the relative positions and orientations of digit joints and tips on both hands. For the evaluation, it is based on a two-object scene with a virtual cube and a CT (computed tomography) volume created for demonstration of real-time immersive object manipulation. The system is shown to provide a correct user view of objects and hands in 3D with depth, as well as to enable a user to use a number of simple hand gestures to perform basic object manipulation tasks involving selection, release, translation, rotation and scaling. Also included in the evaluation are some quantitative tests of the system performance in terms of speed and latency.

Classification of low level surface electromyogram using independent component analysis

There is an urgent need for a simple yet robust system to identify natural hand actions and gestures for controlling prostheses and other computer-assisted devices. Surface electromyogram (SEMG) is a non-invasive measure of the muscle activities but is not reliable because there are a multiple simultaneously active muscles. This study proposes the use of independent component analysis (ICA) for SEMG to separate activity from different muscles. A mitigation strategy to overcome shortcomings related to order and magnitude ambiguity related to ICA has been developed. This is achieved by using a combination of unmixing matrix obtained from FastICA analysis and weight matrix derived from training of the supervised neural network corresponding to the specific user. This is referred to as ICANN (independent component analysis neural network combination). Experiments were conducted and the results demonstrate a marked improvement in the accuracy. The other advantages of this system are that it is suitable for real time operations and it is easy to train by a lay user.

Sensing Glove for Brain Studies: Design and Assessment of Its Compatibility for fMRI With a Robust Test

In this paper, we describe a biomimetic-fabric-based sensing glove that can be used to monitor hand posture and gesture. Our device is made of a distributed sensor network of piezoresistive conductive elastomers integrated into an elastic fabric. This solution does not affect natural movement and hand gestures, and can be worn for a long time with no discomfort. The glove could be fruitfully employed in behavioral and functional studies with functional MRI (fMRI) during specific tactile or motor tasks. To assess MR compatibility of the system, a statistical test on phantoms is introduced. This test can also be used for testing the compatibility of mechatronic devices designed to produce different stimuli inside the MR environment. We propose a statistical test to evaluate changes in SNR and time-domain standard deviations between image sequences acquired under different experimental conditions. fMRI experiments on subjects wearing the glove are reported. The reproducibility of fMRI results obtained with and without the glove was estimated. A good similarity between the activated regions was found in the two conditions.

Understanding Hand Gestures Using Approximate Graph Matching

We live in a society that depends on high-tech devices for assistance with everyday tasks, including everything from transportation to health care, communication, and entertainment. Tedious tactile input interfaces to these devices result in inefficient use of our time. Appropriate use of natural hand gestures will result in more efficient communication if the underlying meaning is understood. Overcoming natural hand gesture understanding challenges is vital to meet the needs of these increasingly pervasive devices in our every day lives. This work presents a graph-based approach to understand the meaning of hand gestures by associating dynamic hand gestures with known concepts and relevant knowledge. Conceptual-level processing is emphasized to robustly handle noise and ambiguity introduced during generation, data acquisition, and low-level recognition. A simple recognition stage is used to help relax scalability limitations of conventional stochastic language models. Experimental results show that this graph-based approach to hand gesture understanding is able to successfully understand the meaning of ambiguous sets of phrases consisting of three to five hand gestures. The presented approximate graph-matching technique to understand human hand gestures supports practical and efficient communication of complex intent to the increasingly pervasive high-tech devices in our society.

Differential video coding of face and gesture events in presentation videos

Currently, bandwidth limitations pose a major challenge for delivering high-quality multimedia information over the Internet to users. In this research, we aim to provide a better compression of presentation videos (e.g., lectures). The approach is based on the idea that people tend to pay more attention to the face and gesturing hands, and therefore these regions are given more resolution than the remaining image. Our method first detects and tracks the face and hand regions using color-based segmentation and Kalman filtering. Next, different classes of natural hand gesture are recognized from the hand trajectories by identifying gesture holds, position/velocity changes, and repetitive movements. The detected face/hand regions and gesture events in the video are then encoded at higher resolution than the remaining lower-resolution background. We present results of the tracking and gesture recognition approach, and evaluate and compare videos compressed with the proposed method to uniform compression.

Interactive Computer Animation of Hand Gestures using Status Estimation with Multiple Regression Analysis

This paper presents a method of interactively generating natural hand gesture animation using reduced dimensionality from multiple captured data sequences of finger motions conducting specific tasks. This method is achieved by introducing an estimation with multiple regression analysis. Even when the skeletal structure of the user who inputs the motion is different from that of the shape model in the computer, the motion that a user imagines is generated. Experimental results obtained from the interface applied to virtual object manipulation showed that the proposed method generates animation naturally, just as users would expect. This method enables us to make input devices that require minimal user training and computer calibration, and helps to make the user interface intuitive and easy to use.

Evaluation of virtual reality interface for product shape designs

It is generally acknowledged that existing computer-aided design systems have inefficient user interfaces. Especially during the concept shape design stage, these systems prove to be cumbersome because of two reasons: (i) they require the usage of two-dimensional (2D) input devices, while the designs are typically three-dimensional (3D); and, (ii) CAD systems require the specification of dimensions, which may not be precisely known at the concept stage. To overcome these limitations, this research proposes the use of virtual reality (VR) devices to provide a physically intuitive interface for concept shape creation. The intuitiveness of the interface arises from the use of natural hand gestures and voice commands that emulate the way in which designers discuss concept shapes. In this scenario, the interface between the human and computer plays a central role with respect to usability, usefulness and accuracy. The focus of this research is on using two modalities: (i) hand input; and, (ii) voice driven commands, or a combination of these modalities to accomplish typical CAD tasks. Based on experience with a conventional CAD system, a set of typical CAD tasks are identified. A series of tests are then performed to determine the relative efficiency of the different modality combinations to achieve each task. The interface test results indicate that while voice commands are intuitive in initiating operations such as viewpoint zooming in/out and object creation/deletion, hand inputs are effective in performing spatial tasks such as interactive dimensioning and re-location of shapes. It was also found that a combination of voice and hand input can be used for accomplishing certain tasks more effectively such as, zooming in/out a particular direction (hand orientation indicates direction and voice is used for indicating zoom in/out operation). Based on the experience with the prototypical system developed it is concluded that voice and hand input are effective ways of building three-dimensional shapes in a virtual reality environment. To verify the efficiency of the VR-CAD interface, sample injection molded parts are built on the current VR-based CAD system and a traditional CAD system, and the times taken to build these parts are compared. The test results indicate that building geometry shapes containing canonical forms, such as block, cylinder, sphere, ⃛ etc, using a VR interface results in a speedup of five to ten times over traditional CAD systems.

Gesture Recognition Based upon Statistical Similarity

One of the improvements virtual reality offers traditional human-computer interfaces is that it enables the user to interact with virtual objects using gestures. The use of natural hand gestures for computer input provides opportunities for direct manipulation in computing environments, but not without some challenges. The mapping of a human gesture onto a particular system function is not nearly so easy as mapping with a keyboard or mouse. Reasons for this difficulty include individual variations in the exact gesture movement, the problem of knowing when a gesture starts and ends, and variation in the relative positions of other body parts that might help to identify a gesture but are not measured. A further difficulty stems from limitations on the number of gestures that a person can reliably remember and reproduce. This paper describes work on the statistical recognition of gestures based on the sum of squares. A DataGlove™ was employed to measure finger position and “train” software to recognize the letters and numbers of the American Sign Language (ASL) manual alphabet. This technique for gesture recognition is more effective than methods commonly employed in VR applications in that it can distinguish dozens of gestures and is not bound by the input sequences of a particular user. The work described here is limited in that it examines only gestures that do not occur across time. Applications for speakers of ASL and for VR are discussed, and future directions for gesture recognition research are introduced. These include adding a motion tracker and potential for recognizing gestures that do occur across time.