Hand Gesture Interaction Research Articles

Context‐aware hand gesture interaction for human–robot collaboration in construction

AbstractConstruction robots play a pivotal role in enabling intelligent processes within the construction industry. User‐friendly interfaces that facilitate efficient human–robot collaboration are essential for promoting robot adoption. However, most of the existing interfaces do not consider contextual information in the collaborative environment. The situation where humans and robots work together in the same jobsite creates a unique environmental context. Overlooking contextual information would limit the potential to optimize interaction efficiency. This paper proposes a novel context‐aware method that utilizes a two‐stream network to enhance human–robot interaction in construction settings. In the proposed network, the first‐person view‐based stream focuses on the relevant spatiotemporal regions for context extraction, while the motion sensory data‐based stream obtains features related to hand motions. By fusing the vision context and motion data, the method achieves gesture recognition for efficient communication between construction workers and robots. Experimental evaluation on a dataset from five construction sites demonstrates an overall classification accuracy of 92.6%, underscoring the practicality and potential benefits of the proposed method.

Implementation of natural hand gestures in holograms for 3D object manipulation

Holograms provide a characteristic manner to display and convey information, and have been improved to provide better user interactions Holographic interactions are important as they improve user interactions with virtual objects. Gesture interaction is a recent research topic, as it allows users to use their bare hands to directly interact with the hologram. However, it remains unclear whether real hand gestures are well suited for hologram applications. Therefore, we discuss the development process and implementation of three-dimensional object manipulation using natural hand gestures in a hologram. We describe the design and development process for hologram applications and its integration with real hand gesture interactions as initial findings. Experimental results from Nasa TLX form are discussed. Based on the findings, we actualize the user interactions in the hologram.

A touch-free human-robot collaborative surgical navigation robotic system based on hand gesture recognition.

Robot-assisted minimally invasive surgery (RAMIS) has gained significant traction in clinical practice in recent years. However, most surgical robots rely on touch-based human-robot interaction (HRI), which increases the risk of bacterial diffusion. This risk is particularly concerning when surgeons must operate various equipment with their bare hands, necessitating repeated sterilization. Thus, achieving touch-free and precise manipulation with a surgical robot is challenging. To address this challenge, we propose a novel HRI interface based on gesture recognition, leveraging hand-keypoint regression and hand-shape reconstruction methods. By encoding the 21 keypoints from the recognized hand gesture, the robot can successfully perform the corresponding action according to predefined rules, which enables the robot to perform fine-tuning of surgical instruments without the need for physical contact with the surgeon. We evaluated the surgical applicability of the proposed system through both phantom and cadaver studies. In the phantom experiment, the average needle tip location error was 0.51 mm, and the mean angle error was 0.34 degrees. In the simulated nasopharyngeal carcinoma biopsy experiment, the needle insertion error was 0.16 mm, and the angle error was 0.10 degrees. These results indicate that the proposed system achieves clinically acceptable accuracy and can assist surgeons in performing contactless surgery with hand gesture interaction.

Online Hand Gesture Detection and Recognition for UAV Motion Planning

Recent advances in hand gesture recognition have produced more natural and intuitive methods of controlling unmanned aerial vehicles (UAVs). However, in unknown and cluttered environments, UAV motion planning requires the assistance of hand gesture interaction in complex flight tasks, which remains a significant challenge. In this paper, a novel framework based on hand gesture interaction is proposed, to support efficient and robust UAV flight. A cascading structure, which includes Gaussian Native Bayes (GNB) and Random Forest (RF), was designed, to classify hand gestures based on the Six Degrees of Freedom (6DoF) inertial measurement units (IMUs) of the data glove. The hand gestures were mapped onto UAV’s flight commands, which corresponded to the direction of the UAV flight.The experimental results, which tested the 10 evaluated hand gestures, revealed the high accuracy of online hand gesture recognition under asynchronous detection (92%), and relatively low latency for interaction (average recognition time of 7.5 ms; average total time of 3 s).The average time of the UAV’s complex flight task was about 8 s shorter than that of the synchronous hand gesture detection and recognition. The proposed framework was validated as efficient and robust, with extensive benchmark comparisons in various complex real-world environments.

American Sign Language Words Recognition of Skeletal Videos Using Processed Video Driven Multi-Stacked Deep LSTM.

Complex hand gesture interactions among dynamic sign words may lead to misclassification, which affects the recognition accuracy of the ubiquitous sign language recognition system. This paper proposes to augment the feature vector of dynamic sign words with knowledge of hand dynamics as a proxy and classify dynamic sign words using motion patterns based on the extracted feature vector. In this method, some double-hand dynamic sign words have ambiguous or similar features across a hand motion trajectory, which leads to classification errors. Thus, the similar/ambiguous hand motion trajectory is determined based on the approximation of a probability density function over a time frame. Then, the extracted features are enhanced by transformation using maximal information correlation. These enhanced features of 3D skeletal videos captured by a leap motion controller are fed as a state transition pattern to a classifier for sign word classification. To evaluate the performance of the proposed method, an experiment is performed with 10 participants on 40 double hands dynamic ASL words, which reveals 97.98% accuracy. The method is further developed on challenging ASL, SHREC, and LMDHG data sets and outperforms conventional methods by 1.47%, 1.56%, and 0.37%, respectively.

4: Interactive Volume Rendering Method Using Dynamic Ray Casting for Autostereoscopic Display

3D volume is a common type of data collected in medical and life science domains. Existing visualization systems of volume data mainly use 2D displays. However, the complexity of volumes makes it challenging to comprehend the internal structures. Volume rendering with autostereoscopic displays can provide depth perception to improve understanding of complex structures. In this paper, we present an interactive volume rendering method for autostereoscopic display. The method is implemented through the combination of dynamic ray casting and hand gesture interaction. The proposed rendering algorithm brightens the organs intersected with an interaction plane and presents different appearances controlled by the user. We further build an application fusing the foreground virtual volume and the background actual human body. The application shows the prospects of our algorithm and system in education and clinical tasks.

Gesture-Based User Interface for Vehicle On-Board System: A Questionnaire and Research Approach

Touchless interaction with electronic devices using gestures is gaining popularity and along with speech-based communication offers their users natural and intuitive control methods. Now, these interaction modes go beyond the entertainment industry and are successfully applied in real-life scenarios such as a car interior. In the paper, we analyse the potential of hand gesture interaction in the vehicle environment by physically challenged drivers. A survey conducted with potential users shows that the knowledge of gesture-based interaction and its practical use by people with disabilities is low. Based on these results we proposed a gesture-based system for vehicle on-board system. It has been developed on the available state-of-the-art solutions and investigated in terms of usability on a group of people with different physical limitations who drive a car on daily basis mostly using steering aid tools. The obtained results are compared with the performance of users without any disabilities.

A novel interactive assembly teaching aid using multi-template augmented reality

Augmented Reality (AR) technology is a fast growing field in the academics and industry. This paper presents a novel design of an interactive assembly teaching aid based on a multi-template AR system, which consists of three units: a multi-template AR unit, an online three-dimension (3D) model assembly unit, a hand-gesture interaction unit. The design of the multi-template AR unit employs an efficient multi-template pose tracking method to detect and track multiple template images simultaneously. The online 3D model assembly unit is enabled, when the pose of each target template is tracked and computed by the multi-template pose tracking method. This method measures the distance between the two templates to determine the 3D rendering mode of the virtual object. The third unit aims to realize a vision-based human-computer interactive system, which combines the AR rendering system with the real-time hand-gesture recognition method to decide the status of AR rendering of a 3D dynamic animation according to the user’s hand gesture. Based on the feedback of twenty-one users of the AR-based interactive teaching-aid, it can be concluded that the system creates an interactive experiences that engages the users and facilitates them to increase their learning interest significantly. In future, we plan to port the proposed AR system to Android and iOS mobile devices while improving the functionality, interactivity, and entertaining qualities of the teaching aid system.

Touch Interactive System Design with Intelligent Vase of Psychotherapy for Alzheimer’s Disease

This paper introduces the concept of an intelligent vase that combines the cultivation of plants with psychotherapy for the Alzheimer’s adjuvant therapy (from now on, referred to as AD) of patients. RECALL (The name of intelligent vase which defined by this paper), which includes hardware and software applications, as well as a unique hand gesture interaction method, is able to collect patients’ hand gestures and play photos, to remind patients of the plant’s growth status. To evaluate our conceptual system, we created a preliminary prototype. We conducted an experiment with leap motion. This project tries to explore a kind of interaction method by cultivating plants to simulate cognitive training activities, to stimulate memories through family pictures, to slow down the cognitive decline from AD, to incorporate non-drug intervention into the family life of AD patients, to reduce the expenditure, time, and energy costs, and to improve the quality of life of patients and their families.

Elicitation study investigating hand and foot gesture interaction for immersive maps in augmented reality

ABSTRACTImmersive maps in augmented reality (AR) are virtual maps that plausibly blend with the physical environment, such that the user perceives them as a part of the real world. While immersive maps can offer unprecedented engaging experiences, the way to perform panning, zooming and other basic map interaction is not obvious. This limitation may hamper widespread adoption of immersive maps. We therefore conducted an elicitation study to identify commonly suggested hand and foot gestures for interacting with large immersive AR maps placed on the floor. Study participants were shown simulations of interaction operations, and asked to design gestures to trigger these operations. We selected interaction with hand gestures because they are natural and familiar from interacting with touchscreens. Foot gestures were included because the users’ feet touch immersive maps placed on the floor. Eighteen participants designed hand and foot gestures for panning, rotating, zooming, changing the height of the map, creating a point marker, and selecting a point marker. The most agreed-on hand gesture was for zooming, consisting of grabbing with both hands, then separating or bringing the hands together. Our study participants suggested numerous other inspirational hand and foot gestures, which can guide the further development of interactive immersive maps. Because user preference does not necessarily align with performance, it is important to evaluate the efficiency, accuracy, ease of learning, and physical fatigue for these gestures in follow-up studies.

Design of hand gesture recognition system for human-computer interaction

Human-Computer interaction (HCI) with gesture recognition is designed to recognize a number of meaningful human expressions, and has become a valuable and intuitive computer input technique. Hand gestures are one of the most intuitive and common forms of communication, and can communicate a wide range of meaning. Vision-based hand gesture recognition has received a significant amount of research attention in recent years. However, the field still presents a number of challenges for researchers. In the vision-based hand gesture interaction process between humans and computers, gesture interpretation must be performed quickly and with high accuracy. In this paper, a low-cost HCI system with hand gesture recognition is proposed. This system uses several vision techniques. Skin and motion detection is used for capturing the region-of-interest from the background regions. A connected component labeling algorithm is proposed to identify the centroid of an object. To identify the exact area of hand gesture, the arm area is removed with the aid of a convex hull algorithm. Moreover, a real-time demonstration system is developed, based on a single-camera mechanism which allows for the use of wearable devices. Simulation results show that the recognition rate is still high, although some interference is encountered in the simulated environments.

What is the Future of Minimally Invasive Sinus Surgery: Computer-Assisted Navigation, 3D-Surgical Planner, Augmented Reality in the Operating Room with ‘in the Air’ Surgeon’s Commands as “Biomechanics” of the New Era in Personalized Contactless Hand-Gesture Non-Invasive Surgeon-Computer Interaction?

Since the very beginning of this most prestigious field of medicine, surgeons all over the world have always tended to conceive and then implement perfect, ideal conditions in the OR

A non-contact interactive stereo display system for exploring human anatomy.

Stereoscopic display based on Virtual Reality (VR) can facilitate doctors to observe the 3 D virtual anatomical models with the depth cues, assist them in intuitively investigating the spatial relationship between different anatomical structures without mental imagination. However, there is few input device can be used in controlling the virtual anatomical models in the sterile operating room. This paper presents a cost-effective VR application system for demonstration of 3 D virtual anatomical models with non-contact interaction and stereo display. The system is integrated with hand gesture interaction and voice interaction to achieve non-contact interaction. Hand gesture interaction is implemented based on a Leap Motion controller mounted on the Oculus Rift DK2. Voice is converted into operation using Bing Speech for English language and Aitalk for Chinese language, respectively. A local relationship database is designed to record the anatomical terminologies to speech recognition engine to query these uncommon words. The hierarchical nature of these terminologies is also recorded in a tree structure. In the experiments, ten participants were asked to perform the evaluation on the proposed system. The results show that our system is more efficient than traditional interactive manner and verify the feasibility and practicability in the sterile operating room.

A non-contact interactive stereo display system for exploring human anatomy

Stereoscopic display based on Virtual Reality (VR) can facilitate clinicians observing 3 D anatomical models with the depth cue which lets them understand the spatial relationship between different anatomical structures intuitively. However, there are few input devices available in the sterile field of the operating room for controlling 3 D anatomical models. This paper presents a cost-effective VR application for stereo display of 3 D anatomical models with non-contact interaction. The system is integrated with hand gesture interaction and voice interaction to achieve non-contact interaction. Hand gesture interaction is based on Leap Motion. Voice interaction is implemented based on Bing Speech for English language and Aitalk for Chinese language. A local database is designed to record the anatomical terminologies organized in a tree structure, and provided to the speech recognition engine for querying these uncommon words. Ten participants were asked to practice the proposed system and compare it with the common interactive manners. The results show that our system is more efficient than the common interactive manner and prove the feasibility and practicability of the proposed system used in the sterile field.

The implementation of a haptic feedback virtual reality simulation clinic with dynamic patient interaction and communication for medical imaging students.

IntroductionAn immersive virtual reality (VR) simulation clinic with dynamic patient interaction and communication was developed to facilitate the training of medical radiation science students. The software “CETSOL VR Clinic” was integrated into the Medical Imaging programme at Monash University in 2016 in order to benchmark student experiences against existing simulation techniques (Shaderware™).MethodsAn iterative approach to development, based on two cycles of user feedback, was used to develop and refine the simulated clinical environment. This environment uses realistic 3D models, embedded clinical scenarios, dynamic communication, 3D hand gesture interaction, gaze and positional stereoscopic tracking and online user capabilities using the Unity™ game and physics engines. Students’ perceptions of educational enhancement of their positioning skills following the use of the simulation tools were analysed via a 5‐point Likert scale questionnaire.ResultsStudent perception scores indicated a significant difference between simulation modalities in favour of the immersive CETSOL VR Clinic, χ2 (4, N = 92) = 9.5, P‐value <0.001.ConclusionStudent perception scores on improvement of their clinical and technical skills were higher for the hand‐positioning tasks performed with the CETSOL VR Clinic™ than with the comparative benchmark simulation that did not provide dynamic patient interaction and communication.

Interactive Design of 3D Dynamic Gesture Based on SVM-LSTM Model

Visual hand gesture interaction is one of the main ways of human-computer interaction, and provides users more interactive degrees of freedom and more realistic interactive experience. Authors present a hybrid model based on SVM-LSTM, and design a three-dimensional dynamic gesture interaction system. The system uses Leap Motion to capture gesture information, combined with SVM powerful static gesture classification ability and LSTM powerful variable-length time series gesture processing ability, enabling real-time recognition of user gestures. The gesture interaction method can automatically define the start and end of gestures, recognition accuracy reached 96.4%, greatly reducing the cost of learning. Experiments have shown that the gesture interaction method proposed by authors is effective. In the simulated mobile environment, the average gesture prediction only takes 0.15 seconds, and ordinary users can quickly grasp this method.

A Scan-Line Forest Growing-Based Hand Segmentation Framework With Multipriority Vertex Stereo Matching for Wearable Devices.

A hand segmentation framework is proposed for 3-D hand gesture interaction for wearable devices. In this framework, all the objects in a scene are regarded as directed trees in a forest, and the problem of the hand segmentation can be converted into finding the target tree (called hand tree) in the forest with proper hand properties including color consistency, space consistency, disparity, and hand shape constraints. The forest grows scan-line by scan-line from high reliable regions to low reliable regions. First, the vertices in each tree are generated through boundary detection. To get rid of the interference of skin-colored objects, an under-segment vertex splitting step is added in. Second, a scan-line-based forest growing method is proposed by constructing the maximum spanning forest (MSF) with the edges weighted by the consistencies of color and space. Meanwhile, the high-order history information of the tree construction propagates along each growing tree to the current scan-line to predict the 3-D location of the growing hand tree, so as to achieve semiglobal optimization. At last, the growing hand tree is trimmed and labeled with the disparity and hand shape constraints to correct the misconnection in the MSF. The vertex's disparity is obtained by our multipriority vertex stereo matching algorithm using the 3-D location prediction to overcome the problem of occlusion and high similarity between fingers. Experimental results demonstrate that by using our method, the hand can be well segmented. The score and accuracy of the hand segmentation result is about 96.3% and 92.9%, respectively.

A survey on human-computer interaction in virtual reality

Human computer interaction is one of the core technologies of virtual reality, which has an important significance on promoting the widely use of virtual reality and improving the user experience. Due to the development of various sensors and hardwares, human computer interaction technology in virtual reality has made remarkable progress. This paper firstly introduces the paradigm of human computer interaction in virtual reality, and then makes a summary about the main research contents and development trends of various virtual reality and augmented reality technologies, including 3D interaction, hand gesture interaction, handheld devices interaction, speech interaction, haptic interaction and multimodal interaction. Finally, the paper proposes some existing problems that need to be further studied.

A real-time vision-based hand gesture interaction system for virtual EAST

The virtual Experimental Advanced Superconducting Tokamak device (VEAST) is a very complicated 3D model, to interact with which, the traditional interaction devices are limited and inefficient. However, with the development of human-computer interaction (HCI), the hand gesture interaction has become a much popular choice in recent years. In this paper, we propose a real-time vision-based hand gesture interaction system for VEAST. By using one web camera, we can use our bare hand to interact with VEAST at a certain distance, which proves to be more efficient and direct than mouse. The system is composed of four modules: initialization, hand gesture recognition, interaction control and system settings. The hand gesture recognition method is based on codebook (CB) background modeling and open finger counting. Firstly, we build a background model with CB algorithm. Then, we segment the hand region by detecting skin color regions with “elliptical boundary model” in CbCr flat of YCbCr color space. Open finger which is used as a key feature of gesture can be tracked by an improved curvature-based method. Based on the method, we define nine gestures for interaction control of VEAST. Finally, we design a test to demonstrate effectiveness of our system.

가상현실 환경에서 3D 가상객체 조작을 위한 인터페이스와 인터랙션 비교 연구

Received 22 September 2015; received in revised form 3 December 2015; accepted 16 December 2015ABSTRACT Recently immersive virtual reality (VR) becomes popular due to the advanced development of I/O interfaces and related SWs for effectively constructing VR environments. In particular, natu-ral and intuitive manipulation of 3D virtual objects is still considered as one of the most import-ant user interaction issues. This paper presents a comparative study on the manipulation andinteraction of 3D virtual objects using different interfaces and interactions in three VR environ-ments. The comparative study includes both quantitative and qualitative aspects. Three differentexperimental setups are 1) typical desktop-based VR using mouse and keyboard, 2) hand ges-ture-supported desktop VR using a Leap Motion sensor, and 3) immersive VR by wearing anHMD with hand gesture interaction using a Leap Motion sensor. In the desktop VR with handgestures, the Leap Motion sensor is put on the desk. On the other hand, in the immersive VR,the sensor is mounted on the HMD so that the user can manipulate virtual objects in the frontof the HMD. For the quantitative analysis, a task completion time and success rate were mea-sured. Experimental tasks require complex 3D transformation such as simultaneous 3D transla-tion and 3D rotation. For the qualitative analysis, various factors relating to user experiencesuch as ease of use, natural interaction, and stressfulness were evaluated. The qualitative andquantitative analyses show that the immersive VR with the natural hand gesture provides moreintuitive and natural interactions, supports fast and effective performance on task completion,but causes stressful condition. Key Words: Virtual reality, Interface and interaction, User experience, Hand gesture, Leap Motion