Gesture Control Research Articles

Drone Control in AR: An Intuitive System for Single-Handed Gesture Control, Drone Tracking, and Contextualized Camera Feed Visualization in Augmented Reality

Traditional drone handheld remote controllers, although well-established and widely used, are not a particularly intuitive control method. At the same time, drone pilots normally watch the drone video feed on a smartphone or another small screen attached to the remote. This forces them to constantly shift their visual focus from the drone to the screen and vice-versa. This can be an eye-and-mind-tiring and stressful experience, as the eyes constantly change focus and the mind struggles to merge two different points of view. This paper presents a solution based on Microsoft’s HoloLens 2 headset that leverages augmented reality and gesture recognition to make drone piloting easier, more comfortable, and more intuitive. It describes a system for single-handed gesture control that can achieve all maneuvers possible with a traditional remote, including complex motions; a method for tracking a real drone in AR to improve flying beyond line of sight or at distances where the physical drone is hard to see; and the option to display the drone’s live video feed in AR, either in first-person-view mode or in context with the environment.

Gesture recognition real-time control system based on YOLOV4

With the development of industrial information technology in recent years, gesture control has attracted wide attention from scholars. Various gesture control methods have emerged, such as visual control, wearable device control, magnetic field feature extraction control. Based on one of the visual gesture control methods, this paper proposes a visual gesture control method applied to music box control by combining YOLOv4 object detection network. We design seven main gestures, reconstruct gesture datasets, and retrain the YOLOv4 object detection network by the means of the self-built datasets, further build a music box gesture control system. In this paper, we obtain the recognition accuracy of 97.8% for the object detection network in the gesture control system after a series of experiments, and recruit eight volunteers to conduct experimental tests on the self-built gesture-controlled music box system, mainly to quantify the time of executing a single command, attention concentration, etc. The results show that compared with the traditional control method, the visual gesture control method ensures the accuracy while has a faster response speed and takes up less of the user’s attention.

Subject-transfer framework with unlabeled data based on multiple distance measures for surface electromyogram pattern recognition

To improve the initial accuracy of wearable sensor-driven human interfaces, inter-subject variabilities must be reduced through transfer learning. If subject transfer can be performed without labeling the target user’s calibration data, an interface that provides stable accuracy can be easily achieved without a cumbersome calibration protocol. Herein, we propose a subject-transfer framework based on multiple distance measures that enables subject transfer using only unlabeled calibration data by minimizing the distance between the data distributions of the target and the source. To assess the performance of this framework, we used two surface electromyogram databases (one private database and one public database called the NinaPro database 5) acquired from the same wearable sensor, the Myo Gesture Control Armband. The proposed framework improved the pattern recognition accuracy compared with well-established classifiers constructed from randomly selected source subject data. In the future, we will apply this framework to online human interfaces that are not based on a specific calibration protocol. The scripts used in this study can be downloaded from https://github.com/aistairc/Unlabeled_STM.

Research on Robot-Based Gesture Interactive Decoration Design

Recently, automata have become a research hott spot in various countries for a wide range of applications in areas such as security, entertainment, and industry. We designed a mobile robot which is based on gesture control with the focus on gesture interaction control which includes: gesture definition, hand gesture acquisition, hand gesture decoration, hand gesture pre-processing and hand pointer recognition in this paper.This paper uses a combination of qualitative analysis and quantitative research to study the mapping characteristics of gesture interactive decoration and its internal connection with user interaction behaviors by using a combination of theoretical analysis and situational experiments, which not only helps to understand the human-computer interaction decoration Complexity, it has important positive significance in theoretical discussion, research methods and experimental methods. The experiments show that the average recognition rate of the five gestures obtained in this paper reaches 92.5%. The robot can accurately recognize gestures and make corresponding actions

Hand Gesture Control for Human–Computer Interaction with Deep Learning

Hand Gesture Control for Human–Computer Interaction with Deep Learning

Development, Control Adjustment, and Gesture Recognition of a Quadrotor Helicopter

Quadrotor helicopters (quadcopters) have become popular in recent years; because they are simple to operate and steady, they are used in aerial photography, competitive flying, and search-and-rescue missions. In addition, wireless sensors have enabled gesture recognition, and therefore, this study investigated the use of gesture recognition to control a quadcopter. A quadcopter was built using Arduino NANO and MPU-6050 modules, which provided insights into the flight principles of a quadrotor system; furthermore, a proportional integrative derivative controller was modified for steady flight. A Leap Motion device was used to understand the logic behind hand gestures and test the success rate of each gesture, and was then combined with a Parrot AR.Drone 2.0 to achieve gesture recognition controls and uncover problems with connections between hand gestures. These problems were addressed by changing the code in the gesture control program. The results of the completed gesture control experiment demonstrated high gesture recognition performance as well as the ability to meet the requirements for controlling a quadcopter.

Gesture Control Technology of CAD Model Based on Situation Model

Gesture Control Technology of CAD Model Based on Situation Model

Software Controller using Hand Gestures

New technologies emerge in response to the passage of time. Robotic hand gesture control is one of them. Collaboration frameworks based on gestures are becoming increasingly popular in the business and at home. The approach we suggested can greatly reduce the utilization of hardware components such as a keyboard and mouse. The goal of this research is to create a system that can recognize hand gestures and use them as an input command to connect with a computer or laptop. According to a recent study, the use of CNN technology is still lacking in Hand Gesture Recognition. Our research aims to leverage CNN technology to recognize gestures in both static and dynamic modes, and then deploy the trained model in real-time applications.

Advanced Adaptive Spatial Augmented Reality utilizing Dynamic in-situ Projection in Industrial Site Assembly

An Adaptive Spatial Augmented Reality system with dynamic in-situ projection of work instructions and information guidance already shows benefits in the field of industrial site assembly of aircraft components. Furthermore, the user-adaptive gesture control offers advantages for a human-centered and natural interaction directly in the assembly process. This paper presents results of enhancing dynamic in-situ projection, information guidance and user-adaptive gesture interaction in an advanced assembly process of large-sized toy building bricks. For this purpose, the dynamic projection system displays the location of the bricks in the material provision area as well as the accurate mounting position and dynamically guides the operator through the assembly process. Furthermore, the optimized gesture interaction with YOLO and an optimized synthetic dataset creation process for site assembly, indicate strong intentions for user-adaptive interaction with the projection system. An evaluation in the TU Wien pilot factory demonstrates benefits in terms of time savings in locating parts, usability and cognitive workload.

Predicting the role of touchless technologies within diagnostic radiography: Results of an international survey

IntroductionThe evolution of technology within healthcare is continuing at a rapid rate. Touchless technologies (i.e. those involving gestures and voice commands) are rapidly being integrated into daily life. The aim of this study was to investigate the potential role for such technologies within diagnostic radiography. MethodsAn online survey was developed, piloted and deployed using SurveyMonkey as part of an online radiology congress. Eligible respondents were radiographers or radiologic technologists, including students. The survey covered ten themes relating to the potential role of touchless technologies within diagnostic radiography. Results were analysed using descriptive and inferential statistics. Results155 people completed the questionnaire. 100 (64.9%) were women and clinical experience ranged from 13.5 (0–40) years. The majority, 54 (35.1%), had a Bachelor's degree with respondents being from 23 different countries (five continents). 34 (21.9%) respondents did not personally own nor intended to purchase touchless technologies. 89 (84.8%) respondents saw themselves using touchless technologies, if available on current imaging equipment. 25 (16.0%) respondents reported that they currently have access to touchless technologies within their workplace. 88 (81.5%) and 67 (65.0%) respondents reported that they saw voice and gesture controls as being key in improving exam efficiency. ConclusionParticipants clearly perceived a role for touchless technologies within diagnostic radiography. Access to such technologies is not yet widely available within X-ray rooms. Voice activated technologies appear more appealing that gesture-based aids. The primary role for such technologies was defined by participants as focusing on improving examination efficiency. Implications for practiceTouchless technologies have been identified and as important and potentially useful in diagnostic radiography. Collaboration between healthcare institutions, industry and academia is required to design and successfully implement these technologies into practice.

Hand Gesture Recognition of Methods-Time Measurement-1 Motions in Manual Assembly Tasks Using Graph Convolutional Networks

ABSTARCT Gesture recognition is gaining popularity in many fields, including gesture control, robotics, or medical applications. However, the technology is barely used in industrial manufacturing processes due to high costs, a time-consuming configuration, and changes in the workflow. This paper proposes a minimally invasive approach to recognize workers' hand motions in manual assembly tasks. The novelty of this approach is the use of only one camera instead of any other sensors and the application of state-of-the-art graph neural networks. The method relies on monocular RGB video data to predict the basic motions of the industry standard motion-time system Methods-Time Measurement-1. Our two-stage neural network composed of hand key point extraction and adaptive graph convolution delivers accurate classification results in real-time. To train and validate the model, we created a dataset containing 22,000 frames of real-world assembly tasks. The data produced by this method in a production line can be used for motion time verification, assembly-line design, or assembly cost estimation. In a use-case study, we show that the proposed approach can generate Methods-Time Measurement analysis tables. These have so far only been accurately created by human experts. Source code: https://github.com/alexriedel1/Hand-Gesture-Recognition-in-manual-assembly-tasks-using-GCN

A Novel Gesture-Based Control System for Fluorescence Volumetric Data in Virtual Reality.

With the development of light microscopy, it is becoming increasingly easy to obtain detailed multicolor fluorescence volumetric data. The need for their appropriate visualization has become an integral part of fluorescence imaging. Virtual reality (VR) technology provides a new way of visualizing multidimensional image data or models so that the entire 3D structure can be intuitively observed, together with different object features or details on or within the object. With the need for imaging advanced volumetric data, demands for the control of virtual object properties are increasing; this happens especially for multicolor objects obtained by fluorescent microscopy. Existing solutions with universal VR controllers or software-based controllers with the need to define sufficient space for the user to manipulate data in VR are not usable in many practical applications. Therefore, we developed a custom gesture-based VR control system with a custom controller connected to the FluoRender visualization environment. A multitouch sensor disk was used for this purpose. Our control system may be a good choice for easier and more comfortable manipulation of virtual objects and their properties, especially using confocal microscopy, which is the most widely used technique for acquiring volumetric fluorescence data so far.

Using gestures to interact with home automation systems

Homes and working spaces are considered significant contributors to the top percentage of energy consumption and carbon emissions worldwide. Previous studies in the field of home- and building automation have demonstrated the sustainability gain brought by smart home solutions, in terms of energy-efficiency, economic savings, and enhanced living and working conditions. A major barrier, however, to the adoption of these solutions is the complexity and poor usability of user interfaces. In addition, various modes of interactions for the control and automation of residential environments are an emerging area of study within Human-Computer Interaction. As a response to these challenges, this study investigates the use of gestures as a natural way of controlling and interacting with home automation systems. After a survey of available motion capture technologies (Microsoft Kinect and LEAP Motion) and studies related to both, a gesture dictionary will be defined as a set of meaning actions in free form in-air movements. A socio-technical study t will be conducted to measure the resulting aspects such as acceptability, ease-of-use, and culturability. Lastly, the study will present the analysis and effects of gestures control for a higher up-take of smart home solutions towards designing and maintaining buildings of the future that are both user-centric and resource efficient to reduce our overall carbon footprint.

Image Processing Methods for Gesture-Based Robot Control

Abstract In this paper we discuss some image processing methods that can be used for motion recognition of human body parts such as hands or arms in order to interact with robots. This interaction is usually associated to gesture-based control. The considered image processing methods have been experienced for feature recognition in applications involving human robot interaction. They are namely: Sequential Similarity Detection Algorithm (SSDA), an appearance-based approach that uses image databases to model objects, and Kanade-Lucas-Tomasi (KLT) algorithm which is usually used for feature tracking. We illustrate the gesture-based interaction by using KLT algorithm. We discuss the adaptation of each of these methods to the context of gesture-based robot interaction and some of their related issues.

From luxury to necessity: Progress of touchless interaction technology

Touchless Technology is facilitating the move to Zero User Interface(UI) propelled by the COVID-19 pandemic which has accelerated the use of this technology due to hygiene requirements. Zero UI can be defined as a controlled interface that enables user interaction with technology through voice, gestures, hand interaction, eye tracking, and biometrics such as facial recognition and contactless fingerprints. Smart devices, IoT sensors, smart appliances, smart TVs, smart assistants and consumer robotics are predominant examples of devices in which Zero UI is becoming increasingly adopted. These control interfaces include natural interaction modes such as voice or gestures.Touchscreens and shared devices such as kiosks, self-service counters and interactive displays are present in our everyday lives. Each of these interactions however is a concern for consumers in a post-COVID-19 world where hygiene is of utmost importance.The one-stop solution to hygienic interactions includes touchless technology such as voice control, remote mobile screen take over, biometric, and gesture control as Zero User interfaces. With the breakthroughs in image recognition and natural language processing, powered by advanced computer vision and machine learning, “Zero UI” is becoming a new normal. This paper is focusing on the progress of the touchless interaction technology during the COVID-19 pandemic, which actually accelerated development in this concept and moved it from being a luxury to a life necessity.

Comparison of Xbox One and Steam Joystick-based Operating System User Interface using KLM- GOMS

Despite having many advances in video game control such as voice or gesture control, most video game console manufacturers still use a joystick as the main control for their console. Currently, the problem with video game manufacturers is that instead of using the joystick as the point of user interface design, they instead used a design from another device such as a computer or tv. The research will compare Xbox One and Steam OS user interface by using KLM-GOMS Model. Based on the calculation it is concluded that the overall Xbox One has a more efficient design compared to Steam.

Architecture Design and VLSI Implementation of 3D Hand Gesture Recognition System.

With advancements in technology, more and more research is being focused on enhancing daily life quality and convenience. Along with the increase in the development of gesture control systems, many controllers, such as the keyboard, mouse, and other devices, have been replaced with remote control products, which are gradually becoming more intuitive for users. However, vision-based hand gesture recognition systems still have many problems to overcome. Most hand detection methods adopt a skin filter or motion filter for pre-processing. However, in a noisy environment, it is not easy to correctly extract interesting objects. In this paper, a VLSI design with dual-cameras has been proposed to construct a depth map with a stereo matching algorithm and recognize hand gestures. The proposed system adopts an adaptive depth filter to separate interesting foreground objects from the background. We also propose dynamic gesture recognition using depth and coordinate information. The system can perform static and dynamic gesture recognition. The ASIC design is implemented in TSMC 90 nm with about 47.3 K gate counts, and 27.8 mW of power consumption. The average accuracy of each gesture recognition is 83.98%.

Development of Smart Home Gesture-based Control System

Development of Smart Home Gesture-based Control System

Trends in Workplace Wearable Technologies and Connected‐Worker Solutions for Next‐Generation Occupational Safety, Health, and Productivity

The workplace influences the safety, health, and productivity of workers at multiple levels. To protect and promote total worker health, smart hardware, and software tools have emerged for the identification, elimination, substitution, and control of occupational hazards. Wearable devices enable constant monitoring of individual workers and the environment, whereas connected worker solutions provide contextual information and decision support. Here, the recent trends in commercial workplace technologies to monitor and manage occupational risks, injuries, accidents, and diseases are reviewed. Workplace safety wearables for safe lifting, ergonomics, hazard identification, sleep monitoring, fatigue management, and heat and cold stress are discussed. Examples of workplace productivity wearables for asset tracking, augmented reality, gesture and motion control, brain wave sensing, and work stress management are given. Workplace health wearables designed for work‐related musculoskeletal disorders, functional movement disorders, respiratory hazards, cardiovascular health, outdoor sun exposure, and continuous glucose monitoring are shown. Connected worker platforms are discussed with information about the architecture, system modules, intelligent operations, and industry applications. Predictive analytics provide contextual information about occupational safety risks, resource allocation, equipment failure, and predictive maintenance. Altogether, these examples highlight the ground‐level benefits of real‐time visibility about frontline workers, work environment, distributed assets, workforce efficiency, and safety compliance.

Influences of Gesture-Based Mobile Phone Use While Driving

The use of mobile phones while driving has been a hot topic in the field of driving safety for decades. Although there are few studies on the influence of gesture control on in-vehicle secondary tasks, this study aims to investigate the impact of gesture-based mobile phone use without touching while driving from the perspective of multiple-resource workload owing to visual, auditory, cognitive, and psychomotor resource occupation. A novel gesture control technique was adopted for secondary task interactions, to recognize the gestures of drivers. An experiment was conducted to study the influences of two interaction modes, traditional touch-based mobile phone interaction and gesture-based mobile phone interaction, on driving behavior in three different cognitive level task groups. The results indicate that gesture-based mobile phone interaction can improve driving performance with regard to lateral position-keeping ability and steering wheel control; nevertheless, it has no significant impact on longitudinal metrics such as driving speed, driving speed variation, and throttle control variation. Gesture-based mobile phone interactions have a larger effect on secondary tasks with medium cognitive load but not on actual operation tasks. It was also verified that the performance of gesture-based mobile phone interaction was better in secondary mobile phone tasks such as switching (e.g., switching songs) and adjusting (e.g., adjusting volume) than the traditional interaction mode. This study provides the theoretical and experimental support for human–computer interaction using gesture-based mobile phone interactive control in future automobiles.