Gesture Commands Research Articles

Skeleton-based traffic command recognition at road intersections for intelligent vehicles

Understanding traffic officer commands is a fundamental perception task for intelligent vehicles in driver assistance and autonomous driving. Previous studies have emphasized explicit traffic command gesture recognition but have not considered situations where the traffic officer is controlling the subjects in other directions, which would also influence decision-making of the ego vehicle. To fill in the gap, this article aims to research visual skeleton-based recognition of traffic commands occurring at road intersections, where both command directions and gestures should be determined. Specifically, a two-stage recognition framework for four cross-shaped directions and eight command gestures is proposed. Two kinds of handcrafted features, including upper-body geometric features and keypoint co-occurrence features, are established with estimated 2D human keypoint coordinates and heatmaps and further combined into a deep learning network. The first stage handles human body orientation classification, while the second stage addresses command gesture recognition with extra usage of the output from the first stage. Combining the recognized body orientation and command gesture, the type of traffic command can ultimately be inferred. For training and validation, a dataset termed the Chinese Traffic Command at Intersections (CTCX) is built. The proposed method gains an outperforming edit accuracy of 89.67% on the CTCX test set, demonstrating its effectiveness. This work provides a foundation in this area and is expected to inspire more research on traffic command recognition with directions in the near future.

Adoption of Artificial Intelligence Along with Gesture Interactive Robot in Musical Perception Education Based on Deep Learning Method

The present work aims to improve students’ interest in music teaching and promote modern teaching. A distributed application system of artificial intelligence gesture interactive robot is designed through deep learning technology and applied to music perception education. First, the user’s gesture instruction data is collected through the double channel convolution neural network (DCCNN). It uses the double-size convolution kernel to extract feature information in the image and collect the video frame’s gesture instruction. Secondly, a two-stream convolutional neural network (two-stream CNN) recognizes the collected gesture instruction data. The spatial and temporal information is extracted from RGB color mode (RGB) images and optical flow images and input into the two-stream CNN to fuse the prediction results of each network as the final detection result. Then, the distributed system used by the interactive robot is introduced. This structure can improve the stability of the interactive systems and reduce the requirements for local hardware performance. Finally, experiments are conducted to test the gesture command acquisition and recognition network, and the performance of the gesture interactive robot in practice. The results indicate that combining convolution kernels of [Formula: see text] and [Formula: see text] can increase the recognition accuracy of DCCNN to 98% and effectively collect gesture instruction data. The gesture recognition accuracy of two-stream CNN after training reaches 90%, higher than the mainstream dynamic gesture recognition algorithm trained with the same data set. Finally, the recognition test of gesture instructions is carried out on the gesture interactive robot reported here. The results show that the recognition accuracy of the gesture interactive robots is more than 90%, meeting the routine interaction needs. Therefore, the interactive gesture robot has good reliability and stability and is applicable to music perception teaching. The research reported here has guiding significance for establishing music teaching with multiple perception modes.

Simple But Effective: Upper-Body Geometric Features for Traffic Command Gesture Recognition

Recognizing traffic command gestures with high accuracy and quick response at a low computational cost is a requisite for driver assistance or autonomous driving. However, it has been understudied for a long time. Existing research takes advantage of increasing development in human action recognition but pays little attention to onboard conditions. In this article, we propose a simple but effective recognition model based on human upper-body geometric features and a long short-term memory (LSTM) network. The handcrafted geometric features can easily be calculated with estimated 2-D human keypoints at a low computational cost but are discriminative and sufficient in classification. Offline and online inferences are implemented to comprehensively evaluate the proposed model. For the sake of robustness required in the automotive domain, dual voting is designed to filter the output in online inference. On the recently published Chinese traffic police gesture (CTPG) dataset, the presented approach is the best with a remarkable improvement of approximately 8% compared to previous LSTM-based methods with handcrafted spatial features and is competitive with advanced GCN-based deep learning methods. The tradeoff pattern is explored to demonstrate how accuracy and response time alter with different training and inference strategies so that a balanced setup can be manually chosen under various application scenarios. Field tests are also carried out with an experimental vehicle, and the results uncover the present gap between research and practical application to some extent, moving a step closer to real-life traffic command gesture recognition.

A Novel Deep Learning Technique for Higher Education System

This study aims to enhance the interest of university classrooms and the learning efficiency of students. From deep learning, gesture recognition algorithms (GRA) and long short-term memory (LSTM) are adopted to build prediction modules. After the system process is completed, a brand-new human-computer interaction (HCI) system is established. The deep learning GRA augments the operation of the recognition algorithm of the pattern matching method created on the previous image segmentation. Its enticements on two merits of gesture estimation are as follows: rapid detection of gesture joint features and convolutional neural network (CNN) image classification, thereby improving GRA. The recognition rates of “take”, “pinch”, and “point” are 98.6%, 99.5%, and 99.4%, respectively. When the data volume of the LSTM network model is less than 10, the prediction accuracy can reach 70%, and the performance is relatively stable. The designed HCI system can better recognize the teacher’s gesture intent in teaching, execute gesture commands, and optimize the interactive teaching method between the teacher and the computer. This system is a derivative form of gamification teaching. While it improves the interest in teaching, it is also of great significance for improving the teaching efficiency of general higher education.

Self-Powered Gesture Recognition Wristband Enabled by Machine Learning for Full Keyboard and Multicommand Input.

Virtual reality is a brand-new technology that can be applied extensively. To realize virtual reality, certain types of human-computer interaction equipment are necessary. Existing virtual reality technologies often rely on cameras, data gloves, game pads, and other equipment. These equipment are either bulky, inconvenient to carry and use, or expensive to popularize. Therefore, the development of a convenient and low-cost high-precision human-computer interaction device can contribute positively to the development of virtual reality technology. In this study, a gesture recognition wristband that can realize a full keyboard and multicommand input is developed. The wristband is convenient to wear, low in cost, and does not affect other daily operations of the hand. This wristband is based on physiological anatomy as well as aided by active sensor and machine learning technology; it can achieve a maximum accuracy of 92.6% in recognizing 26 letters. This wristband offers broad application prospects in the fields of gesture command recognition, assistive devices for the disabled, and wearable electronics.

Design and Application of Interactive Algorithm for Advertising Media Screen Based on Smart Sensor

Touch is one of the most important human senses. With the popularization of touch-screen mobile phones, tablet computers, and other devices, touch-screen interactive technology has become a norm in people’s daily lives, and advertisements that were once dominated by vision and hearing have added an interactive experience in the dimension of touch. Traditional advertising media screens can only complete simple information dissemination functions and cannot interact with users in a two-way manner. They can only receive information one-way and passively and lack interactivity. Touch-screen interactive advertising forms a good interaction with the target audience, thereby disseminating advertising information to achieve the purpose of promotion or brand image building. This paper designs a set of advertising media screen interaction systems based on smart sensors, including a gesture interaction module, a remote interaction module, and a touch interaction module. The gesture interaction module can recognize 5 static gestures and send gesture commands to control the advertising media screen. The remote interaction module can remotely control the advertising media screen, and the touch interaction module can control the advertising media screen through the touch screen. According to the functional requirements, the overall design of software and hardware is given, and the technical background of each module of the software is introduced. Next, the depth image-based gesture recognition method is studied. The number of fingers and the center distance feature are fused as feature vectors, and the weighted template matching method is used to classify and recognize gestures. Finally, the design and implementation of the interactive system are introduced.

Real-Time Hand Gesture Recognition for Media Players

In this modern age of advancement in computing which has made the use of natural user interface much needed. Human gesture recognition is a mode of non-verbal interaction medium and can provide the most intuitive and natural way to interact with computer. The main aim is to make the interaction between human and computer as natural as the interaction between humans. The objective of this paper is to recognize the dynamic hand gesture images based on shapes and relative positions or direction of hand which is extracted from input video stream recording. These hand gestures can be used to recognition system to control multimedia applications running on computer using different gesture commands

‘Address and command’: Two-handed mid-air interactions with multiple home devices

Mid-air gestural interaction with multiple targets (or devices) in the physical space presents several challenges: gesture design gets more complex in the attempt to avoid gesture conflicts and accidental activations; the potential commands are many but the gesture set must be kept small to enable memorability; and, from a design-process perspective, there are not readily available technologies for implementation and prototyping. In this paper, we present the Address and Command (A&C) interaction model that enables two-handed mid-air interactions with multiple remote devices, in a research approach that includes gesture elicitation, prototyping, and evaluation of the user experience. The A&C model requires that users employ the non-dominant hand to address a device (address gestures) and the dominant hand to provide a command to it (command gestures). This approach affords interactions that simultaneously address and command multiple devices in ubiquitous environments that respond to mid-air input. A&C interactions also afford the design of a single gesture for the same command when applicable for multiple devices, thus decreasing the total number of gestures to be memorized and promoting end-user learning of the gesture set. We have (a) conducted an elicitation study (n = 18) to define A&C gestures for seven smart home devices and twelve commands; (b) developed a spatial augmented reality prototype that responds to mid-air gesture commands exploiting MS Kinect sensor and Visual Gesture Builder, Unity 3D and projection-mapping technologies onto foam mockups; (c) tested the approach and the gesture vocabulary in terms of usability, memorability and user experience at a scenario of 36 tasks among devices (n = 17). We found that A&C interactions are feasible, fast, error-free, easy to learn and remember, and are highly valued in terms of user experience.

Implementation of wheelchair controller using mouth and tongue gesture

In this paper, a simple human-machine interface allowing people with severe disabilities to control a motorized wheelchair using mouth and tongue gesture is presented. The development of the proposed system consists of three principal phases: the first phase is mouth detection which performed by using haar cascade to detect the face area and template matching to detect mouth and tongue gestures from the lower face region. The second phase is command extraction; it is carried by determining the mouth and tongue gesture commands according to the detected gesture, the time taken to execute the gestures, and the previous command which is stored in each frame. Finally, the gesture commands are sent to the wheelchair as instruction using the Bluetooth serial port. The hardware used for this project were; laptop with universal serial bus (USB) webcam as a vision-based control unit, Bluetooth module to receive instructions comes from the vision control unit, standard joystick used in case of emergency, joystick emulator which delivers to the control board signals similar to the signals that are usually generated by the standard joystick, and ultrasonic sensors to provide safe navigation. The experimental results showed the success of the proposed control system based on mouth and tongue gestures.

Visual Commands for Control of Food Assistance Robot

Assistance robots improve people's quality of life in residential and office tasks, especially for people with physical limitations. In the case of the elderly or people with upper limb motor disabilities, an assistance robot for food support is necessary. This development is based on a mixed environment, a real and virtual environment working interactively. A camera located in front of the user is used, at a distance of 60 cm, so that it has an excellent visual range to capture the user's hand gestures for the commands. Pattern recognition based on a deep learning algorithm is made with convolutional neural networks to identify the user's hand gestures. This work exposes the network's training and the results of the robot command's execution. A virtual environment is presented in which a robotic arm with a spoon-like effector is used in a machine vision system that allows eight different types of commands to be recognized for the robot by training a faster R-CNN network for which a database of 640 images is used, achieving a degree of system performance of over 95%. The average time in the execution of a cycle from detecting and identify the command gesture to move the robot towards the food and return in front of the user is 21 seconds, making the development useful for real-time applications.

Using Deep Learning Technology to Realize the Automatic Control Program of Robot Arm Based on Hand Gesture Recognition

In this study, the robot arm control, computer vision, and deep learning technologies are combined to realize an automatic control program. There are three functional modules in this program, i.e., the hand gesture recognition module, the robot arm control module, and the communication module. The hand gesture recognition module records the user’s hand gesture images to recognize the gestures’ features using the YOLOv4 algorithm. The recognition results are transmitted to the robot arm control module by the communication module. Finally, the received hand gesture commands are analyzed and executed by the robot arm control module. With the proposed program, engineers can interact with the robot arm through hand gestures, teach the robot arm to record the trajectory by simple hand movements, and call different scripts to satisfy robot motion requirements in the actual production environment.

Action recognition for the robotics and manufacturing automation using 3-D binary micro-block difference

Vision-based control systems play an important role in modern robotics systems. An important task in implementing such a system is developing an effective algorithm for recognizing human actions and the working environment and the design of intuitive gesture commands. This paper proposes an action recognition algorithm for robotics and manufacturing automation. The key contributions are (1) fusion of multimodal information obtained by depth sensors and cameras of the visible range, (2) modified Gabor-based and 3-D binary-based descriptor using micro-block difference, (3) efficient skeleton-based descriptor, and (4) recognition algorithm using the combined descriptor. The proposed binary micro-block difference representation of 3-D patches from video with a complex background in several scales and orientations leads to an informative description of the scene action. The experimental results showed the effectiveness of the proposed algorithm on datasets.

User-Interactive Robot Skin With Large-Area Scalability for Safer and Natural Human-Robot Collaboration in Future Telehealthcare.

With the fourth revolution of healthcare, i.e., Healthcare 4.0, collaborative robotics is spilling out from traditional manufacturing and will blend into human living or working environments to deliver care services, especially telehealthcare. Because of the frequent and seamless interaction between robots and care recipients, it poses several challenges that require careful consideration: 1) the ability of the human to collaborate with the robots in a natural manner; and 2) the safety of the human collaborating with the robot. In this regard, we have proposed a proximity sensing solution based on the self-capacitive technology to provide an extended sense of touch for collaborative robots, allowing approach and contact measurement to enhance safe and natural human-robot collaboration. The modular design of our solution enables it to scale up to form a large-area sensing system. The sensing solution is proposed to work in two operation modes: the interaction mode and the safety mode. In the interaction mode, utilizing the ability of the sensor to localize the point of action, gesture command is used for robot manipulation. In the safety mode, the sensor enables the robot to actively avoid obstacles.

Gesture commands for controlling high-level UAV behavior

Directing groups of unmanned air vehicles (UAVs) is a task that typically requires the full attention of several operators. This can be prohibitive in situations where an operator must pay attention to their surroundings. In this paper we present a gesture device that assists operators in commanding UAVs in focus-constrained environments. The operator influences the UAVs’ behavior by using intuitive hand gesture movements. Gestures are captured using an accelerometer and gyroscope and then classified using a logistic regression model. Ten gestures were chosen to provide behaviors for a group of fixed-wing UAVs. These behaviors specified various searching, following, and tracking patterns that could be used in a dynamic environment. A novel variant of the Monte Carlo Tree Search algorithm was developed to autonomously plan the paths of the cooperating UAVs. These autonomy algorithms were executed when their corresponding gesture was recognized by the gesture device. The gesture device was trained to classify the ten gestures and accurately identified them 95% of the time. Each of the behaviors associated with the gestures was tested in hardware-in-the-loop simulations and the ability to dynamically switch between them was demonstrated. The results show that the system can be used as a natural interface to assist an operator in directing a fleet of UAVs.Article highlightsA gesture device was created that enables operators to command a group of UAVs in focus-constrained environments.Each gesture triggers high-level commands that direct a UAV group to execute complex behaviors.Software simulations and hardware-in-the-loop testing shows the device is effective in directing UAV groups.

Detection and Following of Moving Target by an Indoor Mobile Robot using Multi-sensor Information

Abstract The following behavior is an important aspect in developing of mobile robot control algorithms. It is frequently a substantial part of the control strategies used to control robots that are designed to co-exist and collaborate with human beings in a shared working environment. In this research, an approach is proposed and investigated allowing a holonomic mobile platform to detect and follow targets, moving on the ground, in an indoor environment. Using multi-sensor data flow, the control algorithm is able to localize the target object by its shape and to start following it. The robot will maintain the desired orientation and distance while following the target. Additionally, the mobile platform has also a fully manual control mode option and is able to recognize human gesture commands.

Natural Interaction Design for Navigation in Virtual Environment of Sino-Portuguese Architecture in Museum

Virtual Environment (VE) is an artificial environment created for providing the information in Virtual Reality (VR). The audiences are able to obtain the information via interaction to the environment. Virtual Reality for Culture heritage buildings, which are inherited from past generations and maintained in the present, is an interested topic because its contents often built for giving the information and knowledge about the past to the new generation in virtual museums. However, there is a limitation of Virtual Reality (VR) applications in reproducing cultural sites as most of these applications are not successful in catching public’s attention and involvement. In this research, we focus on using Natural Interaction (NI) interfaces based on a body movement in Virtual Reality for museums to grab attention from audience and to increase public’s involvement. This paper presents the development of Natural Interaction using Microsoft Kinect to define the new grammar of gesture for Sino Portuguese Architecture which is a cultural heritage of Thailand. Results of early tests improved the prototype in progress which alllows more complexity in natural interchanges and connection between users and virtual world. The final result was evaluated after the system was installed in Peranakannitat museum and by interviewing 6 docents. It was found that our system was easy to use as it took only about 3 - 5 min to familiarize the system for navigation in the virtual environment without recognizing the grammar of gesture commands.

Joint Design of Sensing and Communication Systems for Smart Homes

With the recent advance of technologies, smart homes are no longer just the subjects of science fiction and are now becoming a reality. it is essential for a smart home to understand its residents and provide entertainment and healthcare services. Researchers have demonstrated the great potential of RF sensing in smart homes, including understanding the residents' gesture commands and monitoring the residents' health status unobtrusively. Although RF sensing has appealing properties, it brings congestion to wireless networks. To make the situation worse, there is an unprecedented amount of wireless traffic in smart homes. in this article, we investigate the joint design of sensing and communication systems to alleviate the strain on wireless resources. instead of designing sensing and communication systems as separate objectives, we tend to address them jointly. We discuss this issue from two directions. The first is to empower the traditional sensing device with communication capability, so that the sensing device will become a dual-function apparatus; the second is to enhance the sensing capability of WiFi signals by redesigning the packet format so that the current communication infrastructure can also act as a sensing device. We believe that only with joint design can the sensing and communication system work harmoniously in smart homes, providing both unobtrusive sensing and ubiquitous connectivity.

Advances in industrial robots programming applying gestural guidance techniques

In this article, artificial vision is presented as an application for the generation of trajectories through the capture of an operator’s movements using a structured light camera (Kinect) and applying a recognition algorithm to the body’s joints, a three-dimensional representation is made of the skeleton, taking as points of interest the articulation of the left hand to generate the gestural commands and the right hand to indicate the path that the robot must execute. Simulation results are obtained through a development environment where the kinematic model of the robot is incorporated, this indicates that it is possible to generate a trajectory through the human body using artificial vision.

Mid-Air Gesture Control of Multiple Home Devices in Spatial Augmented Reality Prototype

Touchless, mid-air gesture-based interactions with remote devices have been investigated as alternative or complementary to interactions based on remote controls and smartphones. Related studies focus on user elicitation of a gesture vocabulary for one or a few home devices and explore recommendations of respective gesture vocabularies without validating them by empirical testing with interactive prototypes. We have developed an interactive prototype based on spatial Augmented Reality (AR) of seven home devices. Each device responds to touchless gestures (identified from a previous elicitation study) via the MS Kinect sensor. Nineteen users participated in a two-phase test (with and without help provided by a virtual assistant) according to a scenario that required from each user to apply 41 gestural commands (19 unique). We report on main usability indicators: task success, task time, errors (false negative/positives), memorability, perceived usability, and user experience. The main conclusion is that mid-air interaction with multiple home devices is feasible, fairly easy to learn and apply, and enjoyable. The contributions of this paper are (a) validation of a previously elicited gesture set; (b) development of a spatial AR prototype for testing of mid-air gestures, and (c) extensive assessment of gestures and evidence in favor of mid-air interaction in smart environments.

Human Movement Representation on Multivariate Time Series for Recognition of Professional Gestures and Forecasting Their Trajectories.

Human-centered artificial intelligence is increasingly deployed in professional workplaces in Industry 4.0 to address various challenges related to the collaboration between the operators and the machines, the augmentation of their capabilities, or the improvement of the quality of their work and life in general. Intelligent systems and autonomous machines need to continuously recognize and follow the professional actions and gestures of the operators in order to collaborate with them and anticipate their trajectories for avoiding potential collisions and accidents. Nevertheless, the recognition of patterns of professional gestures is a very challenging task for both research and the industry. There are various types of human movements that the intelligent systems need to perceive, for example, gestural commands to machines and professional actions with or without the use of tools. Moreover, the interclass and intraclass spatiotemporal variances together with the very limited access to annotated human motion data constitute a major research challenge. In this paper, we introduce the Gesture Operational Model, which describes how gestures are performed based on assumptions that focus on the dynamic association of body entities, their synergies, and their serial and non-serial mediations, as well as their transitioning over time from one state to another. Then, the assumptions of the Gesture Operational Model are translated into a simultaneous equation system for each body entity through State-Space modeling. The coefficients of the equation are computed using the Maximum Likelihood Estimation method. The simulation of the model generates a confidence-bounding box for every entity that describes the tolerance of its spatial variance over time. The contribution of our approach is demonstrated for both recognizing gestures and forecasting human motion trajectories. In recognition, it is combined with continuous Hidden Markov Models to boost the recognition accuracy when the likelihoods are not confident. In forecasting, a motion trajectory can be estimated by taking as minimum input two observations only. The performance of the algorithm has been evaluated using four industrial datasets that contain gestures and actions from a TV assembly line, the glassblowing industry, the gestural commands to Automated Guided Vehicles as well as the Human–Robot Collaboration in the automotive assembly lines. The hybrid approach State-Space and HMMs outperforms standard continuous HMMs and a 3DCNN-based end-to-end deep architecture.