Operator-Friendly UAV Control System with HMI Using Speech and Gesture Recognition

Abstract

AbstractUAVs (Unmanned Aerial Vehicles) and drones have been used in various fields such as delivery of goods, fire suppression, and traffic monitoring. As a result, the number of beginners in UAV control is also increasing. Traditional methods of UAV control include manual flight using twin-stick controllers or automatic flight by uploading Waypoint-based missions to the FC (Flight Controller) in advance using GCS (Ground Control System) software. However, these conventional UAV control methods take a long time for beginners to get used to it. In manual flight, considerable proficiency is required for UAV control, which increases the control fatigue of novice operators. For beginners, it may be difficult not only to control UAV, but also to handle other tasks such as manipulating a gimbal-camera or starting/pausing missions at the same time as UAV operations. Especially in emergency situations, simultaneous work can be very burdensome because the ability to respond quickly is required. Therefore, it is necessary to develop an operator-friendly control system with HMI (Human Machine Interface) (Zimmer D, Rhodes D (2006) Human–machine interfaces. IEEE Indus Appl Mag 12:29–35) that reduces the operator's controlling stress and does not require high-level control capabilities. We have developed the UAV control system with speech recognition, which is used for AI speakers that are easily seen around us, and gesture recognition, which is used for motion recognition cameras like KINECT, based on the existing control systems. We have implemented speech recognition techniques using PyAudio and SpeechRecognition, speech recognition open-source libraries, and gesture recognition techniques using YOLOv3, an object-detecting open-source library. First of all, to select speech and gesture commands, we analyze the UAV operation procedure and divided step by step. Second, the commands are adopted according to the divided UAV operation steps. Finally, we develop modules to process these commands and do the SITL (Simulation In The Loop). In this paper, we have designed a prototype of an Operator-friendly Control System with HMI (Human Machine Interface) capable of simple speech and gesture commands, and proceed with the SITL. Based on our UAV Control System with HMI, we can identify which sensors and which commands are Operator-friendly. The entire process is simulated using ROS Melodic and gazebo9 in Ubuntu 18.04 environments. As a result, we expect that the people who have no experience in operating UAVs can operate the UAV with HMI control system using speech and gesture recognition in the simulation.KeywordsUAVFlight control systemHMISpeech recognitionGesture recognition