Multi-degree-of-freedom unmanned aerial vehicle control combining a hybrid brain-computer interface and visual obstacle avoidance

Abstract

ObjectiveThe difficulty of unmanned aerial vehicle (UAV) control recently lies in multidirectional movement in 3-dimensional space, improving control accuracy and manipulation safety. To address these challenges, a UAV control system that incorporates a hybrid brain-computer interface (hBCI), gyroscope and visual obstacle avoidance based on monocular depth estimation is proposed. Approach. We propose an efficient steady-state visual evoked potential (SSVEP) classification network (CL-NET) featuring a one-dimensional convolutional neural network, a long short-term memory module and an attention module to identify the user's intention for UAV movement in the front, back, left and right directions. The take-off, landing and rising control of the UAV is realized by an electrooculogram (EOG) signal detection algorithm, a blink state detector. In addition, the UAV can fly in an oblique state and rotate according to the current head posture detected by a gyroscope. Furthermore, an improved monocular depth estimation network is employed to design the autonomous obstacle avoidance module of the UAV, ensuring the safety of the brain-controlled system in practice. Main results. The proposed CL-NET delivers an accuracy of 98.67% on the public dataset and an accuracy of 97.92% on the self-collected dataset, both of which surpass the performance of state-of-the-art models. Additionally, we set up a brain control group and a remote control group to conduct practical experiments in a realistic environment. In the experiments involving sixteen subjects, the proposed UAV control system reached an average information transfer rate (ITR) of 44.09 bits/min, and the brain control group had a lower collision rate than the remote control group. Significance. The hybrid control method ensures that the multi-degree-of-freedom (multi-DOF) UAV control system maintains outstanding performance while ensuring good safety.