Autonomous Landing Of Unmanned Aerial Vehicle Research Articles

A semantic SLAM-based method for navigation and landing of UAVs in indoor environments

Autonomous navigation and landing of Unmanned aerial vehicles (UAVs) are critical functions towards full intelligence in unknown environments. However, current methods heavily rely on positioning devices or geometric maps, leading to two limitations: (1) UAVs fail to work normally indoors when signals are blocked, and (2) Navigation systems are not capable of achieving high-level decision-making missions. In this paper, we propose a novel and complete framework to realize the autonomous landing of UAVs in unknown indoor scenes based on visual SLAM, semantic segmentation, terrain estimation, and a decision-making model. Firstly, our 3D map is built on top of the visual SLAM system with semantic features, and then multiple types of maps (e.g., texture map, octo-map, grid-map, and semantic topology) are created for different function requirements. To achieve high-level scene understanding, we design a data association rule to fuse semantic features extracted by a deep learning model into the mapping process of topology structure. Next, a terrain estimation strategy is performed in the lightweight grid-map, which is modeled only via low-level elevation representation. Through multiple terrain constraints factor, a few optimized landing sites are selected in safe regions via clustering algorithm. Finally, based on the guidance of semantic topology, we construct a decision-making model for UAV landing that effectively integrates terrain safety, environment perception, and path planning. We perform extensive autonomous landing experiments on multiple indoor scenes from the TUM RGB-D dataset, and the full unknown environment maps are created simultaneously. The experiment proves that the proposed framework can select an optimal landing site for the advanced mission of UAVs effectively.

A nonlinear model predictive control based control method to quadrotor landing on moving platform

AbstractTo address the problems that the UAV (Unmanned Aerial Vehicle) is vulnerable to distance limitation and environmental interference when tracking and landing on a moving platform autonomously, the accuracy of position estimation relying only on visual odometry in the point‐featureless environment is insufficient, and the traditional linear path planning solvers and controllers cannot meet the fast and safe requirements under the non‐linear strong coupling characteristics of the cooperative landing system, an nonlinear model predictive control (NMPC)‐based multi‐sensor fusion method for autonomous landing of UAVs on motion platforms is proposed. The UAV combines the position information obtained by the RTK‐GPS and the image information obtained by the camera and uses the special identification codes placed in the landing area of the UAV to carry out cooperative planning and navigation while using UKF (Unscented Kalman Filter) to estimate the position of the moving platform and using the interference‐resistant NMPC algorithm to optimise the UAV tracking trajectory based on the precise positioning of the two platforms to achieve the autonomous landing control of the UAV. The simulation and practical experimental results show the feasibility and effectiveness of the proposed algorithm and the autonomous landing control method and provide an effective solution for the autonomous landing of quadrotors on arbitrarily moving platforms.

Vision-Based UAV Landing with Guaranteed Reliability in Adverse Environment

Safe and accurate landing is crucial for Unmanned Aerial Vehicles (UAVs). However, it is a challenging task, especially when the altitude of the landing target is different from the ground and when the UAV is working in adverse environments, such as coasts where winds are usually strong and changing rapidly. UAVs controlled by traditional landing algorithms are unable to deal with sudden large disturbances, such as gusts, during the landing process. In this paper, a reliable vision-based landing strategy is proposed for UAV autonomous landing on a multi-level platform mounted on an Unmanned Ground Vehicle (UGV). With the proposed landing strategy, visual detection can be retrieved even with strong gusts and the UAV is able to achieve robust landing accuracy in a challenging platform with complex ground effects. The effectiveness of the landing algorithm is verified through real-world flight tests. Experimental results in farm fields demonstrate the proposed method’s accuracy and robustness to external disturbances (e.g., wind gusts).

Coarse-to-fine visual autonomous unmanned aerial vehicle landing on a moving platform

Autonomous unmanned aerial vehicle (UAV) landing is a challenging task, especially on a moving platform in an unstructured environment. Under such a scenario, successful UAV landing is mainly affected by poor UAV localization performance. To solve this problem, we propose a coarse-to-fine visual autonomous UAV landing system based on an enhanced visual positioning approach. The landing platform is marked with a specially designed QR code marker, which is developed to improve the landing accuracy when the UAV approaches the landing site. Besides, we employ the you only look once framework to enhance the visual positioning accuracy, thereby promoting the landing platform detection when the UAV is flying far away. The framework recognizes the QR code and decodes the position of a UAV by the corner points of the QR code. Further, we use the Kalman filter to fuse the position data decoded from the QR code with those from the inertia measurement unit sensor. Then, the position data are used for UAV landing with a developed hierarchical landing strategy. To verify the effectiveness of the proposed system, we performed experiments in different environments under various light conditions. The experimental results demonstrate that the proposed system can achieve UAV landing with high accuracy, strong adaptability, and robustness. In addition, it can achieve accurate landing in different operating environments without external real-time kinematic global positioning system (RTK-GPS) signals, and the average landing error is 11.5 cm, which is similar to the landing error when using RTK-GPS signals as the ground truth.

An Integrated UWB-IMU-Vision Framework for Autonomous Approaching and Landing of UAVs

Unmanned Aerial Vehicles (UAVs) autonomous approaching and landing on mobile platforms always play an important role in various application scenarios. Such a complicated autonomous task requires an integrated multi-sensor system to guarantee environmental adaptability in contrast to using each sensor individually. Multi-sensor fusion perception demonstrates great feasibility to compensate for adverse visual events, undesired vibrations of inertia sensors, and satellite positioning loss. In this paper, a UAV autonomous landing scheme based on multi-sensor fusion is proposed. In particular, Ultra Wide-Band (UWB) sensor, Inertial Measurement Unit (IMU), and vision feedback are integrated to guide the UAV to approach and land on a moving object. In the approaching stage, a UWB-IMU-based sensor fusion algorithm is proposed to provide relative position estimation of vehicles with real time and high consistency. Such a sensor integration addresses the open challenge of inaccurate satellite positioning when the UAV is near the ground. It can also be extended to satellite-denied environmental applications. When the landing platform is detected by the onboard camera, the UAV performs autonomous landing. In the landing stage, the vision sensor is involved. With the visual feedback, a deep-learning-based detector and local pose estimator are enabled when the UAV approaches the landing platform. To validate the feasibility of the proposed autonomous landing scheme, both simulation and real-world experiments in extensive scenes are performed. As a result, the proposed landing scheme can land successfully with adequate accuracy in most common scenarios.

Monocular-Vision-Based Precise Runway Detection Applied to State Estimation for Carrier-Based UAV Landing.

Improving the level of autonomy during the landing phase helps promote the full-envelope autonomous flight capability of unmanned aerial vehicles (UAVs). Aiming at the identification of potential landing sites, an end-to-end state estimation method for the autonomous landing of carrier-based UAVs based on monocular vision is proposed in this paper, which allows them to discover landing sites in flight by using equipped optical sensors and avoid a crash or damage during normal and emergency landings. This scheme aims to solve two problems: the requirement of accuracy for runway detection and the requirement of precision for UAV state estimation. First, we design a robust runway detection framework on the basis of YOLOv5 (you only look once, ver. 5) with four modules: a data augmentation layer, a feature extraction layer, a feature aggregation layer and a target prediction layer. Then, the corner prediction method based on geometric features is introduced into the prediction model of the detection framework, which enables the landing field prediction to more precisely fit the runway appearance. In simulation experiments, we developed datasets applied to carrier-based UAV landing simulations based on monocular vision. In addition, our method was implemented with help of the PyTorch deep learning tool, which supports the dynamic and efficient construction of a detection network. Results showed that the proposed method achieved a higher precision and better performance on state estimation during carrier-based UAV landings.

Visual Navigation Algorithm for Night Landing of Fixed-Wing Unmanned Aerial Vehicle

In the recent years, visual navigation has been considered an effective mechanism for achieving an autonomous landing of Unmanned Aerial Vehicles (UAVs). Nevertheless, with the limitations of visual cameras, the effectiveness of visual algorithms is significantly limited by lighting conditions. Therefore, a novel vision-based autonomous landing navigation scheme is proposed for night-time autonomous landing of fixed-wing UAV. Firstly, due to the difficulty of detecting the runway caused by the low-light image, a strategy of visible and infrared image fusion is adopted. The objective functions of the fused and visible image, and the fused and infrared image, are established. Then, the fusion problem is transformed into the optimal situation of the objective function, and the optimal solution is realized by gradient descent schemes to obtain the fused image. Secondly, to improve the performance of detecting the runway from the enhanced image, a runway detection algorithm based on an improved Faster region-based convolutional neural network (Faster R-CNN) is proposed. The runway ground-truth box of the dataset is statistically analyzed, and the size and number of anchors in line with the runway detection background are redesigned based on the analysis results. Finally, a relative attitude and position estimation method for the UAV with respect to the landing runway is proposed. New coordinate reference systems are established, six landing parameters, such as three attitude and three positions, are further calculated by Orthogonal Iteration (OI). Simulation results reveal that the proposed algorithm can achieve 1.85% improvement of AP on runway detection, and the reprojection error of rotation and translation for pose estimation are 0.675∘ and 0.581%, respectively.

Deep Reinforcement Learning with Corrective Feedback for Autonomous UAV Landing on a Mobile Platform

Autonomous Unmanned Aerial Vehicle (UAV) landing remains a challenge in uncertain environments, e.g., landing on a mobile ground platform such as an Unmanned Ground Vehicle (UGV) without knowing its motion dynamics. A traditional PID (Proportional, Integral, Derivative) controller is a choice for the UAV landing task, but it suffers the problem of manual parameter tuning, which becomes intractable if the initial landing condition changes or the mobile platform keeps moving. In this paper, we design a novel learning-based controller that integrates a standard PID module with a deep reinforcement learning module, which can automatically optimize the PID parameters for velocity control. In addition, corrective feedback based on heuristics of parameter tuning can speed up the learning process compared with traditional DRL algorithms that are typically time-consuming. In addition, the learned policy makes the UAV landing smooth and fast by allowing the UAV to adjust its speed adaptively according to the dynamics of the environment. We demonstrate the effectiveness of the proposed algorithm in a variety of quadrotor UAV landing tasks with both static and dynamic environmental settings.

A physics-based digital twin for model predictive control of autonomous unmanned aerial vehicle landing

This paper proposes a two-level, data-driven, digital twin concept for the autonomous landing of aircraft, under some assumptions. It features a digital twin instance (DTI) for model predictive control (MPC); and an innovative, real-time, digital twin prototype for fluid-structure interaction and flight dynamics to inform it. The latter digital twin is based on the linearization about a pre-designed glideslope trajectory of a high-fidelity, viscous, nonlinear computational model for flight dynamics; and its projection onto a low-dimensional approximation subspace to achieve real-time performance, while maintaining accuracy. Its main purpose is to predict in realtime, during flight, the state of an aircraft and the aerodynamic forces and moments acting on it. Unlike static lookup tables or regression-based surrogate models based on steady-state wind tunnel data, the aforementioned real-time digital twin prototype allows the DTI for MPC to be informed by a truly dynamic flight model, rather than a less accurate set of steady-state aerodynamic force and moment data points. The paper describes in detail the construction of the proposed two-level digital twin concept and its verification by numerical simulation. It also reports on its preliminary flight validation in autonomous mode for an off-the-shelf unmanned aerial vehicle instrumented at Stanford University. This article is part of the theme issue 'Data-driven prediction in dynamical systems'.

VIAE-Net: An End-to-End Altitude Estimation through Monocular Vision and Inertial Feature Fusion Neural Networks for UAV Autonomous Landing.

Altitude estimation is one of the fundamental tasks of unmanned aerial vehicle (UAV) automatic navigation, where it aims to accurately and robustly estimate the relative altitude between the UAV and specific areas. However, most methods rely on auxiliary signal reception or expensive equipment, which are not always available, or applicable owing to signal interference, cost or power-consuming limitations in real application scenarios. In addition, fixed-wing UAVs have more complex kinematic models than vertical take-off and landing UAVs. Therefore, an altitude estimation method which can be robustly applied in a GPS denied environment for fixed-wing UAVs must be considered. In this paper, we present a method for high-precision altitude estimation that combines the vision information from a monocular camera and poses information from the inertial measurement unit (IMU) through a novel end-to-end deep neural network architecture. Our method has numerous advantages over existing approaches. First, we utilize the visual-inertial information and physics-based reasoning to build an ideal altitude model that provides general applicability and data efficiency for neural network learning. A further advantage is that we have designed a novel feature fusion module to simplify the tedious manual calibration and synchronization of the camera and IMU, which are required for the standard visual or visual-inertial methods to obtain the data association for altitude estimation modeling. Finally, the proposed method was evaluated, and validated using real flight data obtained during a fixed-wing UAV landing phase. The results show the average estimation error of our method is less than 3% of the actual altitude, which vastly improves the altitude estimation accuracy compared to other visual and visual-inertial based methods.

Real-Time Monocular Vision System for UAV Autonomous Landing in Outdoor Low-Illumination Environments.

Landing an unmanned aerial vehicle (UAV) autonomously and safely is a challenging task. Although the existing approaches have resolved the problem of precise landing by identifying a specific landing marker using the UAV’s onboard vision system, the vast majority of these works are conducted in either daytime or well-illuminated laboratory environments. In contrast, very few researchers have investigated the possibility of landing in low-illumination conditions by employing various active light sources to lighten the markers. In this paper, a novel vision system design is proposed to tackle UAV landing in outdoor extreme low-illumination environments without the need to apply an active light source to the marker. We use a model-based enhancement scheme to improve the quality and brightness of the onboard captured images, then present a hierarchical-based method consisting of a decision tree with an associated light-weight convolutional neural network (CNN) for coarse-to-fine landing marker localization, where the key information of the marker is extracted and reserved for post-processing, such as pose estimation and landing control. Extensive evaluations have been conducted to demonstrate the robustness, accuracy, and real-time performance of the proposed vision system. Field experiments across a variety of outdoor nighttime scenarios with an average luminance of 5 lx at the marker locations have proven the feasibility and practicability of the system.

Real-Time Performance Comparison of Vision-Based Autonomous Landing of Quadcopter on a Ground Moving Target

The tremendous applications of unmanned aerial vehicles (UAVs), such as inspection in complex environments, search, and rescue missions, have established this area of research. The domain of UAVs autonomous navigation and landing has received considerable amount of interest of robotics researchers in the past decades. Nevertheless, the limited capability of UAVs for autonomous landing severely hampers the use of aerial vehicles specially in GPS-denied areas. Fortunately, with the rapid development of computer vision, vision-based techniques have become an important and cheap tool to be used in UAVs for moving target detection and landing. In this paper, four major vision-based techniques namely PID controller, fuzzy logic, sliding mode control and model predictive control have been investigated and their landing performance is compared to each other. These landing techniques are implemented on a quadcopter for the purpose of ground moving target detection and then landing on it. The Raspberry Pi board with two Pi cameras for 3D scene construction and depth estimation along with ultrasonic sensor have been used in the quadcopter. A wheeled mobile robot with landing platform is taken as the target. The landing performance of different techniques has been observed and compared in terms of landing displacement error, time taken and the distance travelled to finish the operation in an open and obstacle-free environment. The simulation results are further verified by the hardware experiments.

Control of Autonomous Landing of UAV of Airplane-Type on the Static and Dynamic Sites with Using of "Flexible" Kinematic Trajectories

The final and one of the most important stages of the aircraft-type unmanned aerial vehicles (UAV) flight is landing. In this regard the problem of automating the control by UAV landing in difficult meteorological conditions is becoming increasingly urgent. In some cases for refueling and recharging UAVs it is advisable to use the dynamic mobile landing site (MLS) instead of the traditional stationary landing site (SLS). In the present paper we consider the setting of and solution of the control problem by the terminal landing maneuver of a UAV. It provides its transfer from the current initial state to the target final state along " flexible" kinematic trajectories both on the SLS and on the MLS. To solve the problem of automatic landing of UAV on SLS or MLS the mathematical model of the dynamics of its movement was developed. It’s based on the concept of " flexible" kinematic trajectories with spatial synchronization of controlled movements. The control algorithm by the terminal vertical landing maneuver of UAV on SLS by the method of dynamics inverse problem using the principle of " flexible" kinematic trajectories is developed. And also the control algorithm by the terminal landing maneuver of UAV on MLS by the method of dynamics inverse problems using the principles of " flexible" kinematic trajectories and aiming into the target point was also developed. Computer approbation of the synthesized control algorithms for the landing maneuver of UAV "Aerosonde" under conditions of various wind disturbances was carried out using digital modeling in the Matlab environment.

Camera Assisted Autonomous UAV Landing

This article describes the use of a monocular camera attached to a multirotor perpendicular to the horizon, to recognize visual cues or artefacts (AprilTag) and use it as an anchor for aerial alignment to finally land on it, thus attempting to make autonomous flights safer and usable in slightly hard-to-reach locations. A Hexacopter frame with the DJI N3 flight controller was used for prototyping and realizing the desired algorithm. Factors like wind speed and gusts were taken into account as well as the center of gravity of the multirotor and the position of the molecular camera attached to the copter facing downwards or at a 90-degree angle. The results of the experiments conducted were verified against existing methods like the GPS (Global Positioning System) waypoint mission provided by major commercial Unmanned Aerial Vehicle(UAV) or Flight Controller manufacturers and were also compared to experimental methods presented in related research articles, fairing excellent results

Sim-to-Real Quadrotor Landing via Sequential Deep Q-Networks and Domain Randomization

The autonomous landing of an Unmanned Aerial Vehicle (UAV) on a marker is one of the most challenging problems in robotics. Many solutions have been proposed, with the best results achieved via customized geometric features and external sensors. This paper discusses for the first time the use of deep reinforcement learning as an end-to-end learning paradigm to find a policy for UAVs autonomous landing. Our method is based on a divide-and-conquer paradigm that splits a task into sequential sub-tasks, each one assigned to a Deep Q-Network (DQN), hence the name Sequential Deep Q-Network (SDQN). Each DQN in an SDQN is activated by an internal trigger, and it represents a component of a high-level control policy, which can navigate the UAV towards the marker. Different technical solutions have been implemented, for example combining vanilla and double DQNs, and the introduction of a partitioned buffer replay to address the problem of sample efficiency. One of the main contributions of this work consists in showing how an SDQN trained in a simulator via domain randomization, can effectively generalize to real-world scenarios of increasing complexity. The performance of SDQNs is comparable with a state-of-the-art algorithm and human pilots while being quantitatively better in noisy conditions.

Autonomous Landing Design of UAVs using Feedback Linearization Controller with Anti Windup Scheme

This paper presents autonomous landing controller design of unmanned aerial vehicles (UAVs). Feedback linearization approach with anti windup scheme is designed as autonomous landing controller. Anti windup basically controls the integrator component accumulation in the autopilot design and restricts the output within saturation limits. Autonomous landing of a fixed wing UAV consists of approach, glideslope and flare phases. During approach, aerial vehicle aligns itself with runway and reduces its lateral deviations with respect to runway central line. In glide slope phase, aerial vehicle maintains a fixed flight path angle and descends with a constant sink rate. In flare phase, aerial vehicle follows an exponential trajectory and descends with a lower sink rate which keeps reducing further as it goes to lower altitudes. Flare controller is designed with integrator and antiwindup scheme is used to handle the controller output within saturation limits. Landing is primarily a longitudinal mode operation but due to disturbances and coupling lateral and directional modes also gets activated. In this paper pitch angle, roll angle and yaw angle nonlinear dynamic equations are linearized to obtain the control commands in terms of elevator deflection, aileron deflection and rudder deflections. Similarly total velocity of aerial vehicle being an important parameter is controlled using thrust command. A first order linearized model of velocity is used to obtain thrust control command. Autonomous landing of UAV with feedback linearization controller and anti windup scheme is simulated with Six-DOF model of AE2 UAV. The algorithm is implemented with wind disturbances to show the autonomous landing performance of UAV.

Vision-based Autonomous Landing of UAV on Moving Platform using a New Marker

The research of Unmanned Aerial Vehicle (UAV) flight autonomous landing on a moving platform based on computer vision has important significance, especially for the rescue and spray UAV in the future. A vision-based relative position and attitude estimation algorithm with a novel color marker and its recognition method was proposed for the UAV autonomous landing system. The new marker is designed with a pattern which make it is as visible as possible in the whole landing process. The proposed relative position and attitude estimation algorithm is based on the four corner points on the marker which are detected by the combined approach of vanishing point of parallel lines and Levenberg-Marquardt (L-M) optimization method. Indoor and outdoor experiments were carried out. The results indicate that the method of marker detection has real-time and robust performance and the position and attitude estimation accuracy attain cm-level, which shows the feasibility of landings on moving platform automatically.

Autonomous Landing of UAV Based on Artificial Neural Network Supervised by Fuzzy Logic

Autonomous Unmanned Aerial Vehicles (UAVs) become an important field of research in which multiple applications can be designed, such as surveillance, deliveries, and others. Thus, studies aiming to improve the performance of these vehicles are being proposed: from new sensing solutions to more robust control techniques. Additionally, the autonomous UAV has challenges in flight stages as the landing. This procedure needs to be performed safely with a reduced error margin in static and dynamic targets. To solve this imperative issue, many applications with computer vision and control theory have been developed. Therefore, this paper presents an alternative method to train a multilayer perceptron neural network based on fuzzy Mamdani logic to control the landing of a UAV on an artificial marker. The advantage of this method is the reduction in computational complexity while maintaining the characteristics and intelligence of the fuzzy logic controller. Results are presented with simulation and real tests for static and dynamic landing spots. For the real experiments, a quadcopter with an onboard computer and ROS is used.

Hybrid Camera Array-Based UAV Auto-Landing on Moving UGV in GPS-Denied Environment

With the rapid development of Unmanned Aerial Vehicle (UAV) systems, the autonomous landing of a UAV on a moving Unmanned Ground Vehicle (UGV) has received extensive attention as a key technology. At present, this technology is confronted with such problems as operating in GPS-denied environments, a low accuracy of target location, the poor precision of the relative motion estimation, delayed control responses, slow processing speeds, and poor stability. To address these issues, we present a hybrid camera array-based autonomous landing UAV that can land on a moving UGV in a GPS-denied environment. We first built a UAV autonomous landing system with a hybrid camera array comprising a fisheye lens camera and a stereo camera. Then, we integrated a wide Field of View (FOV) and depth imaging for locating the UGV accurately. In addition, we employed a state estimation algorithm based on motion compensation for establishing the motion state of the ground moving UGV, including its actual motion direction and speed. Thereafter, according to the characteristics of the designed system, we derived a nonlinear controller based on the UGV motion state to ensure that the UGV and UAV maintain the same motion state, which allows autonomous landing. Finally, to evaluate the performance of the proposed system, we carried out a large number of simulations in AirSim and conducted real-world experiments. Through the qualitative and quantitative analyses of the experimental results, as well as the analysis of the time performance, we verified that the autonomous landing performance of the system in the GPS-denied environment is effective and robust.

Multirotor UAV sensor fusion for precision landing

PurposeIn relation to rapid development of possible applications of unmanned vehicles, new opportunities for their use are emerging. Among the most dynamic, we can distinguish package shipments, rescue and military applications, autonomous flights and unattended transportation. However, most of the UAV solutions have limitations related to their power supplies and the field of operation. Some of these restrictions can be overcome by implementing the cooperation between unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). The purpose of this paper is to explore the problem of sensor fusion for autonomous landing of a UAV on the UGV by comparing the performance of precision landing algorithms using different sensor fusions to have precise and reliable information about the position and velocity.Design/methodology/approachThe difficulties in this scenario, among others, are different coordination systems and necessity for sensor data from air and ground. The most suitable solution seems to be the use of widely available Global Navigational Satellite System (GNSS) receivers. Unfortunately, the position measurements obtained from cheap receivers are encumbered with errors when desiring precision. The different approaches are based on the usage of sensor fusion of Inertial Navigation System and image processing. However most of these systems are very vulnerable to lightning.FindingsIn this paper, methods based on an exchange of telemetry data and sensor fusion of GNSS, infrared markers detection and others are used. Different methods are compared.Originality/valueThe subject of sensor fusion and high-precision measurements in reference to the autonomous vehicle cooperation is very important because of the increasing popularity of these vehicles. The proposed solution is efficient to perform autonomous landing of UAV on the UGV.