Vision-based Autonomous Landing Research Articles

A Two-Step Controller for Vision-Based Autonomous Landing of a Multirotor with a Gimbal Camera

This article presents a novel vision-based autonomous landing method utilizing a multirotor and a gimbal camera, which is designed to be applicable from any initial position within a broad space by addressing the problems of a field of view and singularity to ensure stable performance. The proposed method employs a two-step controller based on integrated dynamics for the multirotor and the gimbal camera, where the multirotor approaches the landing site horizontally in the first step and descends vertically in the second step. The multirotor and the camera converge simultaneously to the desired configuration because we design the stabilizing controller for the integrated dynamics of the multirotor and the gimbal camera. The controller requires only one feature point and decreases unnecessary camera rolling. The effectiveness of the proposed method is demonstrated through simulation and real environment experiments.

VALNet: Vision-Based Autonomous Landing with Airport Runway Instance Segmentation

VALNet: Vision-Based Autonomous Landing with Airport Runway Instance Segmentation

Real-time safe validation of autonomous landing in populated areas: from virtual environments to Robot-In-The-Loop

Safe autonomous landing for Unmanned Aerial Vehicles (UAVs) in populated areas is a crucial aspect for successful integration of UAVs in populated environments. Nonetheless, validating autonomous landing in real scenarios is a challenging task with a high risk of injuring people. In this work, we propose a framework for safe real-time and thorough evaluation of vision-based autonomous landing in populated scenarios, using photo-realistic virtual environments and physics-based simulation. The proposed evaluation pipeline includes the use of Unreal graphics engine coupled with AirSim for realistic drone simulation to evaluate landing strategies. Then, Software-/Hardware-In-The-Loop can be used to test beforehand the performance of the algorithms. The final validation stage consists in a Robot-In-The-Loop evaluation strategy where a real drone must perform autonomous landing maneuvers in real-time, with an avatar drone in a virtual environment mimicking its behavior, while the detection algorithms run in the virtual environment (virtual reality to the robot). This method determines the safe landing areas based on computer vision and convolutional neural networks to avoid colliding with people in static and dynamic scenarios. To test the robustness of the algorithms in adversary conditions, different urban-like environments were implemented, including moving agents and different weather conditions. We also propose different metrics to quantify the performance of the landing strategies, establishing a baseline for comparison with future works on this challenging task, and analyze them through several randomized iterations. The proposed approach allowed us to safely validate the autonomous landing strategies, providing an evaluation pipeline, and a benchmark for comparison. An extensive evaluation showed a 99% success rate in static scenarios and 87% in dynamic cases, demonstrating that the use of autonomous landing algorithms considerably prevents accidents involving humans, facilitating the integration of drones in human-populated spaces, which may help to unleash the full potential of drones in urban environments. Besides, this type of development helps to increase the safety of drone operations, which would advance drone flight regulations and allow their use in closer proximity to humans.

Vision-Based Autonomous Landing and Charging System for a Hexacopter Drone

Vision-Based Autonomous Landing and Charging System for a Hexacopter Drone

A Vision-Based Autonomous Landing Guidance Strategy for a Micro-UAV by the Modified Camera View

Autonomous landing is one of the key technologies for unmanned aerial vehicles (UAVs) which can improve task flexibility in various fields. In this paper, a vision-based autonomous landing strategy is proposed for a quadrotor micro-UAV based on a novel camera view angle conversion method, fast landing marker detection, and an autonomous guidance approach. The front-view camera of the micro-UAV video is first modified by a new strategy to obtain a top-down view. By this means, the landing marker can be captured by the onboard camera of the micro-UAV and is then detected by the YOLOv5 algorithm in real time. The central coordinate of the landing marker is estimated and used to generate the guidance commands for the flight controller. After that, the guidance commands are sent by the ground station to perform the landing task of the UAV. Finally, the flight experiments using DJI Tello UAV are conducted outdoors and indoors, respectively. The original UAV platform is modified using the proposed camera view angle-changing strategy so that the top-down view can be achieved for performing the landing mission. The experimental results show that the proposed landing marker detection algorithm and landing guidance strategy can complete the autonomous landing task of the micro-UAV efficiently.

Vision-Based Autonomous Landing of a Hybrid Robot on a Powerline

In recent years, various types of inspection robots have been developed to automate powerline inspection. The hybrid robot combines the advantages of climbing and flying robots and has a promising prospect in powerline inspection. But landing a hybrid robot on the target powerline among multiple ones is challenging. Flights require robust detection of powerlines and stable tracking of the target powerline. We propose a complete solution for the autonomous landing of a hybrid robot on a powerline. First, a special feature extraction operator and the corresponding density-based feature recognition algorithm are designed to detect multiscale powerlines. Second, a binocular vision-based depth estimation method for the landing point in the powerline is described. Third, two spatio-temporal dictionaries are established to track the target one in multiple powerlines. Meanwhile, landing strategies and control methods are presented to achieve a stable landing task. Finally, a hybrid robot is designed to validate the proposed method. The experiment results demonstrate the accuracy of the powerline detection and depth estimation algorithm, as well as the effectiveness of the robot in tracking and landing tasks.

Vision-Based Autonomous Landing for the UAV: A Review

With the rapid development of the UAV, it is widely used in rescue and disaster relief, where autonomous landing is the key technology. Vision-based autonomous landing has the advantages of strong autonomy, low cost, and strong anti-interference ability. Moreover, vision navigation has higher guidance and positioning accuracy combined with other navigation methods, such as GPS/INS. This paper summarizes the research results in the field of vision-based autonomous landing for the UAV, and divides it into static, dynamic, and complex scenarios according to the type of landing destination. Among them, the static scenario includes two categories: cooperative targets and natural landmarks; the dynamic scenario is divided into two categories: vehicle-based autonomous landing and ship-based autonomous landing. The key technologies are summarized, compared, and analyzed and the future development trends are pointed out, which can provide a reference for the research on vision-based autonomous landing of UAVs.

Visual Navigation Algorithm for Night Landing of Fixed-Wing Unmanned Aerial Vehicle

In the recent years, visual navigation has been considered an effective mechanism for achieving an autonomous landing of Unmanned Aerial Vehicles (UAVs). Nevertheless, with the limitations of visual cameras, the effectiveness of visual algorithms is significantly limited by lighting conditions. Therefore, a novel vision-based autonomous landing navigation scheme is proposed for night-time autonomous landing of fixed-wing UAV. Firstly, due to the difficulty of detecting the runway caused by the low-light image, a strategy of visible and infrared image fusion is adopted. The objective functions of the fused and visible image, and the fused and infrared image, are established. Then, the fusion problem is transformed into the optimal situation of the objective function, and the optimal solution is realized by gradient descent schemes to obtain the fused image. Secondly, to improve the performance of detecting the runway from the enhanced image, a runway detection algorithm based on an improved Faster region-based convolutional neural network (Faster R-CNN) is proposed. The runway ground-truth box of the dataset is statistically analyzed, and the size and number of anchors in line with the runway detection background are redesigned based on the analysis results. Finally, a relative attitude and position estimation method for the UAV with respect to the landing runway is proposed. New coordinate reference systems are established, six landing parameters, such as three attitude and three positions, are further calculated by Orthogonal Iteration (OI). Simulation results reveal that the proposed algorithm can achieve 1.85% improvement of AP on runway detection, and the reprojection error of rotation and translation for pose estimation are 0.675∘ and 0.581%, respectively.

Vision-Based Autonomous Landing on Unprepared Field With Rugged Surface

Landing on unprepared terrains is a challenging task for fully autonomous micro unmanned aerial vehicles (UAVs). Most of the existing methods mainly rely on manual control when the terrain of the target area is unknown in advance. In this article, we propose an autonomous landing method based on a learning-based multi-view stereo (MVS) system. UAV acquires multiple RGB pictures of the terrain after cruise, and then extracts speed-up robust features (SURF) and perform structure from motion (SFM) to obtain sparse feature point clouds. By utilizing the generated initial depth map, we further propose a novel 3-D reconstruction algorithm named PatchmatchNet-A, which can help to obtain precise and stable estimates of the depth information. In PatchmatchNet-A, we also use a new activation function, adjustable-arctangent linear units (ALU), to improve the accuracy and robustness. We have tested our algorithms in a DTU dataset and found that it can speed up the processing by around 25% while guaranteeing competitive performance. Flight experiments have also been conducted to verify the effectiveness of the whole landing system by using a commercial M210V2 quadcopter.

Vision-Based Autonomous Landing for Unmanned Aerial and Ground Vehicles Cooperative Systems

In this work, we consider a cooperative system in which UAVs perform long-distance missions with assistance of unmanned ground vehicles (UGVs) for battery charging. We propose an autonomous landing scheme for the UAV to land on a mobile UGV with high precision by leveraging multiple-scale Quick Response (QR)-codes for different altitudes. These QR-codes support the acquisition of the relative distance and direction between the UGV and UAV. As a result, the UGV is not required to report its accurate state to the UAV. In contrast to the conventional landing algorithms based on the global positioning system (GPS), the proposed vision-based autonomous landing algorithm is available for both outdoor and GPS-denied scenarios. To cope with the challenge on the moving platform landing, a quadratic programming (QP) problem is formulated to design the velocity controller by combining a control barrier function (CBF) and a control Lyapunov function (CLF). Finally, both extensive computer simulation and a prototype of the proposed system are shown to confirm the feasibility of the proposed system and landing scheme.

Landing Site Detection for Autonomous Rotor Wing UAVs Using Visual and Structural Information

The technology of unmanned aerial vehicles (UAVs) has increasingly become part of many civil and research applications in recent years. UAVs offer high-quality aerial imaging and the ability to perform quick, flexible and in-depth data acquisition over an area of interest. While navigating in remote environments, UAVs need to be capable of autonomously landing on complex terrains for security, safety and delivery reasons. This is extremely challenging as the structure of these terrains is often unknown, and no prior knowledge can be leveraged. In this study, we present a vision-based autonomous landing system for rotor wing UAVs equipped with a stereo camera and an inertial measurement unit (IMU). The landing site detection algorithm introduces and evaluates several factors including terrain’s flatness, inclination and steepness. Considering these features we compute map metrics that are used to obtain a landing-score map, based on which we detect candidate landing sites. The 3D reconstruction of the scene is acquired by stereo processing and the pose of the UAV at any given time is estimated by fusing raw data from the inertial sensors with the pose obtained from stereo ORB-SLAM2. Real-world trials demonstrate successful landing in unknown and complex terrains such as suburban and forest areas.

Photorealistic simulation platform for autonomous landing of fixed-wing aircraft

To investigate the vision-based autonomous landing of fixed-wing aircraft, we propose a photorealistic simulation platform that leverages ROS, Unreal Engine, and Pixhawk 4. This platform adopts a software-in-the-loop model consisting of rendering, control, communication, and sensor modules. To achieve realistic rendering control, the seawater hydrodynamics and fixed-wing aircraft dynamics are modeled. Based on the platform, we create simulation scenarios under different weather and disturbance conditions for autonomous landing of the aircraft carrier. The platform solves the problem that datasets of related scenes are difficult to collect and experiments are difficult to carry out. To verify the developability of the platform, we design and implement a runway line feature point extraction method (vision-based pose estimation algorithm), and evaluate the performance of the method under various conditions. Experiments show that our software-in-the-loop platform enables vision-based algorithm verification and supports simulation research for the autonomous landing of fixed-wing aircraft.

Quadcopter Precision Landing on Moving Targets via Disturbance Observer-Based Controller and Autonomous Landing Planner

Unmanned aerial vehicles, especially quadcopters, play key roles in many real-world applications and the related quadcopter autonomous control algorithms have attracted a great deal of attention. In this paper, we address the vision-based autonomous landing problem of a quadcopter on a ground moving target. Firstly, we propose a disturbance observer-based control algorithm, consisting of a nonlinear disturbance observer and robust altitude and attitude controllers. This algorithm is based on the quadcopter dynamics model, and its stability is strictly proved using Lyapunov’s theory. Secondly, we develop an autonomous landing planner which we test for various landing scenarios to deliver improved reliability and accuracy of the landing mission. These theoretical studies are complemented by a numerical feasibility study, before demonstrating the effectiveness of our approach under actual flight conditions with an experimental quadcopter platform.

Vision-based Autonomous Landing Control of a Multi-rotor Aerial Vehicle on a Moving Platform with Experimental Validations

This paper studies the problem of landing a multi-rotor unmanned aerial vehicle (UAV) on a moving platform (i.e., surface craft, car) based on visual servo. An efficient method is proposed to get the landmark position, in which the camera can follow the moving landmark to supply the relative position information between UAV and the platform. To avoid the divergence of the collected image caused by the swinging camera, an inertial measurement unit (IMU) is used to get the attitude of camera, which will provide the rectification information for the collected image. By using the collected image and the compensation information, we could obtain the proper image and then the relative position can be expressed through the principle of monocular ranging. To demonstrate the effectiveness of the proposed landing method, we present and analyze the experimental results, showing that the landing procedure is steady with a high accuracy.

Real-Time Performance Comparison of Vision-Based Autonomous Landing of Quadcopter on a Ground Moving Target

The tremendous applications of unmanned aerial vehicles (UAVs), such as inspection in complex environments, search, and rescue missions, have established this area of research. The domain of UAVs autonomous navigation and landing has received considerable amount of interest of robotics researchers in the past decades. Nevertheless, the limited capability of UAVs for autonomous landing severely hampers the use of aerial vehicles specially in GPS-denied areas. Fortunately, with the rapid development of computer vision, vision-based techniques have become an important and cheap tool to be used in UAVs for moving target detection and landing. In this paper, four major vision-based techniques namely PID controller, fuzzy logic, sliding mode control and model predictive control have been investigated and their landing performance is compared to each other. These landing techniques are implemented on a quadcopter for the purpose of ground moving target detection and then landing on it. The Raspberry Pi board with two Pi cameras for 3D scene construction and depth estimation along with ultrasonic sensor have been used in the quadcopter. A wheeled mobile robot with landing platform is taken as the target. The landing performance of different techniques has been observed and compared in terms of landing displacement error, time taken and the distance travelled to finish the operation in an open and obstacle-free environment. The simulation results are further verified by the hardware experiments.

Vision-based Autonomous Landing of UAV on Moving Platform using a New Marker

The research of Unmanned Aerial Vehicle (UAV) flight autonomous landing on a moving platform based on computer vision has important significance, especially for the rescue and spray UAV in the future. A vision-based relative position and attitude estimation algorithm with a novel color marker and its recognition method was proposed for the UAV autonomous landing system. The new marker is designed with a pattern which make it is as visible as possible in the whole landing process. The proposed relative position and attitude estimation algorithm is based on the four corner points on the marker which are detected by the combined approach of vanishing point of parallel lines and Levenberg-Marquardt (L-M) optimization method. Indoor and outdoor experiments were carried out. The results indicate that the method of marker detection has real-time and robust performance and the position and attitude estimation accuracy attain cm-level, which shows the feasibility of landings on moving platform automatically.

Autonomous Landing of a Micro Aerial Vehicle on a Moving Platform Using a Composite Landmark

In the existing vision-based autonomous landing systems for micro aerial vehicles (MAVs) on moving platforms, the limited range of landmark localization, the unknown measurement bias of the moving platform (such as wheel-slip or inaccurate calibration of encoders), and landing trajectory knotting seriously affect system performance. To overcome the above shortcomings, an autonomous landing system using a composite landmark is proposed in this paper. In the proposed system, a notched ring landmark and two-dimensional landmark are combined as an R2D landmark to provide visual localization over a wide range. In addition, the wheel-slip and imprecise calibration of encoders are modeled as the unknown measurement bias of the encoders and estimated online via an extended Kalman filter. The landing trajectory is planned by a solver as a convex quadratic programming problem in each control cycle. Meanwhile, an iterative algorithm for adding equality constraints is proposed and used to verify whether the planned trajectory is feasible or not. The simulation and actual landing experiment results verify the following: the visual localization with the R2D landmark has the advantages of wide localization range and high localization accuracy, the pose estimation result of the moving platform with unknown encoder measurement bias is continuous and accurate, and the proposed landing trajectory planning algorithm provides a continuous trajectory for reliable landing.

Vision-based autonomous landing of a quadrotor using a gimbaled camera

This paper proposes a novel vision-based autonomous landing scheme for micro aerial vehicles with perturbations by using a gimbaled camera. There are no sensors available on the moving target in the task. The relative position between the drone and moving target is obtained by the camera mounted on a two-degree-of-freedom pan-tile platform. Firstly, the detection algorithm runs in real time and outputs the pixel tracking errors. Then, an adaptive vision-based controller for the pan-tilt platform guarantees that the line of sight of the camera always tracks the target. Next, the relative position tracking error is approximated according to the gimbal's rotation angles. Disturbance observer-based control is applied for flight control to attenuate the effect of disturbance and recover the hign tracking performance, where relative velocity and lumped disturbances are estimated by extend disturbance observers. The proposed flight controller guarantees that the tracking errors are ultimately bounded with tunable ultimate bounds. The convergence property is demonstrated through Lyapunov theory. The simulations and experiments illustrate the effectiveness and the superiority performance of the proposed control system.

Deep learning for vision-based micro aerial vehicle autonomous landing

Vision-based techniques are widely used in micro aerial vehicle autonomous landing systems. Existing vision-based autonomous landing schemes tend to detect specific landing landmarks by identifying their straightforward visual features such as shapes and colors. Though efficient to compute, these schemes only apply to landmarks with limited variability and require strict environmental conditions such as consistent lighting. To overcome these limitations, we propose an end-to-end landmark detection system based on a deep convolutional neural network, which not only easily scales up to a larger number of various landmarks but also exhibit robustness to different lighting conditions. Furthermore, we propose a separative implementation strategy which conducts convolutional neural network training and detection on different hardware platforms separately, i.e. a graphics processing unit work station and a micro aerial vehicle on-board system, subject to their specific implementation requirements. To evaluate the performance of our framework, we test it on synthesized scenarios and real-world videos captured by a quadrotor on-board camera. Experimental results validate that the proposed vision-based autonomous landing system is robust to landmark variability in different backgrounds and lighting situations.

Vision-Based Autonomous Landing of a Quadrotor on the Perturbed Deck of an Unmanned Surface Vehicle

Autonomous landing on the deck of an unmanned surface vehicle (USV) is still a major challenge for unmanned aerial vehicles (UAVs). In this paper, a fiducial marker is located on the platform so as to facilitate the task since it is possible to retrieve its six-degrees of freedom relative-pose in an easy way. To compensate interruption in the marker’s observations, an extended Kalman filter (EKF) estimates the current USV’s position with reference to the last known position. Validation experiments have been performed in a simulated environment under various marine conditions. The results confirmed that the EKF provides estimates accurate enough to direct the UAV in proximity of the autonomous vessel such that the marker becomes visible again. Using only the odometry and the inertial measurements for the estimation, this method is found to be applicable even under adverse weather conditions in the absence of the global positioning system.