UAV Platform Research Articles

Computer Vision for Autonomous UAV Flight Safety: An Overview and a Vision-based Safe Landing Pipeline Example

Recent years have seen an unprecedented spread of Unmanned Aerial Vehicles (UAVs, or “drones”), which are highly useful for both civilian and military applications. Flight safety is a crucial issue in UAV navigation, having to ensure accurate compliance with recently legislated rules and regulations. The emerging use of autonomous drones and UAV swarms raises additional issues, making it necessary to transfuse safety- and regulations-awareness to relevant algorithms and architectures. Computer vision plays a pivotal role in such autonomous functionalities. Although the main aspects of autonomous UAV technologies (e.g., path planning, navigation control, landing control, mapping and localization, target detection/tracking) are already mature and well-covered, ensuring safe flying in the vicinity of crowds, avoidance of passing over persons, or guaranteed emergency landing capabilities in case of malfunctions, are generally treated as an afterthought when designing autonomous UAV platforms for unstructured environments. This fact is reflected in the fragmentary coverage of the above issues in current literature. This overview attempts to remedy this situation, from the point of view of computer vision. It examines the field from multiple aspects, including regulations across the world and relevant current technologies. Finally, since very few attempts have been made so far towards a complete UAV safety flight and landing pipeline, an example computer vision-based UAV flight safety pipeline is introduced, taking into account all issues present in current autonomous drones. The content is relevant to any kind of autonomous drone flight (e.g., for movie/TV production, news-gathering, search and rescue, surveillance, inspection, mapping, wildlife monitoring, crowd monitoring/management), making this a topic of broad interest.

An Over-Actuated Multi-Rotor Aerial Vehicle With Unconstrained Attitude Angles and High Thrust Efficiencies

Fully-actuated multi-rotor aerial platforms are receiving increasing research interests for the capability of six degree-of-freedom (DOF) motions such as hovering at non-horizontal attitude angles. Existing real world hardware and experiments have demonstrated such capability for a limited range of angles. This letter presents an aerial platform that achieves maneuvering at arbitrary attitudes with uniformly high thrust efficiencies. We propose a novel vectoring thrust force actuator by mounting a regular quadcopter on a passive mechanical gimbal mechanism. The UAV platform achieves full six-DOF motion with redundancies from four of these vectoring thrust actuators. We present the hierarchical controller, which generates high-level virtual wrench commands, allocations to actuators, and low-level actuator controls. We demonstrate the first real-world experiments of any pose maneuver over 360 degrees without requiring unwrapping rotations.

Nullspace-Based Control Allocation of Overactuated UAV Platforms

Multirotor copters with full six Degree of Freedom (DoF) maneuvering are often overactuated. The control allocation of overactuated UAV platforms can have an infinite number of solutions due to their redundancy. The most common allocation framework is based on Force Decomposition (FD), which provides a robust least-square solution and is easy to implement, but it does not account for input constraints. A control allocation framework based on Quadratic Programming (QP) can take input constraints; however, it will introduce linear approximation error and it has not been formulated and implemented on physical systems. In this letter, we propose a nullspace-based allocation framework for overactuated UAV platforms that combines the advantages of the FD-based framework and QP-based framework. Our proposed nullspace-based framework can take input constraints while eliminating the linear approximation error in the control allocation. It is also robust against controller sampling frequency and measurement noise. These three allocation frameworks are implemented and compared on our novel overactuated UAV platform in simulation to evaluate the robustness of the proposed method and in experiment for a vertical rotation trajectory to demonstrate its performance.

Bio-efficacy of Unmanned Aerial Vehicle based spraying to manage pests

An agricultural spraying system for UAV platform was developed at the Department of Farm Machinery and Power Engineering, Punjab Agricultural University, Ludhiana during 2019 to spray pesticides in cotton, rice and moong crops for the control of pests. The developed aerial spraying system was operated at three flying speeds, i.e. S1 (2.0 m/s), S2 (3.0 m/s) and S3 (4.0 m/s) and at three flying heights, i.e. H1 (0.50 m), H2 (0.75 m) and H3 (1.0 m). Whitefly and brown plant hopper (BPH) were the targeted pest and its bio-efficacy data were collected after spray application. Results indicated that flying height and forward speed had significant effect on bio-efficacy in all the three crops. In cotton, whitefly populations varied between 19.00-21.40 per three leaves. The maximum percent, i.e. 84.85% reduction in white fly population was observed at 7 DAS in H1S1 treatment. In rice, BPH population was varied between 6.60-9.47 per hill followed by control 17.20 per hill. Maximum percent reduction in BPH population was observed 67.42% at 7 DAS in H3S1 treatment. In moong, reduction in whitefly population was varied between 4.87-5.80 per plant followed by control (7.00 per plant). After three days of spray, combination of flying heights (H1, H2 and H3) and forward speeds (S1, S2 and S3) were found to be significantly effective in dropping whitefly and BPH population. Increase in flying height and forward speed reduced percentage reduction of pests. Combination of flying heights H1 and H2 and forward speeds S1 and S2 treatments were found significantly effective in controlling pests in all three crops.

A Study on Intelligent Recognition Method of High-voltage Line Faults in Mountainous Areas Based on UAV Aerial Photography

The high-altitude high-voltage line is an important part of the power grid system, which plays an important role in the stable operation of the entire power grid. High-altitude high-voltage lines usually cross various complex terrains such as mountains, rivers, and hills, and suffer from severe weather conditions for a long time. And they are prone to various forms of line failures, such as broken strands, damaged insulators, and suspension of foreign objects. With the rapid development of economy and technology, UAV aerial photography has become an important development direction in the field of aviation aerial photography due to its own characteristics of lightness, high efficiency, convenience, low energy consumption, and clear images, which fully meets the application needs of modern society. Embedding the control program on the UAV platform can realize the intelligence of aerial photography, thereby resolving the inherent disadvantages of the original aerial photography technology, and can especially adapt to the application scenarios that cannot be collected manually due to the complex terrain.

Spatio-temporal-spectral-angular observation model that integrates observations from UAV and mobile mapping vehicle for better urban mapping

ABSTRACT In a complex urban scene, observation from a single sensor unavoidably leads to voids in observations, failing to describe urban objects in a comprehensive manner. In this paper, we propose a spatio-temporal-spectral-angular observation model to integrate observations from UAV and mobile mapping vehicle platform, realizing a joint, coordinated observation operation from both air and ground. We develop a multi-source remote sensing data acquisition system to effectively acquire multi-angle data of complex urban scenes. Multi-source data fusion solves the missing data problem caused by occlusion and achieves accurate, rapid, and complete collection of holographic spatial and temporal information in complex urban scenes. We carried out an experiment on Baisha Town, Chongqing, China and obtained multi-sensor, multi-angle data from UAV and mobile mapping vehicle. We first extracted the point cloud from UAV and then integrated the UAV and mobile mapping vehicle point cloud. The integrated results combined both the characteristics of UAV and mobile mapping vehicle point cloud, confirming the practicability of the proposed joint data acquisition platform and the effectiveness of spatio-temporal-spectral-angular observation model. Compared with the observation from UAV or mobile mapping vehicle alone, the integrated system provides an effective data acquisition solution toward comprehensive urban monitoring.

MULTI-TEMPORAL IMAGE CO-REGISTRATION OF UAV BLOCKS: A COMPARISON OF DIFFERENT APPROACHES

Abstract. Traditionally, data co-registration of survey epochs in photogrammetry relied on Ground Control Points (GCP) to keep the reference system unchanged. In the last years, Unmanned Aerial Systems (UAV) are increasingly used in photogrammetric environmental monitoring. The diffusion of affordable UAV platforms equipped with GNSS (Global Navigation Satellite System) centimetre-grade receivers might reduce, but not eliminate, the need for GCP. Conversely, if GNSS-assisted orientation cannot be used or if additional ground control and reliability checks are required, alternatives to repeated GCP survey have been proposed, taking advantage of Structure from Motion (SfM) photogrammetry. In particular, co-registering different epochs image blocks together, identifying corresponding features, has been demonstrated as a viable and efficient approach. In this paper four different strategies easily implementable in a generic commercial photogrammetric software are presented and compared considering three different test sites in Italy subject to different amounts of environmental changes. The influence of the amount and distribution of inter-epoch corresponding points on the accuracy of the reconstruction is investigated. The results show that some of the tested strategies obtains very good results and can be used (although not needed) also in RTK centimetre-grade UAV surveys, leveraging the additional information coming from previous epochs survey to actually increase the survey accuracy and reliability.

A precision viticulture approach based on pure canopy pixel extraction from high resolution images acquired by a multisensor UAV platform

ISHS International Symposium on Precision Management of Orchards and Vineyards A precision viticulture approach based on pure canopy pixel extraction from high resolution images acquired by a multisensor UAV platform

End-to-end performance of an uplink NB-IoT transmission relayed on a low-altitude UAV platform with Non-Orthogonal Single-Carrier FDMA in the optical wireless backhaul link

End-to-end performance of an uplink NB-IoT transmission relayed on a low-altitude UAV platform with Non-Orthogonal Single-Carrier FDMA in the optical wireless backhaul link

Reconstruction of 3D Building Model Using Point of Interest Technique at Different Altitude and Range

The essential techniques used in 3D model reconstruction are LIDAR scanning, close-range photogrammetry, and aerial photogrammetry. The aim of this study is to investigate 3D building model results using a UAV platform using different altitudes and ranges. The methodology was divided into four phases. Phase one was a preliminary study and phase two was the data acquisition phase which included flight planning, camera calibration, and image acquisition at three different altitudes; 150m, 170m, and 190m and at different ranges; 180m, 190m, and 200m, respectively. Phase three was data processing whereby the resulting 3D models were generated using five steps which are image masking, image aligning, image dense clouding, image meshing, and image texturing. Phase four contained analyses on the 3D model such as accuracy assessment of the 3D model and analysis on the different altitudes and ranges employed. This study stated that the maximum accuracy was 0.195m without a control point, while the minimum accuracy was 0.123m with a control point. It shows that the 3D model with a control point produced measurements that were 37% more accurate than the 3D model without a control point. In the final analysis, the results showed that the built 3D model resulting from the UAV platform can provide an accuracy of less than 0.200m with or without control points. These findings can be applied to many fields, such as heritage and conservation in the efforts to preserve the 3D model of heritage objects.

Optimized Deep Learning Model as a Basis for Fast UAV Mapping of Weed Species in Winter Wheat Crops

Weed maps should be available quickly, reliably, and with high detail to be useful for site-specific management in crop protection and to promote more sustainable agriculture by reducing pesticide use. Here, the optimization of a deep residual convolutional neural network (ResNet-18) for the classification of weed and crop plants in UAV imagery is proposed. The target was to reach sufficient performance on an embedded system by maintaining the same features of the ResNet-18 model as a basis for fast UAV mapping. This would enable online recognition and subsequent mapping of weeds during UAV flying operation. Optimization was achieved mainly by avoiding redundant computations that arise when a classification model is applied on overlapping tiles in a larger input image. The model was trained and tested with imagery obtained from a UAV flight campaign at low altitude over a winter wheat field, and classification was performed on species level with the weed species Matricaria chamomilla L., Papaver rhoeas L., Veronica hederifolia L., and Viola arvensis ssp. arvensis observed in that field. The ResNet-18 model with the optimized image-level prediction pipeline reached a performance of 2.2 frames per second with an NVIDIA Jetson AGX Xavier on the full resolution UAV image, which would amount to about 1.78 ha h−1 area output for continuous field mapping. The overall accuracy for determining crop, soil, and weed species was 94%. There were some limitations in the detection of species unknown to the model. When shifting from 16-bit to 32-bit model precision, no improvement in classification accuracy was observed, but a strong decline in speed performance, especially when a higher number of filters was used in the ResNet-18 model. Future work should be directed towards the integration of the mapping process on UAV platforms, guiding UAVs autonomously for mapping purpose, and ensuring the transferability of the models to other crop fields.

Using UAV Borne, Multi-Spectral Imaging for the Field Phenotyping of Shoot Biomass, Leaf Area Index and Height of West African Sorghum Varieties under Two Contrasted Water Conditions

Meeting food demand for the growing population will require an increase to crop production despite climate changes and, more particularly, severe drought episodes. Sorghum is one of the cereals most adapted to drought that feed millions of people around the world. Valorizing its genetic diversity for crop improvement can benefit from extensive phenotyping. The current methods to evaluate plant biomass, leaves area and plants height involve destructive sampling and are not practical in breeding. Phenotyping relying on drone based imagery is a powerful approach in this context. The objective of this study was to develop and validate a high throughput field phenotyping method of sorghum growth traits under contrasted water conditions relying on drone based imagery. Experiments were conducted in Bambey (Senegal) in 2018 and 2019, to test the ability of multi-spectral sensing technologies on-board a UAV platform to calculate various vegetation indices to estimate plants characteristics. In total, ten (10) contrasted varieties of West African sorghum collection were selected and arranged in a randomized complete block design with three (3) replicates and two (2) water treatments (well-watered and drought stress). This study focused on plant biomass, leaf area index (LAI) and the plant height that were measured weekly from emergence to maturity. Drone flights were performed just before each destructive sampling and images were taken by multi-spectral and visible cameras. UAV-derived vegetation indices exhibited their capacity of estimating LAI and biomass in the 2018 calibration data set, in particular: normalized difference vegetative index (NDVI), corrected transformed vegetation index (CTVI), seconded modified soil-adjusted vegetation index (MSAVI2), green normalize difference vegetation index (GNDVI), and simple ratio (SR) (r2 of 0.8 and 0.6 for LAI and biomass, respectively). Developed models were validated with 2019 data, showing a good performance (r2 of 0.92 and 0.91 for LAI and biomass accordingly). Results were also promising regarding plant height estimation (RMSE = 9.88 cm). Regression plots between the image-based estimation and the measured plant height showed a r2 of 0.83. The validation results were similar between water treatments. This study is the first successful application of drone based imagery for phenotyping sorghum growth and development in a West African context characterized by severe drought occurrence. The developed approach could be used as a decision support tool for breeding programs and as a tool to increase the throughput of sorghum genetic diversity characterization for adaptive traits.

Design, Sensing, and Control of a Novel UAV Platform for Aerial Drilling and Screwing

Hole drilling and bolt screwing are frequently performed tasks in construction, decoration, and maintenance. Traditionally, sending human workers to perform these tasks in hard-to-reach locations is both dangerous and costly. In this letter, we present an aerial manipulation platform that allows a human user to remotely conduct omnidirectional drilling and screwing. The design of the platform features a quadrotor UAV with each pair of rotors independently tilted by a servo, forming an “H” configuration, on which a 1-DOF manipulator carrying a motorized drill or screw driver is mounted. With such a design, the end-effector can face any direction on the longitudinal plane and exert a big enough contact force for drilling and screwing without the need of changing the vehicle body's orientation. Compared to previous UAVs that can only drill holes vertically into the ground, the proposed design also allows horizonal drilling/screwing into a wall or a cliff, making it suitable for a vast range of real applications. Based on the dynamic equations of the system, a dual-level control law is proposed. The low-level attitude controller uses an adaptive robust control (ARC) to accurately regulate the attitude angles in the presence of force/torque uncertainties that may occur during the drilling and screwing process, while a selective impedance controller is implemented at high level to indirectly control the contact force commanded by the user. In addition, a vision-based real-time target identification and tracking method integrating a YOLO v3 real-time object detector with feature tracking, and morphological operations is developed to identify and track the target point for drilling and screwing specified by the user. Various in-lab experiments on a self-made prototype demonstrate the feasibility and effectiveness of the proposed approach for aerial drilling and screwing.

Application Research of Tethered UAV Platform in Marine Emergency Communication Network

In order to meet the needs of emergency communication for major emergency disaster rescue, a wireless emergency communication relay system based on tethered UAV platform is studied. From the perspective of practical application, the characteristics and network coverage of the emergency communication system are analyzed. The mooring UAV platform is equipped with various communication loads such as MESH(wireless grid network communication), 4G-LTE( Long term evolution fourth generation mobile communication) base station, AIS (Automatic Identification System) and so on, which are kept on the communication support ship. When the communication support ship enters the scene of maritime emergencies, the tethered UAV platform lifts off ,stays for a long time and realizes the relay communication service of various carriers within a radius of tens of kilometers through its various communication payloads, which provides key communication support for the Marine emergency communication network. The actual field test of the prototype system shows that the data transmission is stable and reliable, and the short message transmission is normal, which can meet the emergency communication demand of disaster rescue.

Building synergetic consensus for dynamic gas-plume tracking applications using UAV platforms

This article investigates the problem of deploying a swarm of UAVs equipped with gas sensors for industrial remote gas-plume sensing. This setup’s objective is to continuously adjust the swarm formation to maximize the combined perception for the dynamically evolved plume’s cloud, focusing around areas with the highest concentration/intensity. Initially, such a setup is formulated into an optimization problem, the solution of which could be acquired by the maximization of an appropriately defined objective function. Due to the model-free approach, this objective function’s analytical form is not available, prohibiting standard gradient descent methodologies. To this end, a tracking algorithm is developed and studied, which operates in a distributed manner and enables the UAV swarm to build a common consensus dynamically, during the evolution of the leakage phenomenon. The overall performance is tested in a simulative yet realistic environment using ANSYS Fluent suite, considering a simultaneous gas-leak incident at two different points. Aside from the standalone evaluation study, the proposed gas-plume tracking scheme is able to outperform a state-of-the-art alternative algorithm, namely Efficient Global Optimization (EGO), in various simulation setups, deploying a different number of UAVs on the field.

Application of LiDAR UAV for High-Resolution Flood Modelling

To overcome the difficulty of collecting fine-grained terrain data that is important for flood modelling, this work presents a method for the application of UAV-based LiDAR techniques to drive high-resolution flood propagation and inundation modelling. This paper comprehensively introduces the UAV platform, LiDAR sensor and data processing techniques required and proposes the approach for obtaining refined DEM for flood management using which the DEM accuracy can reach ±3 cm. Accordingly, two kinds of terrains, a small mountain area and a large urban area, have been measured and the time requirements for the method are 5 h and 2 days respectively. Based on the collected data, a full hydrodynamic numerical flood model is applied to simulate a flash flood in the mountain catchment and an urban flood at high-resolution. The results show that the water depth and velocity affected by key micro terrain features, such as tiny channels and roads, can be captured and considered, indicating that LiDAR UAV techniques are an efficient and reliable method for surveying terrain making them highly important for creating high accurate flood simulation.

Monitoring Asphalt Pavement Aging and Damage Conditions from Low-Altitude UAV Imagery Based on a CNN Approach

Conventional methods for monitoring pavement healthy states have the disadvantages of low efficiency and being time-consuming and destructive. Current studies indicate that traditional machine learning algorithms showed poor performance and low generalization capacity in identifying asphalt pavement aging and damage conditions. Further, deep learning network models have less been applied to the detection of asphalt pavement aging types and damage objects from UAV imagery. In this study, we first used a low-altitude UAV platform to acquire multispectral images of road pavement with centimeter-level spatial resolution. The fine spatial resolution can provide detailed textural information of the pavement damage objects such as cracks and potholes. Afterwards, we combined multiscale semantic segmentation, using the CNN model and SVM classifier into a framework to extract pavement potholes and cracks and classify the pavement surfaces into three aging states. Results demonstrated that the proposed framework achieved the highest overall accuracy (87.83% and 92.96%) and recall rate (85.4% and 90.65%) in the classification of the asphalt pavement images in the two segments of roads in Xinjiang, China. We concluded that the combination of the CNN + SVM and low-altitude UAV multispectral images would contribute to improve the accuracy in the detection of asphalt pavement aging states and damaged objects.

High-throughput phenotyping analysis of maize at the seedling stage using end-to-end segmentation network.

Image processing technologies are available for high-throughput acquisition and analysis of phenotypes for crop populations, which is of great significance for crop growth monitoring, evaluation of seedling condition, and cultivation management. However, existing methods rely on empirical segmentation thresholds, thus can have insufficient accuracy of extracted phenotypes. Taking maize as an example crop, we propose a phenotype extraction approach from top-view images at the seedling stage. An end-to-end segmentation network, named PlantU-net, which uses a small amount of training data, was explored to realize automatic segmentation of top-view images of a maize population at the seedling stage. Morphological and color related phenotypes were automatic extracted, including maize shoot coverage, circumscribed radius, aspect ratio, and plant azimuth plane angle. The results show that the approach can segment the shoots at the seedling stage from top-view images, obtained either from the UAV or tractor-based high-throughput phenotyping platform. The average segmentation accuracy, recall rate, and F1 score are 0.96, 0.98, and 0.97, respectively. The extracted phenotypes, including maize shoot coverage, circumscribed radius, aspect ratio, and plant azimuth plane angle, are highly correlated with manual measurements (R2 = 0.96-0.99). This approach requires less training data and thus has better expansibility. It provides practical means for high-throughput phenotyping analysis of early growth stage crop populations.

Research on multispectral correlation imaging of moving target based on phase modulation

In order to overcome the inability of scanning multi-spectral imaging to capture multi-spectral data in dynamic scenes, a single-exposure multispectral imaging method for moving targets was proposed based on phase modulation. This method combined associated imaging technology, compressed sensing technology and spectral imaging, introduced a spatial random phase modulator into the imaging light path, modulated and compressed the three-dimensional map information data of the moving target object, then used the two-dimensional aliasing signal obtained by the detector to reconstruct the three-dimensional map information to achieve a single exposure and simultaneously obtained the three-dimensional map information of the moving target. It had the advantages of high utilization rate of light energy, short imaging time, and simple system structure. The experimental results show that when the average electron number of a single frame of CCD detection signal increases from 200 e− at intervals of 100 e− to 1300 e−, as the average rRMSE value of the electron number increases, the relative root mean square error of the reconstructed image decreases correspondingly, and the reconstruction improved image quality; when the stepper motor drives the target object to continuously move at a speed of 30 Hz, a multi-spectral reconstructed image of the moving object with better quality can be obtained; a spectrometer is used to test the spectral distribution curves of different spectrum bands in the target object, and the results obtained are basically consistent with the spectral distribution curves of the reconstructed image, which proves the effectiveness of the method. The research results provide a useful reference for the application of multi-spectral correlation imaging technology in UAV platforms, dynamic monitoring and other fields.

BRDF feature observation method and modeling of desert site based on UAV platform

BRDF feature observation method and modeling of desert site based on UAV platform