Unmanned Aerial Vehicle Video Research Articles

The Influence of Camera Calibration on Nearshore Bathymetry Estimation from UAV Videos

Measuring the nearshore bathymetry is critical in coastal management and morphodynamic studies. The recent advent of Unmanned Aerial Vehicles (UAVs), in combination with coastal video monitoring techniques, allows for an alternative and low cost evaluation of the nearshore bathymetry. Camera calibration and stabilization is a critical issue in bathymetry estimation from video systems. This work introduces a new methodology in order to obtain such bathymetries, and it compares the results to echo-sounder ground truth data. The goal is to gain a better understanding on the influence of the camera calibration and stabilization on the inferred bathymetry. The results show how the proposed methodology allows for accurate evaluations of the bathymetry, with overall root mean square errors in the order of 40 cm. It is shown that the intrinsic calibration of the camera, related to the lens distortion, is the most critical aspect. Here, the intrinsic calibration that was obtained directly during the flight yields the best results.

RETRACTED ARTICLE: Temporal and non-temporal contextual saliency analysis for generalized wide-area (GWA) search within UAV video

Unmanned aerial vehicles (UAVs) can be used to great effect for wide-area searches such as search and rescue operations. UAV enables search and rescue teams to cover large areas more efficiently and in less time. However, using UAV for this purpose involves the creation of large amounts of data, typically in video format, which must be analyzed before any potential findings can be uncovered and actions taken. This is a slow and expensive process, which can result in significant delays to the response time after a target is seen by the UAV. To solve this problem, a deep model using a visual saliency approach to automatically analyze and detect anomalies in UAV video is proposed. Temporal contextual saliency model is based on the state of the art in visual saliency detection using deep convolutional neural networks and considers local and scene context, with novel additions in utilizing temporal information through a convolutional LSTM layer and modifications to the base model. Additionally, the impact of temporal versus non-temporal reasoning for this task is evaluated. This model achieves improved results on a benchmark dataset with the addition of temporal reasoning showing significantly improved results compared to the state of the art in saliency detection.

Recognition of UAV Video Signal Using RF Fingerprints in the Presence of WiFi Interference

This paper investigates the problem of unmanned aerial vehicle (UAV) recognition in the presence of WiFi interference using passive radio frequency (RF) detection. The proposed method relies on machine learning based RF recognition and considers the scenario in the bandwidth of the video signal (VS) and WiFi are identical. Our machine learning strategy involves extracting 31 features from the WiFi signal and the UAV VS, which are then input to the classifier. Among the 31 features, 30 are statistical in the time and frequency domain, while the remaining one involves the effective subcarrier feature. We evaluate four different machine learning (ML) classifier variants and demonstrate through simulation and experiments that the proposed method can accurately recognize UAV VS in the presence of WiFi interference. We also improve the feature-vector compactness and reduce the 31-feature vector to a 6-feature vector composed of the most significant features and demonstrate that the recognition performance of random forest method (RandF) classifier is not compromised. The most appealing performance for the 6-feature vector case is attained by the RandF, managing a recognition rate of 100% in the indoor experiment and 96.26% in the 2 km outdoor experiment. The recognition rate of the four ML classifiers is larger than 95.52% in the 2 km outdoor experiment, which is better than other UAV detection methods such as radar, acoustic and video.

ERA: A Data Set and Deep Learning Benchmark for Event Recognition in Aerial Videos [Software and Data Sets

As a result of the increasing use of unmanned aerial vehicles (UAVs), large volumes of aerial videos have been produced. It is unrealistic for humans to screen such big data and understand the contents. Hence, methodological research on the automatic understanding of UAV videos is of paramount importance (Figure 1). In this article, we introduce a novel problem of event recognition in unconstrained aerial videos in the remote sensing community and present the large-scale, human-annotated Event Recognition in Aerial Videos (ERA) data set, consisting of 2,864 videos, each with a label from 25 different classes corresponding to an event unfolding for five seconds. All these videos are collected from YouTube. The ERA data set is designed to have significant intraclass variation and interclass similarity and captures dynamic events in various circumstances and at dramatically various scales. Moreover, to offer a benchmark for this task, we extensively validate existing deep networks. We expect that the ERA data set will facilitate further progress in automatic aerial video comprehension. The data set and trained models can be downloaded from https://lcmou.github.io/ERA_Dataset/.

Vision and Deep Learning-Based Algorithms to Detect and Quantify Cracks on Concrete Surfaces from UAV Videos.

Immediate assessment of structural integrity of important civil infrastructures, like bridges, hospitals, or dams, is of utmost importance after natural disasters. Currently, inspection is performed manually by engineers who look for local damages and their extent on significant locations of the structure to understand its implication on its global stability. However, the whole process is time-consuming and prone to human errors. Due to their size and extent, some regions of civil structures are hard to gain access for manual inspection. In such situations, a vision-based system of Unmanned Aerial Vehicles (UAVs) programmed with Artificial Intelligence algorithms may be an effective alternative to carry out a health assessment of civil infrastructures in a timely manner. This paper proposes a framework of achieving the above-mentioned goal using computer vision and deep learning algorithms for detection of cracks on the concrete surface from its image by carrying out image segmentation of pixels, i.e., classification of pixels in an image of the concrete surface and whether it belongs to cracks or not. The image segmentation or dense pixel level classification is carried out using a deep neural network architecture named U-Net. Further, morphological operations on the segmented images result in dense measurements of crack geometry, like length, width, area, and crack orientation for individual cracks present in the image. The efficacy and robustness of the proposed method as a viable real-life application was validated by carrying out a laboratory experiment of a four-point bending test on an 8-foot-long concrete beam of which the video is recorded using a camera mounted on a UAV-based, as well as a still ground-based, video camera. Detection, quantification, and localization of damage on a civil infrastructure using the proposed framework can directly be used in the prognosis of the structure’s ability to withstand service loads.

Deep Multimodality Learning for UAV Video Aesthetic Quality Assessment

Despite the growing number of unmanned aerial vehicles (UAVs) and aerial videos, there is a paucity of studies focusing on the aesthetics of aerial videos that can provide valuable information for improving the aesthetic quality of aerial photography. In this article, we present a method of deep multimodality learning for UAV video aesthetic quality assessment. More specifically, a multistream framework is designed to exploit aesthetic attributes from multiple modalities, including spatial appearance, drone camera motion, and scene structure. A novel specially designed motion stream network is proposed for this new multistream framework. We construct a dataset with 6,000 UAV video shots captured by drone cameras. Our model can judge whether a UAV video was shot by professional photographers or amateurs together with the scene type classification. The experimental results reveal that our method outperforms the video classification methods and traditional SVM-based methods for video aesthetics. In addition, we present three application examples of UAV video grading, professional segment detection and aesthetic-based UAV path planning using the proposed method.

Silver gull harassment of humpback whales in Exmouth Gulf, Western Australia

We share observations of silver gulls (Chroicocephalus novaehollandiae) harassing humpback whales (Megaptera novaeangliae) in Exmouth Gulf, Western Australia, between August and October in 2018 and 2019. Sixteen events were recorded of gulls pecking at the skin of 16 mothers and 2 calves resting on the surface. Data were collected opportunistically from a research vessel covering 5474.2km (565h). Events were documented through photographs and unmanned aerial vehicle video recordings and persisted for an average minimum time of 20.73min (range 2.50–46.85min). At least one event of a gull consistently gouging in a pre-existing lesion was recorded on a mother. Changes in behavioural activity from the whales, which varied from slipping under the surface to travelling and performing instantaneous behavioural events, were observed in 94% (15/16) of events. These are the first known records of birds attacking humpback whales. The results are comparable to early events of kelp gull (Larus dominicanus) harassment of southern right whales (Eubalaena australis) off Península Valdés, Argentina, where events have escalated over the past 50 years to consistent attacks on the whales’ skin and blubber. Future research is required to monitor the trajectory of these interactions to inform management.

Deep learning and handcrafted features for one-class anomaly detection in UAV video

Visual surveillance systems have recently captured the attention of the research community. Most of the proposed surveillance systems deal with stationary cameras. Nevertheless, these systems may reflect minor applicability in anomaly detection when multiple cameras are required. Lately, under technological progress in electronic and avionics systems, Unmanned Aerial Vehicles (UAVs) are increasingly used in a wide variety of urban missions. Especially, in the surveillance context, UAVs can be used as mobile cameras to overcome weaknesses of stationary cameras. One of the principal advantages that makes UAVs attractive is their ability to provide a new aerial perspective. Despite their numerous advantages, there are many difficulties associated with automatic anomalies detection by an UAV, as there is a lack in the proposed contributions describing anomaly detection in videos recorded by a drone. In this paper, we propose new anomaly detection techniques for assisting UAV based surveillance mission where videos are acquired by a mobile camera. To extract robust features from UAV videos, three different features extraction methods were used, namely a pretrained Convolutional Neural Network (CNN) and two popular handcrafted methods (Histogram of Oriented Gradient (HOG) and HOG3D). One Class Support Vector Machine (OCSVM) has been then applied for the unsupervised classification. Extensive experiments carried on a dataset containing videos taken by an UAV monitoring a car parking, prove the efficiency of the proposed techniques. Specifically, the quantitative results obtained using the challenging Area Under Curve (AUC) evaluation metric show that, despite the variation among them, the proposed methods achieve good results in comparison to the existing technique with an AUC = 0.78 at worst and an AUC = 0.93 at best.

REMOVED: Detecting anomalies within Unmanned Aerial Vehicle (UAV) video based on contextual saliency

Abstract Unmanned Aerial Vehicles (UAV) can be used to great effect for the purposes of surveillance or search and rescue operations. UAV enable search and rescue teams to cover large areas more efficiently and in less time. However, using UAV for this purpose involves the creation of large amounts of data (typically video) which must be analyzed before any potential findings can be uncovered and actions taken. This is a slow and expensive process which can result in significant delays to the response time after a target is seen by the UAV. To solve this problem, it is proposed a deep model using a visual saliency approach to automatically analyze and detect anomalies in UAV video. Contextual Saliency for Anomaly Detection in UAV Video (CSADUV) model is based on the state-of-the-art in visual saliency detection using deep convolutional neural networks and considers local and scene context, with novel additions in utilizing temporal information through a convolutional LSTM layer and modifications to the base model. This model achieves promising results with the addition of the temporal implementation producing significantly improved results compared to the state-of-the-art in saliency detection. However, due to limitations in the dataset used the model fails to generalize well to other data, failing to beat the state-of-the-art in anomaly detection in UAV footage. The approach taken shows promise with the modifications made yielding significant performance improvements and is worthy of future investigation. The lack of a publicly available dataset for anomaly detection in UAV video poses a significant roadblock to any deep learning approach to this task, however despite this paper shows that leveraging temporal information for this task, which the state-of-the-art does not currently do, can lead to improved performance.

Multi-level prediction Siamese network for real-time UAV visual tracking

Existing deployed Unmanned Aerial Vehicles (UAVs) visual trackers are usually based on the correlation filter framework. Although these methods have certain advantages of low computational complexity, the tracking performance of small targets and fast motion scenarios is not satisfactory. In this paper, we present a novel multi-level prediction Siamese network (MLPS) for object tracking in UAV videos, which consists of Siamese feature extraction module and multi-level prediction module. The multi-level prediction module can make full use of the characteristics of each layer features to achieve robust evaluation of targets with different scales. Meanwhile, for small-size target tracking, we design a residual feature fusion block, which is used to constrain the low-level feature representation by using high-level abstract semantics, and obtain the improvement of the tracker's ability to distinguish scene details. In addition, we propose a layer attention fusion block which is sensitive to the informative features of each layers to achieve adaptive fusion of different levels of correlation responses by dynamically balancing the multi-layer features. Sufficient experiments on several UAV tracking benchmarks demonstrate that MLPS achieves state-of-the-art performance and runs at a speed over 97 FPS.

Semantic-Aware Real-Time Correlation Tracking Framework for UAV Videos

Discriminative correlation filter (DCF) has contributed tremendously to address the problem of object tracking benefitting from its high computational efficiency. However, it has suffered from performance degradation in unmanned aerial vehicle (UAV) tracking. This article presents a novel semantic-aware real-time correlation tracking framework (SARCT) for UAV videos to enhance the performance of DCF trackers without incurring excessive computing cost. Specifically, SARCT first constructs an additional detection module to generate ROI proposals and to filter any response regarding the target irrelevant area. Then, a novel semantic segmentation module based on semantic template generation and semantic coefficient prediction is further introduced to capture semantic information, which can provide precise ROI mask, thereby effectively suppressing background interference in the ROI proposals. By sharing features and specific network layers for object detection and semantic segmentation, SARCT reduces parameter redundancy to attain sufficient speed for real-time applications. Systematic experiments are conducted on three typical aerial datasets in order to evaluate the performance of the proposed SARCT. The results demonstrate that SARCT is able to improve the accuracy of conventional DCF-based trackers significantly, outperforming state-of-the-art deep trackers.

Inferring Visual Biases in UAV Videos from Eye Movements

Unmanned Aerial Vehicle (UAV) imagery is gaining a lot of momentum lately. Indeed, gathered information from a bird-point-of-view is particularly relevant for numerous applications, from agriculture to surveillance services. We herewith study visual saliency to verify whether there are tangible differences between this imagery and more conventional contents. We first describe typical and UAV contents based on their human saliency maps in a high-dimensional space, encompassing saliency map statistics, distribution characteristics, and other specifically designed features. Thanks to a large amount of eye tracking data collected on UAV, we stress the differences between typical and UAV videos, but more importantly within UAV sequences. We then designed a process to extract new visual attention biases in the UAV imagery, leading to the definition of a new dictionary of visual biases. We then conduct a benchmark on two different datasets, whose results confirm that the 20 defined biases are relevant as a low-complexity saliency prediction system.

Automated Safety Diagnosis Based on Unmanned Aerial Vehicle Video and Deep Learning Algorithm

This paper presents an automated traffic safety diagnostics solution named “Automated Roadway Conflict Identification System” (ARCIS) that uses deep learning techniques to process traffic videos collected by unmanned aerial vehicle (UAV). Mask region convolutional neural network (R-CNN) is employed to improve detection of vehicles in UAV videos. The detected vehicles are tracked by a channel and spatial reliability tracking algorithm, and vehicle trajectories are generated based on the tracking algorithm. Missing vehicles can be identified and tracked by identifying stationary vehicles and comparing intersect of union (IOU) between the detection results and the tracking results. Rotated bounding rectangles based on the pixel-to-pixel manner masks that are generated by mask R-CNN detection are introduced to obtain precise vehicle size and location data. Based on the vehicle trajectories, post-encroachment time (PET) is calculated for each conflict event at the pixel level. By comparing the PET values and the threshold, conflicts with the corresponding pixels in which the conflicts happened can be reported. Various conflict types: rear-end, head on, sideswipe, and angle, can also be determined. A case study at a typical signalized intersection is presented; the results indicate that the proposed framework could significantly improve the accuracy of the output data. Moreover, safety diagnostics for the studied intersection are conducted by calculating the PET values for each conflict event. It is expected that the proposed detection and tracking method with UAVs could help diagnose road safety problems efficiently and appropriate countermeasures could then be proposed.

Use of unmanned aerial vehicles (UAVs) for mark-resight nesting population estimation of adult female green sea turtles at Raine Island

Nester abundance is a key measure of the performance of the world's largest green turtle rookery at Raine Island, Australia, and has been estimated by mark-resight counts since 1984. Nesters are first marked by painting their carapace with a longitudinal white stripe. Painted and unpainted turtles are then counted by a surface observer on a small boat in waters adjacent to the reef. Unmanned aerial vehicles (UAV) and underwater video may provide more cost-effective and less biased alternatives to this approach, but estimates must be comparable with historical estimates. Here we compare and evaluate the three methods. We found comparatively little variation in resighting probabilities between consecutive days of sampling or time of day, which supports an underlying assumption of the method (i.e. demographic closure during sampling). This lack of bias in the location availability for detection of painted versus unpainted turtles and further supported by a parallel satellite tracking study of 40 turtles at Raine Island. Our results demonstrated that surface observers consistently reported higher proportions of marked turtles than either the UAV or underwater video method. This in turn yielded higher population estimates with UAV or underwater video compared to the historical surface observer method, which suggested correction factors of 1.53 and 1.73 respectively. We attributed this to observer search error because a white marked turtle is easier to spot than the non-marked turtle. In contrast, the UAV and underwater video methods allowed subsequent frame-by-frame review, thus reducing observer search error. UAVs were the most efficient in terms of survey time, personnel commitment and weather tolerance compared to the other methods. However, underwater video may also be a useful alternative for in-water mark-resight surveys of turtles.

Advanced framework for microscopic and lane‐level macroscopic traffic parameters estimation from UAV video

Unmanned aerial vehicle (UAV) is at the heart of modern traffic sensing research due to its advantages of low cost, high flexibility, and wide view range over traditional traffic sensors. Recently, increasing efforts in UAV-based traffic sensing have been made, and great progress has been achieved on the estimation of aggregated macroscopic traffic parameters. Compared to aggregated macroscopic traffic data, there has been extensive attention on higher-resolution traffic data such as microscopic traffic parameters and lane-level macroscopic traffic parameters since they can help deeply understand traffic patterns and individual vehicle behaviours. However, little existing research can automatically estimate microscopic traffic parameters and lane-level macroscopic traffic parameters using UAV videos with a moving background. In this study, an advanced framework is proposed to bridge the gap. Specifically, three functional modules consisting of multiple processing streams and the interconnections among them are carefully designed with the consideration of UAV video features and traffic flow characteristics. Experimental results on real-world UAV video data demonstrate promising performances of the framework in microscopic and lane-level macroscopic traffic parameters estimation. This research pushes off the boundaries of the applicability of UAVs and has an enormous potential to support advanced traffic sensing and management.

Image recognition in UAV videos using convolutional neural networks

In recent years, unmanned aerial vehicles (UAVs) have been used in different areas of applications such as rescue operations, surveillance, agriculture, aerial mapping, engineering applications and research, among others, in order to perform tasks with greater efficiency. This work focuses on the use of UAVs in the fishing sector in order to optimise the detection process of a shoal of fish. In this sense, the main idea is to perform images recognition using the images acquired through videos captured by UAV in the open sea; to achieve the objective the convolutional neural networks were used, a new dataset with different images captured through UAV videos in the open sea were taken into account, these classes correspond to dolphin, dolphin_pod, open_sea, and seabirds. The training tests were by transfer of learning using the following models: Inception V3, MobileNet V2, and NASNet-A (large) trained on TensorFlow platform. The experimental results show the detection performance with high-precision values in reasonable processing time. This study ends with a critical discussion of the experimental results.

Real‐time running detection system for UAV imagery based on optical flow and deep convolutional networks

A fast-running human detection system for the unmanned aerial vehicle (UAV) based on optical flow and deep convolution networks is proposed in this study. In the system, running humans can be detected in real-time at the speed of 15 frames per second (fps) with an 81.1% detection accuracy. To fast locate the candidate targets, optical flow representing the motion information is calculated with every two successive frames. A series of prior-processing operations, including spatial average filtering, morphological expansion and outer contour extraction, are performed to extract the regions of interest. A classification model based on small-kernel convolution networks is proposed to achieve the accurate recognition of the running people in various backgrounds. In the model, small convolutional filters are adopted to accelerate the speed of the data representation. Moreover, a total of 60,000 samples are collected to enhance the robustness of the model to adapt to the complex outdoor UAV scenes. The proposed method is compared with other deep learning frameworks for object detection. Field experiments on UAV videos are performed to verify that the proposed system can effectively detect the running people targets in real-time.

Online Semantic Subspace Learning with Siamese Network for UAV Tracking

In urban environment monitoring, visual tracking on unmanned aerial vehicles (UAVs) can produce more applications owing to the inherent advantages, but it also brings new challenges for existing visual tracking approaches (such as complex background clutters, rotation, fast motion, small objects, and realtime issues due to camera motion and viewpoint changes). Based on the Siamese network, tracking can be conducted efficiently in recent UAV datasets. Unfortunately, the learned convolutional neural network (CNN) features are not discriminative when identifying the target from the background/clutter, In particular for the distractor, and cannot capture the appearance variations temporally. Additionally, occlusion and disappearance are also reasons for tracking failure. In this paper, a semantic subspace module is designed to be integrated into the Siamese network tracker to encode the local fine-grained details of the target for UAV tracking. More specifically, the target’s semantic subspace is learned online to adapt to the target in the temporal domain. Additionally, the pixel-wise response of the semantic subspace can be used to detect occlusion and disappearance of the target, and this enables reasonable updating to relieve model drifting. Substantial experiments conducted on challenging UAV benchmarks illustrate that the proposed method can obtain competitive results in both accuracy and efficiency when they are applied to UAV videos.

EyeTrackUAV2: A Large-Scale Binocular Eye-Tracking Dataset for UAV Videos

The fast and tremendous evolution of the unmanned aerial vehicle (UAV) imagery gives place to the multiplication of applications in various fields such as military and civilian surveillance, delivery services, and wildlife monitoring. Combining UAV imagery with study of dynamic salience further extends the number of future applications. Indeed, considerations of visual attention open the door to new avenues in a number of scientific fields such as compression, retargeting, and decision-making tools. To conduct saliency studies, we identified the need for new large-scale eye-tracking datasets for visual salience in UAV content. Therefore, we address this need by introducing the dataset EyeTrackUAV2. It consists of the collection of precise binocular gaze information (1000 Hz) over 43 videos (RGB, 30 fps, 1280 × 720 or 720 × 480). Thirty participants observed stimuli under both free viewing and task conditions. Fixations and saccades were then computed with the dispersion-threshold identification (I-DT) algorithm, while gaze density maps were calculated by filtering eye positions with a Gaussian kernel. An analysis of collected gaze positions provides recommendations for visual salience ground-truth generation. It also sheds light upon variations of saliency biases in UAV videos when opposed to conventional content, especially regarding the center bias.

Mixed Road User Trajectory Extraction From Moving Aerial Videos Based on Convolution Neural Network Detection

Vehicle trajectory data under mixed traffic conditions provides critical information for urban traffic flow modeling and analysis. Recently, the application of unmanned aerial vehicles (UAV) creates a potential of reducing traffic video collection cost and enhances flexibility at the spatial-temporal coverage, supporting trajectory extraction in diverse environments. However, accurate vehicle detection is a challenge due to facts such as small vehicle size and inconspicuous object features in UAV videos. In addition, camera motion in UAV videos hardens the trajectory construction procedure. This research aims at proposing a novel framework for accurate vehicle trajectory construction from UAV videos under mixed traffic conditions. Firstly, a Convolution Neural Network (CNN)-based detection algorithm, named You Only Look Once (YOLO) v3, is applied to detect vehicles globally. Then an image registration method based on Shi-Tomasi corner detection is applied for camera motion compensation. Trajectory construction methods are proposed to obtain accurate vehicle trajectories based on data correlation and trajectory compensation. At last, the ensemble empirical mode decomposition (EEMD) is applied for trajectory data denoising. Our framework is tested on three aerial videos taken by an UAV on urban roads with one including intersection. The extracted vehicle trajectories are compared with manual counts. The results show that the proposed framework achieves an average Recall of 91.91% for motor vehicles, 81.98% for non-motorized vehicles and 78.13% for pedestrians in three videos.