Aerial Video Research Articles

Low-Altitude Aerial Video Surveillance via One-Class SVM Anomaly Detection from Textural Features in UAV Images

In recent years, small-scale Unmanned Aerial Vehicles (UAVs) have been used in many video surveillance applications, such as vehicle tracking, border control, dangerous object detection, and many others. Anomaly detection can represent a prerequisite of many of these applications thanks to its ability to identify areas and/or objects of interest without knowing them a priori. In this paper, a One-Class Support Vector Machine (OC-SVM) anomaly detector based on customized Haralick textural features for aerial video surveillance at low-altitude is presented. The use of a One-Class SVM, which is notoriously a lightweight and fast classifier, enables the implementation of real-time systems even when these are embedded in low-computational small-scale UAVs. At the same time, the use of textural features allows a vision-based system to detect micro and macro structures of an analyzed surface, thus allowing the identification of small and large anomalies, respectively. The latter aspect plays a key role in aerial video surveillance at low-altitude, i.e., 6 to 15 m, where the detection of common items, e.g., cars, is as important as the detection of little and undefined objects, e.g., Improvised Explosive Devices (IEDs). Experiments obtained on the UAV Mosaicking and Change Detection (UMCD) dataset show the effectiveness of the proposed system in terms of accuracy, precision, recall, and F1-score, where the model achieves a 100% precision, i.e., never misses an anomaly, but at the expense of a reasonable trade-off in its recall, which still manages to reach up to a 71.23% score. Moreover, when compared to classical Haralick textural features, the model obtains significantly higher performances, i.e., ≈20% on all metrics, further demonstrating the approach effectiveness.

Video-based trajectory extraction with deep learning for High-Granularity Highway Simulation (HIGH-SIM)

High-granularity vehicle trajectory data can help researchers develop traffic simulation models, understand traffic flow characteristics, and thus propose insightful strategies for road traffic management. This paper proposes a novel vehicle trajectory extraction method that can extract high-granularity vehicle trajectories from aerial videos. The proposed method includes video calibration, vehicle detection and tracking, lane marking identification, and vehicle motion characteristics calculation. In particular, the authors propose a Monte-Carlo-based lane marking identification approach to identify each vehicle's lane. This is a challenging problem for vehicle trajectory extraction, especially when the aerial videos are taken from a high altitude. The authors applied the proposed method to extract vehicle trajectories from several high-resolution aerial videos recorded from helicopters. The extracted dataset is named by the High-Granularity Highway Simulation (HIGH-SIM) vehicle trajectory dataset. To demonstrate the effectiveness of the proposed method and understand the quality of the HIGH-SIM dataset, we compared the HIGH-SIM dataset with a well-known dataset, the NGSIM US-101 dataset, regarding the accuracy and consistency aspects. The comparison results showed that the HIGH-SIM dataset has more reasonable speed and acceleration distributions than the NGSIM US-101 dataset. Also, the internal and platoon consistencies of the HIGH-SIM dataset give lower errors compared to the NGSIM US-101 dataset. To benefit future research, the authors have published the HIGH-SIM dataset online for public use.

Comparative of Ziegler Nichols, Fuzzy Logic and Extremum Seeking Based Proportional Integral Derivative Controller for Quadcopter Unmanned Aerial Vehicle Stability Control

Unmanned aerial vehicle is potentially recognized in autonomous sectors where intelligence gathering, surveillance, reconnaissance missions, power line inspection, aerial video, search and rescue monitoring devices are required. It is essential in modern era control and monitoring especially a rotary unit where quadcopter performed a crucial task. However, the flight behavior of a quadcopter is determined by the synchronous speed of each of the motors as the speed changes with load torque variations. The dynamics model equation of the system, external disturbances and its parameters variation of the motor makes it difficult for the manual tuning techniques employed into the system to perform its stability operation. The purpose of this work is to employ adaptive controllers to enhance the stability performance so as to prevent the risk of human lives and financial implication that may arise from improper monitoring of the system. Therefore, Ziegler Nichols, fuzzy logic and extremum seeking controllers were employed to auto-tuned the parameters of proportional integral derivative (PID) gains controller to optimize and give a satisfactory performance of motor speed control at different operating condition. The altitude, pitch, roll and yaw parameters of the quadcopter are simulated using the x-plane II flight simulator MATLAB tools. The simulation results presented in this work show better performance for extremum seeking-PID in terms of decrease in rise time, settling time and overshoot relative to Zigler-Nichols-PID and Fuzzy-PID controllers.

A Self-Adaptive Method for Mapping Coastal Bathymetry On-The-Fly from Wave Field Video

Mapping coastal bathymetry from remote sensing becomes increasingly more attractive for the coastal community. It is facilitated by a rising availability of drone and satellite data, advances in data science, and an open-source mindset. Coastal bathymetry, but also wave directions, celerity and near-surface currents can simultaneously be derived from aerial video of a wave field. However, the required video processing is usually extensive, requires skilled supervision, and is tailored to a fieldsite. This study proposes a video-processing algorithm that resolves these issues. It automatically adapts to the video data and continuously returns mapping updates and thereby aims to make wave-based remote sensing more inclusive to the coastal community. The code architecture for the first time includes the dynamic mode decomposition (DMD) to reduce the data complexity of wavefield video. The DMD is paired with loss-functions to handle spectral noise and a novel spectral storage system and Kalman filter to achieve fast converging measurements. The algorithm is showcased for fieldsites in the USA, the UK, the Netherlands, and Australia. The performance with respect to mapping bathymetry was validated using ground truth data. It was demonstrated that merely 32 s of video footage is needed for a first mapping update with average depth errors of 0.9–2.6 m. These further reduced to 0.5–1.4 m as the videos continued and more mapping updates were returned. Simultaneously, coherent maps for wave direction and celerity were achieved as well as maps of local near-surface currents. The algorithm is capable of mapping the coastal parameters on-the-fly and thereby offers analysis of video feeds, such as from drones or operational camera installations. Hence, the innovative application of analysis techniques like the DMD enables both accurate and unprecedentedly fast coastal reconnaissance. The source code and data of this article are openly available.

Large‐scale turbulent mixing at a mesoscale confluence assessed through drone imagery and eddy‐resolved modelling

AbstractConfluences are sites of intense turbulent mixing in fluvial systems. The large‐scale turbulent structures largely responsible for this mixing have been proposed to fall into three main classes: vertically orientated (Kelvin–Helmholtz) vortices, secondary flow helical cells and smaller, strongly coherent streamwise‐orientated vortices. Little is known concerning the prevalence and causal mechanisms of each class, their interactions with one another and their respective contributions to mixing. Historically, mixing processes have largely been interpreted through statistical moments derived from sparse pointwise flow field and passive scalar transport measurements, causing the contribution of the instantaneous flow field to be largely overlooked. To overcome the limited spatiotemporal resolution of traditional methods, herein we analyse aerial video of large‐scale turbulent structures made visible by turbidity gradients present along the mixing interface of a mesoscale confluence and complement our findings with eddy‐resolved numerical modelling. The fast, shallow main channel (Mitis) separates over the crest of the scour hole's avalanche face prior to colliding with the slow, deep tributary (Neigette), resulting in a streamwise‐orientated separation cell in the lee of the avalanche face. Nascent large‐scale Kelvin–Helmholtz instabilities form along the collision zone and expand as the high‐momentum, separated near‐surface flow of the Mitis pushes into them. Simultaneously, the strong downwelling of the Mitis is accompanied by strong upwelling of the Neigette. The upwelling Neigette results in ∼50% of the Neigette's discharge crossing the mixing interface over the short collision zone. Helical cells were not observed at the confluence. However, the downwelling Mitis, upwelling Neigette and separation cell interact to generate considerable streamwise vorticity on the Mitis side of the mixing interface. This streamwise vorticity is strongly coupled to the large‐scale Kelvin–Helmholtz instabilities, which greatly enhances mixing. Comparably complex interactions between large‐scale Kelvin–Helmholtz instabilities and coherent streamwise vortices are expected at other typical asymmetric confluences exhibiting a pronounced scour hole.

ESPADA: Extended Synthetic and Photogrammetric Aerial-Image Dataset

We present a new aerial image dataset, named ESPADA, intended for the training of deep neural networks for depth image estimation from a single aerial image. Given the difficulty of creating aerial image datasets containing image pairs of chromatic images related to their depth images, simulators such as AirSim have been proposed to generate synthetic images from photorealistic scenes. The latter enables the generation of thousands of images that can be used to train and evaluate neural models. However, we argue that synthetic photorealistic aerial image datasets can be improved by adding images generated from photogrammetric models imported into the simulator, thus enabling a less artificial generation of both chromatic and depth images. To assess the quality of these images, we compare the performance of 4 deep neural networks whose pre-trained models and code for re-training are publicly available. We also use ORB-SLAM, in its RGB-D version, to indirectly assess the estimated depth image. To accomplish this, chromatic images from 3 aerial videos and their depth images, estimated with the networks trained with ESPADA, are fed into ORB-SLAM. The estimated camera pose is compared against the trajectory retrieved from the GPS flight trajectory. Our results indicate that images generated from photogrammetric models improve the performance of depth estimation from a single aerial image.

Robust Edge Computing in UAV Systems via Scalable Computing and Cooperative Computing

Unmanned aerial vehicle (UAV) systems are of increasing interest to academia and industry due to their mobility, flexibility, and maneuverability, and are an effective alternative to various uses such as surveillance and mobile edge computing. However, due to their limited computational and communications resources, it is difficult to serve all computation tasks simultaneously. This article tackles this problem by first proposing a scalable aerial computing solution, which is applicable for computation tasks of multiple quality levels, corresponding to different computation workloads and computation results of distinct performance. It opens up the possibility to maximally improve the overall computing performance with limited computational and communications resources. To meet the demands for timely video analysis that exceed the computing power of a UAV, we propose an aerial video streaming enabled cooperative computing solution, namely, UAVideo, which streams videos from a UAV to ground servers. As a complement to scalable aerial computing, UAVideo minimizes the video streaming time under the constraints on UAV trajectory, video features, and communications resources. Simulation results reveal the substantial advantages of the proposed solutions. Furthermore, we highlight relevant directions for future research.

Extracting water channels from aerial videos based on image-to-BIM registration and spatio-temporal continuity

Unmanned aerial vehicles (UAV) are increasingly being used in water supply infrastructure inspection, resulting in a large volume of aerial visual assets (static photographs or videos). How to efficiently extract water channels from such assets for the benefits of hazard detection remains a challenge. This paper provides an aerial video processing approach for the extraction of various water channel structures, which includes a main algorithm based on the registration of video frames into a building information model (BIM), and a complementary algorithm enabled by aerial video spatio-temporal continuity. Experimental results demonstrate the effectiveness of the proposed approach, which can reliably extract water channels with least manual intervention. The results reveal the promise of exploiting readily-available information from BIM to automate aerial video processing. Future research is suggested to further explore the generalizability of the approach concerning variant flight settings and external environments.

Leveraging social media and deep learning to detect rare megafauna in video surveys.

Deep learning has become a key tool for the automated monitoring of animal populations with video surveys. However, obtaining large numbers of images to train such models is a major challenge for rare and elusive species because field video surveys provide few sightings. We designed a method that takes advantage of videos accumulated on social media for training deep‐learning models to detect rare megafauna species in the field. We trained convolutional neural networks (CNNs) with social media images and tested them on images collected from field surveys. We applied our method to aerial video surveys of dugongs (Dugong dugon) in New Caledonia (southwestern Pacific). CNNs trained with 1303 social media images yielded 25% false positives and 38% false negatives when tested on independent field video surveys. Incorporating a small number of images from New Caledonia (equivalent to 12% of social media images) in the training data set resulted in a nearly 50% decrease in false negatives. Our results highlight how and the extent to which images collected on social media can offer a solid basis for training deep‐learning models for rare megafauna detection and that the incorporation of a few images from the study site further boosts detection accuracy. Our method provides a new generation of deep‐learning models that can be used to rapidly and accurately process field video surveys for the monitoring of rare megafauna.

DCT-Based Local Descriptor for Robust Matching and Feature Tracking in Wide Area Motion Imagery

We introduce a novel discrete cosine transform-based feature (DCTF) descriptor designed for both robustly matching features in aerial video and tracking features across wide-baseline oblique views in aerial wide area motion imagery (WAMI). Our DCTF descriptor preserves local structure more compactly in the frequency domain by utilizing the mathematical properties of the discrete cosine transform (DCT) and outperforms widely used the spatial-domain feature extraction methods, such as speeded up robust features (SURF) and scale-invariant feature transform (SIFT). The DCTF descriptor can be used in combination with other feature detectors, such as SURF and features from accelerated segment test (FAST), for which we provide experimental results. The performance of DCTF for image matching and feature tracking is evaluated on two city-scale aerial WAMI data sets (ABQ-215 and LA-351) and a synthetic aerial drone video data set digital imaging and remote sensing image generation (Rochester Institute of Technology (RIT)-DIRSIG). DCTF is a compact 120-D descriptor that is less than half the dimensionality of state-of-the-art deep learning-based approaches, such as SuperPoint, LF-Net, and DeepCompare, which requires no learning and is domain-independent. Despite its small size, the DCTF descriptor surprisingly produces the highest image matching accuracies ( F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> = 0.76 and ABQ-215), the longest maximum and average feature track lengths, and the lowest tracking error (0.3 pixel, LA-351) compared with both handcrafted and deep learning features.

CNN-SVM based vehicle detection for UAV platform

Conventional surveillance devices are deployed at fixed locations on road sideways, poles or on traffic lights, which provide a constant and fixed surveillance view of the urban traffic. Unmanned aerial vehicles (UAVs) have for last two decades received considerable attention in building smart and effective system with wider coverage using low cost, highly flexible unmanned platform for smart city infrastructure. Unlike fixed monitoring devices, the camera platform of aerial vehicles has many constraints, as it is in constant motion including titling and panning, and thus makes it difficult to process data for real time applications. The inaccuracy in object detection rates from UAV videos has motivated the research community to combine different approaches such as optical flow and supervised learning algorithms. The method proposed in this research incorporates steps that include Kanade-Lucas optical flow method for moving object detection, building connected graphs to isolate objects and convolutional neural network (CNN), followed by support vector machine (SVM) for final classification. The generated optical flow contains background (and tiny) objects detected as vehicle as the camera platform moves. The classifier introduced here rules out the presence of any other (moving) objects to be detected as vehicles. The methodology adopted is tested on a stationary and moving aerial videos. The system is shown to have performance accuracy of 100% in case of stationary video and 98% in case of video from aerial platform.

Using single and multiple unmanned aerial vehicles for microscopic driver behaviour data collection at freeway interchange ramps

This paper presents a detailed methodological framework for collecting microscopic driver and vehicle behaviour data over a long road segment with an application to the entire stretch of a freeway ramp segment using single and multiple unmanned aerial vehicles (UAVs). The methodology allows users to collect reliable and complete trajectories of traffic movements in areas with challenging physical characteristics (long road segment, horizontal curvature, changing elevation, and presence of shadow), challenging traffic characteristics (high traffic volume, high speeds, and high-speed changes), and restrictive regulations (UAVs prohibited from hovering over the freeway or the right-of-way). Different UAV setups are recommended and can be used depending on the site conditions. Specific commercial software and procedures used to complete the data collection are explained. The methodology was applied at two ramps and verified with speed data acquired from differential GPS receivers using three different error metrics. The results showed good performance of the proposed methodology, including when aerial videos were taken from oblique angles.

Human detection based on deep learning YOLO-v2 for real-time UAV applications

ABSTRACT Recent advancements in the field of Artificial Intelligence (AI) have provided an opportunity to create autonomous devices, robots, and machines characterised particularly with the ability to make decisions and perform tasks without human mediation. One of these devices, Unmanned Aerial Vehicles (UAVs) or drones are widely used to perform tasks like surveillance, search and rescue, object detection and target tracking, and many more. Efficient real-time object detection in aerial videos is an urgent need, especially with the increasing use of UAV in various fields. The sensitivity in performing said tasks demands that drones must be efficient and reliable. This paper presents our research progress in the development of applications for the identification and detection of person using the convolutional neural networks (CNN) YOLO-v2 based on the camera of drone. The position and state of the person are determined with deep-learning-based computer vision. The person detection results show that YOLO-v2 detects and classifies object with a high level of accuracy. For real-time tracking, the tracking algorithm responds faster than conventionally used approaches, efficiently tracking the detected person without losing it from sight.

Automatic Vehicle Counting and Tracking in Aerial Video Feeds using Cascade Region-based Convolutional Neural Networks and Feature Pyramid Networks

Unmanned aerial vehicles, or drones, are poised to solve many problems associated with data collection in complex urban environments. Drones are easy to deploy, have a great ability to move and explore the environment, and are relatively cheaper than other data collection methods. This study investigated the use of Cascade Region-based convolutional neural network (R-CNN) networks to enable automatic vehicle counting and tracking in aerial video streams.The presented technique combines feature pyramid networks and a Cascade R-CNN architecture to enable accurate detection and classification of vehicles.The paper discusses the implementation and evaluation of the detection and tracking techniques and highlights their advantages when they are used to collect traffic data.

A novel vehicle tracking and speed estimation with varying UAV altitude and video resolution

ABSTRACT In this paper, a novel vehicle tracking and speed estimation method based on aerial videos is developed. The main objective of this research is to investigate how depth features of a vehicle extracted from aerial videos by a wide residual network could be used to realize accurate tracking for estimating the speed of the vehicle. The salient characteristics of the proposed method are: (1) Traffic analysis and robust vehicle detection are possible by virtue of the use of deep neural networks. (2) Employing the state-of-the-art object detector, namely the You Only Look Once (YOLO) network, a tracking-by-detection method is then designed to detect and track vehicles. (3) The error rate of data association is reduced by associating the tracking process with a vehicle re-identification algorithm whereby the motion features are combined with depth features of the vehicle. (4) Considering the varying Unmanned Aerial Vehicle (UAV) altitude and video resolution, a novel exponential mapping model of the vehicle speed from the pixel space to the real world is presented. The accuracy of speed estimation is effectively improved by employing the least squares algorithm to fit the measurement data. Simulation results and comparative studies with the state-of-the-art methods demonstrate superior performance of the proposed approach for vehicle tracking and speed estimation.

Tiny moving vehicle detection in satellite video with constraints of multiple prior information

ABSTRACT Ｗith the rapid development of remote sensing, satellite video has become an important data source for vehicle detection, which provides a broader field of surveillance. The achieved work generally focuses on aerial video with moderately sized objects based on feature extraction. However, the moving vehicles in satellite video imagery range from just a few pixels to dozens of pixels and exhibit low contrast with respect to the background, which makes it hard to get available appearance or shape information this paper, a tiny vehicle detection method based on spatio-temporal information is proposed to constrain the significance of the image. Firstly, the background modelling method is used to obtain the motion heat map of the image and constrain the motion region. A significance detection method for small targets was used to obtain the significance mapping of these regions. Finally, the detection results were optimized by combining the significance neighbourhood information and the time information between frames to output the binary target detection map. Finally, taking different urban road scenes in ‘Jilin-1’satellite video as examples and compares a variety of existing algorithms. Experiments prove that the proposed algorithm can maintain false alarm rate of less than 10% when the detection accuracy and recall rate reach 85% and has certain anti-interference ability in the image environment with satellite Angle deviation.

Human action recognition in drone videos using a few aerial training examples

Drones are enabling new forms of human actions surveillance due to their low cost and fast mobility. However, using deep neural networks for automatic aerial action recognition is difficult due to the need for a large number of training aerial human action videos. Collecting a large number of human action aerial videos is costly, time-consuming, and difficult. In this paper, we explore two alternative data sources to improve aerial action classification when only a few training aerial examples are available. As a first data source, we resort to video games. We collect plenty of aerial game action videos using two gaming engines. For the second data source, we leverage conditional Wasserstein Generative Adversarial Networks to generate aerial features from ground videos. Given that both data sources have some limitations, e.g. game videos are biased towards specific actions categories (fighting, shooting, etc.,), and it is not easy to generate good discriminative GAN-generated features for all types of actions, we need to efficiently integrate two dataset sources with few available real aerial training videos. To address this challenge of the heterogeneous nature of the data, we propose to use a disjoint multitask learning framework. We feed the network with real and game, or real and GAN-generated data in an alternating fashion to obtain an improved action classifier. We validate the proposed approach on two aerial action datasets and demonstrate that features from aerial game videos and those generated from GAN can be extremely useful for an improved action recognition in real aerial videos when only a few real aerial training examples are available.

Comparison of spatiotemporal contribution patterns among three crowd-sourcing drone platforms

ABSTRACT Since the introduction of drones to the mass market for private users, drone pilots have started to share geo-tagged aerial photos and videos on a variety of drone platforms. This study compares the spatio-temporal contribution patterns of georeferenced drone-based images and videos between three crowd-sourcing platforms, namely Dronestagram, Travelwithdrone, and Flickr. The comparison addresses aspects of spatial accuracy, geographic coverage, contribution inequality, power-law approximations of different contribution characteristics, and negative binomial multilevel regression models that identify man-made and natural features, socio-economic factors, and land use categories that are associated with image and video contribution numbers. This study provides new insight into the abundance of drone-based image and video contributions around the globe, helps to determine which drone platform is best suited to find drone media for a specific location, and discusses a few potential applications that could benefit from crowd-sourced drone imagery and videos.

Evaluation of metrics and a dynamic thresholding strategy for high precision single sensor scene matching applications

Matching images of a scene with high precision is a core component in real time applications such as image-based navigation of unmanned aerial vehicles, terminal guidance of missiles. The aim of the research discussed in this paper is to develop a scene matching algorithm of high accuracy, robust to the variations between the images compared with scope for parallelism. After a study of state-of-the-art scene matching algorithms, a local feature-based approach was chosen to meet the stated requirements. The paper discusses the techniques for modules of the algorithm chosen and the methodology implemented after a metric study with a dynamic threshold based nearest neighbour distance ratio matching strategy (DT-NNDR). The metrics and the proposed strategy were tested for the matching of planar images from Affine covariant features group, unmanned aerial vehicle videos from a national defence agency, and satellite images from Google. The performance of strategy was compared with the existing NN-DR strategy with an experimental optimal threshold. The overall precision of the proposed strategy is 88%, making it suitable for applications which require high value of true positive.

Approach to tourism support by aerial video using CG

Approach to tourism support by aerial video using CG