Aerial Video Research Articles

Accelerated RANSAC for Accurate Image Registration in Aerial Video Surveillance

Compared with ground views and direct overhead views (for orbital satellites), aerial robotics allow for capturing videos from diverse viewpoints and scenes, thus, the content of aerial image is complex and changeable, and aerial video has complex inter-frame transforms stemming from the blend of camera motion, platform motion and jitter. In addition, poor quality and similar texture are common in long-distance and large-scale aerial video surveillance. All of these interferences make image registration of aerial video difficult. This article puts forward one image registration method suited to aerial video on the basis of the hypothesize-and-verify of RANSAC. The proposed accelerated RANSAC, named PSSC-RANSAC (Prior Sampling & Sample Check RANSAC), incorporates prior sampling, which comes from three levels of sample evaluation, including texture magnitude, spatial consistency and feature similarity, to generate more possibly correct samples in priority. Furthermore, prior information of sample, quality of sample subset and subset invariability are together used to check the sample subsets, and the incompatible arrangements of subsets are immediately ruled out in sample check stage, which speeds up the iteration further. Results of the experiment have proved the good performance of the presented PSSC-RANSAC at 90% contamination level. For typical image pairs, the number of iterations is reduced by at least 16.67% and evaluation computation is reduced by at least 11.01% compared with SVH-RANSAC, and the re-projection error is decreased by at least 4.44% and 6.31% compared with RANSAC and SVH-RANSAC, respectively. It can overcome the interferences, and is very suitable for image registration of aerial images.

UVid-Net: Enhanced Semantic Segmentation of UAV Aerial Videos by Embedding Temporal Information

Semantic segmentation of aerial videos has been extensively used for decision making in monitoring environmental changes, urban planning, and disaster management. The reliability of these decision support systems is dependent on the accuracy of the video semantic segmentation algorithms. The existing CNN based video semantic segmentation methods have enhanced the image semantic segmentation methods by incorporating an additional module such as LSTM or optical flow for computing temporal dynamics of the video which is a computational overhead. The proposed research work modifies the CNN architecture by incorporating temporal information to improve the efficiency of video semantic segmentation. In this work, an enhanced encoder-decoder based CNN architecture (UVid-Net) is proposed for UAV video semantic segmentation. The encoder of the proposed architecture embeds temporal information for temporally consistent labelling. The decoder is enhanced by introducing the feature-refiner module, which aids in accurate localization of the class labels. The proposed UVid-Net architecture for UAV video semantic segmentation is quantitatively evaluated on extended ManipalUAVid dataset. The performance metric mIoU of 0.79 has been observed which is significantly greater than the other state-of-the-art algorithms. Further, the proposed work produced promising results even for the pre-trained model of UVid-Net on urban street scene with fine tuning the final layer on UAV aerial videos.

Thermal Infrared Video Stabilization for Aerial Monitoring of Active Wildfires

Measuring wildland fire behavior is essential for fire science and fire management. Aerial thermal infrared (TIR) imaging provides outstanding opportunities to acquire such information remotely. Variables such as fire rate of spread (ROS), fire radiative power (FRP), and fireline intensity may be measured explicitly both in time and space, providing the necessary data to study the response of fire behavior to weather, vegetation, topography, and firefighting efforts. However, raw TIR imagery acquired by unmanned aerial vehicles (UAVs) requires stabilization and georeferencing before any other processing can be performed. Aerial video usually suffers from instabilities produced by sensor movement. This problem is especially acute near an active wildfire due to fire-generated turbulence. Furthermore, the nature of fire TIR video presents some specific challenges that hinder robust interframe registration. Therefore, this article presents a software-based video stabilization algorithm specifically designed for TIR imagery of forest fires. After a comparative analysis of existing image registration algorithms, the KAZE feature-matching method was selected and accompanied by pre- and postprocessing modules. These included foreground histogram equalization and a multireference framework designed to increase the algorithm's robustness in the presence of missing or faulty frames. The performance of the proposed algorithm was validated in a total of nine video sequences acquired during field fire experiments. The proposed algorithm yielded a registration accuracy between 10 and 1000× higher than other tested methods, returned 10× more meaningful feature matches, and proved robust in the presence of faulty video frames. The ability to automatically cancel camera movement for every frame in a video sequence solves a key limitation in data processing pipelines and opens the door to a number of systematic fire behavior experimental analyses. Moreover, a completely automated process supports the development of decision support tools that can operate in real time during an emergency.

Basic Study on Tourism Support by Aerial Video using 3DCG

Basic Study on Tourism Support by Aerial Video using 3DCG

YOLOv2 Deep Learning Model and GIS Based Algorithms for Vehicle Tracking

The latest advances in Deep Learning based methods and computational capabilities provide new opportunities for vehicle tracking. In this study, YOLOv2 (You Only Look Once—version 2) is used as an open source Convolutional Neural Network (CNN), to process high-resolution satellite images, in order to generate the spatio-temporal GIS (Geographic Information System) tracks of moving vehicles. At first step, YOLOv2 is trained with a set of images of 1024 × 1024 resolution from the VEDAI database. The model showed satisfactory results, with an accuracy of 91%, and then at second step, is used to process aerial images extracted from aerial video. The output vehicle bounding boxes have been processed and fed into the GIS based LinkTheDots algorithm, allowing vehicles identification and spatio-temporal tracks generation in GIS format.

Quality assessment of building a geoinformation system for automated man-made debris search based on aerial video data

Quality assessment of building a geoinformation system for automated man-made debris search based on aerial video data

A Novel Target-Aware Dual Matching and Compensatory Segmentation Tracker for Aerial Videos

Object tracking in aerial scenes is widely used in intelligent transportation and national defense security. Template-based matching trackers currently dominate tracking pattern due to their remarkable performance, but are confined to rectangular bounding box tracking and the lack of effective transformation models, which lead to their low position accuracy in aerial scenes of special difficulties, such as low target resolution, large field of view, and multiple angle changes. To address this issue, we develop a novel target-aware dual matching and compensatory segmentation (TMCS) tracker for aerial videos, which transforms visual object tracking into a video object segmentation problem. Because deep features of the offline trained network are less discriminative in representing these targets of arbitrary form, we develop a regression loss for selecting the most representative convolutional filters to learn salient features of aerial target. In contrast to general trackers, the proposed approach designs the dual match mechanism, which matches the target activation features with the foreground and background features of the target template to generate geometric invariant models, rather than offline remembering the appearance of the target. In addition, most tracking benchmarks require marking the object position as a rectangular bounding box. In this article, an online fast fitting method is proposed to interpret the output form of the target as a rotating rectangular bounding box that is more suitable for the target, which improves object location accuracy. Extensive results on the challenging UAV123 dataset show that our method achieves comparable performance, which basically meets the aerial target tracking accuracy and real-time requirements.

ERA: A Data Set and Deep Learning Benchmark for Event Recognition in Aerial Videos [Software and Data Sets

As a result of the increasing use of unmanned aerial vehicles (UAVs), large volumes of aerial videos have been produced. It is unrealistic for humans to screen such big data and understand the contents. Hence, methodological research on the automatic understanding of UAV videos is of paramount importance (Figure 1). In this article, we introduce a novel problem of event recognition in unconstrained aerial videos in the remote sensing community and present the large-scale, human-annotated Event Recognition in Aerial Videos (ERA) data set, consisting of 2,864 videos, each with a label from 25 different classes corresponding to an event unfolding for five seconds. All these videos are collected from YouTube. The ERA data set is designed to have significant intraclass variation and interclass similarity and captures dynamic events in various circumstances and at dramatically various scales. Moreover, to offer a benchmark for this task, we extensively validate existing deep networks. We expect that the ERA data set will facilitate further progress in automatic aerial video comprehension. The data set and trained models can be downloaded from https://lcmou.github.io/ERA_Dataset/.

Aerial Video Trackers Review.

Target tracking technology that is based on aerial videos is widely used in many fields; however, this technology has challenges, such as image jitter, target blur, high data dimensionality, and large changes in the target scale. In this paper, the research status of aerial video tracking and the characteristics, background complexity and tracking diversity of aerial video targets are summarized. Based on the findings, the key technologies that are related to tracking are elaborated according to the target type, number of targets and applicable scene system. The tracking algorithms are classified according to the type of target, and the target tracking algorithms that are based on deep learning are classified according to the network structure. Commonly used aerial photography datasets are described, and the accuracies of commonly used target tracking methods are evaluated in an aerial photography dataset, namely, UAV123, and a long-video dataset, namely, UAV20L. Potential problems are discussed, and possible future research directions and corresponding development trends in this field are analyzed and summarized.

Viewpoint and Scale Consistency Reinforcement for UAV Vehicle Re-Identification

This paper studies vehicle ReID in aerial videos taken by Unmanned Aerial Vehicles (UAVs). Compared with existing vehicle ReID tasks performed with fixed surveillance cameras, UAV vehicle ReID is still under-explored and could be more challenging, e.g., aerial videos have dynamic and complex backgrounds, different vehicles show similar appearance, and the same vehicle commonly show distinct viewpoints and scales. To facilitate the research on UAV vehicle ReID, this paper contributes a novel dataset called UAV-VeID. UAV-VeID contains 41,917 images of 4601 vehicles captured by UAVs, where each vehicle has multiple images taken from different viewpoints. UAV-VeID also includes a large-scale distractor set to encourage the research on efficient ReID schemes. Compared with existing vehicle ReID datasets, UAV-VeID exhibits substantial variances in viewpoints and scales of vehicles, thus requires more robust features. To alleviate the negative effects of those variances, this paper also proposes a viewpoint adversarial training strategy and a multi-scale consensus loss to promote the robustness and discriminative power of learned deep features. Extensive experiments on UAV-VeID show our approach outperforms recent vehicle ReID algorithms. Moreover, our method also achieves competitive performance compared with recent works on existing vehicle ReID datasets including VehicleID, VeRi-776 and VERI-Wild.

An Analytical Framework for Accurate Traffic Flow Parameter Calculation from UAV Aerial Videos

Unmanned Aerial Vehicles (UAVs) represent easy, affordable, and simple solutions for many tasks, including the collection of traffic data. The main aim of this study is to propose a new, low-cost framework for the determination of highly accurate traffic flow parameters. The proposed framework consists of four segments: terrain survey, image processing, vehicle detection, and collection of traffic flow parameters. The testing phase of the framework was done on the Zagreb bypass motorway. A significant part of this study is the integration of the state-of-the-art pre-trained Faster Region-based Convolutional Neural Network (Faster R-CNN) for vehicle detection. Moreover, the study includes detailed explanations about vehicle speed estimation based on the calculation of the Mean Absolute Percentage Error (MAPE). Faster R-CNN was pre-trained on Common Objects in COntext (COCO) images dataset, fine-tuned on 160 images, and tested on 40 images. A dual-frequency Global Navigation Satellite System (GNSS) receiver was used for the determination of spatial resolution. This approach to data collection enables extraction of trajectories for an individual vehicle, which consequently provides a method for microscopic traffic flow parameters in detail analysis. As an example, the trajectories of two vehicles were extracted and the comparison of the driver’s behavior was given by speed—time, speed—space, and space—time diagrams.

Early forest fire smoke detection based on aerial video

The early detection of forest fire smoke has more research value than the detection of forest fire flames. This article mainly introduces the content of early forest fire smoke detection based on background blur model and aerial video. First, the background image of the video is extracted by establishing a background model. Then, through wavelet transform, the energy value of the background composite image is compared with the energy value of the target composite image when smoke appears, and the presence of smoke is detected by threshold processing. Experimental results show that the average recognition rate of smoke is as high as 96.94%.

Observations of coastal aggregations of the broadnose sevengill shark (Notorynchus cepedianus) in Chilean waters.

The presence of four sharks was documented in coastal waters of Antofagasta (Chile) using an unmanned aerial video camera. Fishers took advantage of this aggregation to catch and sold three adult broadnose sevengill sharks Notorynchus cepedianus. Species identity was determined by using the cox1 gene. One additional video was later recorded 3000 km south of Antofagasta, and shows a large female interacting with a salmon farming facility. Shallow water records of N. cepedianus were previously undocumented in Chilean waters, yet historically have provided an opportunistic event to fishers in Chile.

Deep Multimodality Learning for UAV Video Aesthetic Quality Assessment

Despite the growing number of unmanned aerial vehicles (UAVs) and aerial videos, there is a paucity of studies focusing on the aesthetics of aerial videos that can provide valuable information for improving the aesthetic quality of aerial photography. In this article, we present a method of deep multimodality learning for UAV video aesthetic quality assessment. More specifically, a multistream framework is designed to exploit aesthetic attributes from multiple modalities, including spatial appearance, drone camera motion, and scene structure. A novel specially designed motion stream network is proposed for this new multistream framework. We construct a dataset with 6,000 UAV video shots captured by drone cameras. Our model can judge whether a UAV video was shot by professional photographers or amateurs together with the scene type classification. The experimental results reveal that our method outperforms the video classification methods and traditional SVM-based methods for video aesthetics. In addition, we present three application examples of UAV video grading, professional segment detection and aesthetic-based UAV path planning using the proposed method.

Determining Stingray Movement Patterns in a Wave-Swept Coastal Zone Using a Blimp for Continuous Aerial Video Surveillance

Stingrays play a key role in the regulation of nearshore ecosystems. However, their movement ecology in high-energy surf areas remains largely unknown due to the notorious difficulties in conducting research in these environments. Using a blimp as an aerial platform for video surveillance, we overcame some of the limitations of other tracking methods, such as the use of tags and drones. This novel technology offered near-continuous coverage to characterise the fine-scale movements of stingrays in a surf area in Kiama, Australia, without any invasive procedures. A total of 98 stingray tracks were recorded, providing 6 h 27 min of movement paths. The tracking data suggest that stingrays may use a depth gradient located in the sandflat area of the bay for orientating their movements and transiting between locations within their home range. Our research also indicates that stingray behaviour was influenced by diel periods and tidal states. We observed a higher stingray occurrence during the afternoon, potentially related to foraging and anti-predatory strategies. We also saw a reduced route fidelity during low tide, when the bathymetric reference was less accessible due to stranding risk. Considering the increasing threat of anthropogenic development to nearshore coastal environments, the identification of these patterns can better inform the management and mitigation of threats.

Complete and Accurate Indoor Scene Capturing and Reconstruction Using a Drone and a Robot

Completeness and accuracy are two important factors in image-based indoor scene 3D reconstruction. Thus, an efficient image capturing scheme that could completely cover the scene, and a robust reconstruction method that could accurately reconstruct the scene are required. To this end, in this article we propose a new pipeline for indoor scene capturing and reconstruction using a mini drone and a ground robot, which takes both capturing completeness and reconstruction accuracy into consideration. First, we use a mini drone to capture aerial video of the indoor scene, from which a 3D aerial map is reconstructed. Then, the robot moving path is planned and a set of ground-view reference images are synthesized from the aerial map. After that, the robot enters the scene and captures ground video autonomously while using the reference images to locate its position during the movement. Finally, the ground and aerial images, which are adaptively extracted from the captured videos, are merged to reconstruct a complete and accurate indoor scene model. Experimental results on two indoor scenes demonstrate the effectiveness and robustness of our proposed indoor scene capturing and reconstruction pipeline.

VEHICLE TRACKING AND SPEED ESTIMATION FROM UNMANNED AERIAL VIDEOS

Abstract. In this paper, a solution for vehicle speed estimation using unmanned aerial videos is described. First, convolutional neural networks and Kalman filtering using deep features are used for detecting and tracking vehicles. Then, a photogrammetric approach is developed for estimating the three-dimensional (3D) position of the tracked vehicles on the road, which allows determining their speed. No assumptions are made about either the 3D structure of the road (e.g., constraining it to be a planar surface) or the camera pose (e.g., restricting it to be stationary). Therefore, this solution applies to videos acquired by a moving unmanned aerial vehicle from complex road structures (e.g., multi-level highways). This solution is also robust to changes of viewpoint and scale, which makes it applicable to situations where cars undergo orientation and resolution changes as observed from the sky (e.g., in roundabouts). Experiments showed that a high detection accuracy could be achieved with an F1-score of 94.54%. Besides, the tracking technique performed well, with a multiple-object tracking accuracy of 89.8% at a speed of 11 frames per second on videos of 2720×1530 pixels. Vehicle positioning (and thus, speed estimation) could be performed with an average accuracy of 0.6 m.

High-Resolution Vehicle Trajectory Extraction and Denoising From Aerial Videos

In recent years, unmanned aerial vehicle (UAV) has become an increasingly popular tool for traffic monitoring and data collection on highways due to its advantage of low cost, high resolution, good flexibility, and wide spatial coverage. Extracting high-resolution vehicle trajectory data from aerial videos taken by a UAV flying over target highway segment becomes a critical research task for traffic flow modeling and analysis. This study aims at proposing a novel methodological framework for automatic and accurate vehicle trajectory extraction from aerial videos. The method starts by developing an ensemble detector to detect vehicles in the target region. Then, the kernelized correlation filter is applied to track vehicles fast and accurately. After that, a mapping algorithm is proposed to transform vehicle positions from the Cartesian coordinates in image to the Frenet coordinates to extract raw vehicle trajectories along the roadway curves. The data denoising is then performed using a wavelet transform to eliminate the biased vehicle trajectory positions. Our method is tested on two aerial videos taken on different urban expressway segments in both peak and non-peak hours on weekdays. The extracted vehicle trajectories are compared with manual calibrated data to testify the framework performance. The experimental results show that the proposed method successfully extracts vehicle trajectories with a high accuracy: the measurement error of Mean Squared Deviation is 2.301 m, the Root-mean-square deviation is 0.175 m, and the Pearson correlation coefficient is 0.999. The video and trajectory data in this study are publicly accessible for serving as benchmark at https://seutraffic.com.

Efficient vehicle detection and tracking strategy in aerial videos by employing morphological operations and feature points motion analysis

Real-time automatic detection and tracking of moving vehicles in videos acquired by airborne cameras is a challenging problem due to vehicle occlusion, camera movement, and high computational cost. This paper presents an efficient and robust real-time approach for automatic vehicle detection and tracking in aerial videos that employ both detections and tracking features to enhance the final decision. The use of Top-hat and Bottom-hat transformation aided by the morphological operation in the detection phase has been adopted. After detection, background regions are eliminated by motion feature points’ analysis of the obtained object regions using a combined technique between KLT tracker and K-means clustering. Obtained object features are clustered into separate objects based on their motion characteristic. Finally, an efficient connecting algorithm is introduced to assign the vehicle labels with their corresponding cluster trajectories. The proposed method was tested on videos taken in different scenarios. The experimental results showed that the recall, precision, and tracking accuracy of the proposed method were about 95.1 %, 97.5%, and 95.2%, respectively. The method also achieves a fast processing speed. Thus, the proposed approach has superior overall performance compared to newly published approaches.

Intersection traffic signal optimisation considering the impact of upstream curbside bus stops

Transit-oriented travel mode is considered as one of the most effective travel strategies for improving the level of travel service and reducing travel time and delay. Curbside bus stops are often built to support the operation of the transit system, which leads to new bottlenecks of road segments near such stops and exerts a negative impact on traffic flows of neighbouring intersections. The authors propose a traffic signal optimisation model with the objective to minimise total travel delay by considering the interaction of traffic flows between the intersections and upstream curbside bus stops. First, transits’ following and lane-changing behaviours are explored by analysing their trajectories collected by unmanned vehicle aerial video detectors. Second, transits’ occupancy time at curbside bus stops is statistically analysed by using automatic vehicle location data. They then use an equivalent-flow analysis method to quantify the impact caused by transits’ actions of lane-changing and moving into/out stop basins (including the waiting process). Such impact is integrated into the algorithm of intersection traffic signal optimisation. They finally validate the proposed signal optimisation model based on real-time traffic data collected from Nanjing, China. It is revealed that the improved model can reduce vehicle travel delay by 4.3% at the tested intersection.