Aerial Video Research Articles

Pedestrian Path Prediction for Autonomous Driving at Un-Signalized Crosswalk Using W/CDM and MSFM

Pedestrian trajectory prediction is essential for collision avoidance in autonomous driving, which can help autonomous vehicles have a better understanding of traffic environment and perform tasks such as risk assessment in advance. In this paper, pedestrian path prediction at a time horizon of 2s for autonomous driving is systematically investigated using waiting/crossing decision model (W/CDM) and modified social force model (MSFM), and the possible conflict between pedestrians and straight-going vehicles at an un-signalized crosswalk is focused on. First of all, a W/CDM is efficiently developed to judge pedestrians' waiting/crossing intentions when a straight-going vehicle is approaching. Then the humanoid micro-dynamic MSFM of pedestrians who have been judged to cross is characterized by taking into account the evasion with conflicting pedestrians, the collision avoidance with straight-going vehicles, and the reaction to crosswalk boundary. The influence of pedestrian heterogeneous characteristics is considered for the first time. Moreover, aerial video data of pedestrians and vehicles at an un-signalized crosswalk is collected and analyzed for model calibration. Maximum likelihood estimation (MLE) is proposed to calibrate the non-measurable parameters of the proposed models. Finally, the model validation is conducted with two cases by comparing with the existing methods. The result reveals that the integrated method (W/CDM-MSFM) outperforms the existing methods and accurately predicts the path of pedestrians, which can give us great confidence to use the current method to predict the path of pedestrian for autonomous driving with significant accuracy and highly improve pedestrian safety.

UAV Integration in Current Construction Safety Planning and Monitoring Processes: Case Study of a High-Rise Building Construction Project in Chile

The majority of construction fatalities and accidents in Chile occur in high-rise building construction projects that also suffer from an insufficient number of safety managers on-site. Unmanned aerial vehicles (UAVs) and their generated aerial visual contents have the potential to help the limited number of safety managers in such projects to quickly and properly inspect the inaccessible, hard-to-reach, or unsafe locations on the site and to enhance their safety assessment of those projects. In this study, a case study approach was adopted to investigate how UAV technology and their generated aerial visual contents might affect the current approach of conducting safety planning and monitoring in a high-rise building construction site in Chile with a limited number of safety managers. The case study involved three steps: (1) understanding the current safety planning and monitoring process in a high-rise construction project, (2) investigating how UAV-related tasks and generated visual contents could be integrated into the current process, and (3) assessment of how such UAV integration might affect the current safety planning and monitoring process. The outcome of the case study provided a detailed overview of the new steps required to integrate UAVs in the current safety planning and monitoring process and concluded that adoption of UAV technology had enhanced identification and assessment of hazards in the high-rise project. Hazards associated with unsafe acts and conditions at height (e.g., missing guardrails or safety nets around unprotected edges or openings and loose or unsecured material at height) were the most common types of hazards identified using the UAV in this case study. Safety managers in the project also rated aerial videos captured by the UAV as the most useful type of data for their safety planning and monitoring tasks. The UAV also significantly reduced the amount of time required for safety managers to conduct their site visit walkthroughs on the project site.

Homography vs similarity transformation in aerial mosaicking: which is the best at different altitudes?

Aerial image mosaicking of an area of interest is the process of combining multiple images, of an area with overlapping regions, into a single comprehensive view. In this process, image registration, i.e., the operation of geometric transformation to align and overlay two or more images of the same scene taken from different viewpoints, starting from their common parts, plays a key role in terms of artifacts reduction. In the current state-of-the-art, image registration of aerial images is usually performed through the use of the homography transformation. This occurs because these images are frequently acquired at high altitudes (more than 100 meters) and the homography has always provided excellent performance. The recent widespread of Unmanned Aerial Vehicles (UAVs) has enabled the development of several applications where mosaics are used as reference images for high precision tasks, including Detection, Recognition, and Identification (hereinafter DRI) of people and objects. These tasks need to acquire images at very low altitudes (below 50 meters), in which the homography tends to introduce artifacts during the registration process. Therefore, a different transformation able to limit how an image can be morphed, i.e., the similarity transformation, is necessary to perform the image registration, thus improving the overall accuracy of the obtained mosaics. In this paper, for the first time in literature, a comparison between the homography and similarity transformations is performed. In particular, the comparison is carried out by using three recently released public datasets, i.e., NPU Drone-Map, senseFly, and UAV Mosaicking and Change Detection (UMCD), containing challenging aerial video sequences acquired at high and low altitudes. The experimental tests have pointed out the direct relationship among best image transformation, UAV altitude, and spatial resolution, required to accomplish the DRI tasks reported above.

Detection and Localisation of Life Signs from the Air Using Image Registration and Spatio-Temporal Filtering

In search and rescue operations, it is crucial to rapidly identify those people who are alive from those who are not. If this information is known, emergency teams can prioritize their operations to save more lives. However, in some natural disasters the people may be lying on the ground covered with dust, debris, or ashes making them difficult to detect by video analysis that is tuned to human shapes. We present a novel method to estimate the locations of people from aerial video using image and signal processing designed to detect breathing movements. We have shown that this method can successfully detect clearly visible people and people who are fully occluded by debris. First, the aerial videos were stabilized using the key points of adjacent image frames. Next, the stabilized video was decomposed into tile videos and the temporal frequency bands of interest were motion magnified while the other frequencies were suppressed. Image differencing and temporal filtering were performed on each tile video to detect potential breathing signals. Finally, the detected frequencies were remapped to the image frame creating a life signs map that indicates possible human locations. The proposed method was validated with both aerial and ground recorded videos in a controlled environment. Based on the dataset, the results showed good reliability for aerial videos and no errors for ground recorded videos where the average precision measures for aerial videos and ground recorded videos were 0.913 and 1 respectively.

Image Similarity Metrics Suitable for Infrared Video Stabilization during Active Wildfire Monitoring: A Comparative Analysis

Aerial Thermal Infrared (TIR) imagery has demonstrated tremendous potential to monitor active forest fires and acquire detailed information about fire behavior. However, aerial video is usually unstable and requires inter-frame registration before further processing. Measurement of image misalignment is an essential operation for video stabilization. Misalignment can usually be estimated through image similarity, although image similarity metrics are also sensitive to other factors such as changes in the scene and lighting conditions. Therefore, this article presents a thorough analysis of image similarity measurement techniques useful for inter-frame registration in wildfire thermal video. Image similarity metrics most commonly and successfully employed in other fields were surveyed, adapted, benchmarked and compared. We investigated their response to different camera movement components as well as recording frequency and natural variations in fire, background and ambient conditions. The study was conducted in real video from six fire experimental scenarios, ranging from laboratory tests to large-scale controlled burns. Both Global and Local Sensitivity Analyses (GSA and LSA, respectively) were performed using state-of-the-art techniques. Based on the obtained results, two different similarity metrics are proposed to satisfy two different needs. A normalized version of Mutual Information is recommended as cost function during registration, whereas 2D correlation performed the best as quality control metric after registration. These results provide a sound basis for image alignment measurement and open the door to further developments in image registration, motion estimation and video stabilization for aerial monitoring of active wildland fires.

Visual Object Detection For Autonomous UAV Cinematography

The popularization of commercial, battery-powered, camera-equipped, Vertical Take-off and Landing (VTOL) Unmanned Aerial Vehicles (UAVs) during the past decade, has significantly affected aerial video capturing operations in varying domains. UAVs are affordable, agile and flexible, having the ability to access otherwise inaccessible spots. However, their limited resources burden computation cinematography techniques on operating with high accuracy and real-time speed on such devices. State-of-the-art object detectors and feature extractors are, thus, studied in this work, aiming to find a trade-off between performance and speed that will allow UAV exploitation for intelligent cinematography purposes. Experimental evaluation on three newly introduced datasets of rowing boats, cyclists and parkour athletes is performed and evidence is provided that even limited-resource autonomous UAVs can indeed be used for cinematography applications.

Çanakkale Province: Unmanned Air Vehicle Assisted Surface Survey

Archeology aims to reveal all sorts of details of the ruins of different periods. A survey conducted on a known archaeological settlement aims to determine the characteristics of this settlement (various architectural details etc.). Accurate measurements of the archaeological survey and excavation area are important for the continuity and sustainability of the surveys. Precise measurements of the archaeological survey sites and excavation areas are important for the sustainability of the surveys. In recent years, UAVs have been used extensively in archaeological research as well as in many disciplines. In this study, the Iron Age archaeological sites in Canakkale were firstly identified by surface surveys. Then, aerial videos and aerial photography were taken using UAV to identify the walls and the boundaries of the residential areas in the archaeological sites where the ruins had been identified. UAV images were processed in Agisoft Photoscan software and orthophotos with high spatial resolution were produced. Residue plans of Asarlik Settlement in Ortaca Village, Asarlikkaya Fortress in Osmanlar Village, Damyeri Fortress in Damyeri Village, Madenkaya Fortress in Serceler Village, and Sivricetepe Fortress in Ortaca Village plans were drawn.

Small Object Bird Detection in Infrared Drone Videos Using Mask R-CNN Deep Learning

A reliable method to estimate population sizes of wild turkeys (Meleagris gallopavo) using unmanned aerial vehicles and thermal video imaging data collected at several field sites in Texas is described. Automating the data processing of airborne survey videos provides a fast and reproducible way to count wild turkeys for wildlife management and conservation. A deep learning semantic segmentation pipeline is developed to detect and count roosting Rio-Grande wild turkeys (M.g. intermedia) which appear as small faint objects in drone-based thermal IR videos. The proposed approach to detect roosting turkeys that appear as small objects, relies on Mask R-CNN, a deep architecture semantic segmentation algorithm. This is followed by a post-processing data association and filtering (DAF) process for counting the number of roosting birds. DAF was used to eliminate false positives like rocks and other small bright objects, which often have noisy detections across temporally adjacent video frames, that can be filtered using appearance association and distance-based gating across time. Transfer learning was used to train the Mask R-CNN network by initializing using ImageNet weights. Drone-based thermal IR videos are extremely challenging due to the complexity of the natural environment including weather effects, occlusion of birds, terrain, trees, complex tree shapes, rocks, water and thermal inversion. The transect videos were collected at night at several times and altitudes to optimize data collection opportunities without disturbing the roosting turkeys. Preliminary performance evaluation using 280 video frames is promising.

Lane-based vehicular speed characteristics analysis for freeway work zones using aerial videos

Work zones which are usually referred to as traffic bottlenecks or critical points may lead to reductions in both operational performance and safety. Researchers have conducted extensive studies on speed characteristics in freeway work zones. However, most of these studies are based on simulation data or are travel direction-based rather than lane-based. By utilizing the video recording capability of drones, this paper aims to provide methodologies and the workflow in exploring the vehicular speed characteristics in freeway work zones with considerations for lane and location deviations. A four-to-three lane work zone on Huhang Expressway between Downtown Shanghai and Songjiang in China was selected for data collection. Then free flow speed variation pattern is analyzed, a speed prediction model is proposed, and the speed–flow relationship is investigated on each lane at nine virtual sections along the work zone. It is found that the merging area affects the speeds mainly in the segment from 200 m upstream of the taper to the start of the activity area, which is the most important risk area in the work zone, and the speed characteristics depend on vehicle types (trucks versus automobiles) as well as lane locations. It is also showed that drones are valuable and effective tools for data collection for vehicular speed characteristics analysis in freeway work zones and it is easy to extract the information needed.

Context-aware MDNet for target tracking in UAV remote sensing videos

ABSTRACTWith the maturity and popularity of unmanned aerial vehicle (UAV) technology, remote sensing target tracking in the aerial videos from UAVs has drawn much attention in recent years. Given that UAV aerial video has low resolution, multiple similar disruptors and rapid perspective changes, and the main tracking methods in this research field generally have low tracking performance and timeliness, we propose a remote sensing target tracking method for UAV aerial video based on a context-aware multi-domain convolutional neural network (CAMD). First, in the design of the tracking network structure, we fuse multiple convolutional layers using residual connections to boost the effectiveness of regression learning. Then, in the offline pretraining stage, a Rotation Adversarial Autoencoders (RAAE) was adopted to generate typical easily confused negative samples for enhancing the capacity to distinguish between targets and the background interference. Finally, when the estimated target score was in the ‘fuzzy interval’, we introduced a response-adaptive context-aware correlation filter (RA-CACF) module into our network architecture to improve the tracking performance. Qualitative and quantitative evaluations using public and homemade hard datasets demonstrate that the proposed method can achieve high accuracy and efficiency results compared to state-of-the-art trackers.

Efficient and de-shadowing approach for multiple vehicle tracking in aerial video via image segmentation and local region matching

Multiple vehicle tracking in aerial video has vital applications in intelligent transportation system. However, this is challenging owing to the demand for higher tracking efficiency. Moreover, it is particularly challenging to handle shadows and occlusions in complex traffic scenes. Currently, most approaches mainly address the multiple vehicle tracking problem based on consecutive frame pairs. However, they are inefficient and ineffective with respect to the tracking of vehicles that are shadowed or occluded by trees, buildings, or large vehicles. Therefore, in terms of consecutive multiframe images, we propose an efficient deshadowing approach for multiple vehicle tracking in aerial video via image segmentation and local region matching. By processing a series of frames at a time, this approach will address the occurrence of shadows and occlusions in multiple vehicle tracking. First, image segmentation and road segmentation are combined to construct the region adjacency graph within the road (RAGR) for each frame. Local region matching is then performed among the RAGRs based on the distance transform. In particular, to improve the region matching efficiency, a local matching criterion is proposed by introducing region of interest for the node of the RAGR. Finally, based on the motion dissimilarities and varying spatial distances of the background and moving vehicles, a batch-oriented foreground separation is achieved to separate vehicles (the foreground) from the background. Experiments and comparative analysis conducted on UAV123 and DARPA VIVID data sets demonstrate that the proposed approach for multiple vehicle tracking in aerial video can generate satisfactory and competitive results.

Mixed Road User Trajectory Extraction From Moving Aerial Videos Based on Convolution Neural Network Detection

Vehicle trajectory data under mixed traffic conditions provides critical information for urban traffic flow modeling and analysis. Recently, the application of unmanned aerial vehicles (UAV) creates a potential of reducing traffic video collection cost and enhances flexibility at the spatial-temporal coverage, supporting trajectory extraction in diverse environments. However, accurate vehicle detection is a challenge due to facts such as small vehicle size and inconspicuous object features in UAV videos. In addition, camera motion in UAV videos hardens the trajectory construction procedure. This research aims at proposing a novel framework for accurate vehicle trajectory construction from UAV videos under mixed traffic conditions. Firstly, a Convolution Neural Network (CNN)-based detection algorithm, named You Only Look Once (YOLO) v3, is applied to detect vehicles globally. Then an image registration method based on Shi-Tomasi corner detection is applied for camera motion compensation. Trajectory construction methods are proposed to obtain accurate vehicle trajectories based on data correlation and trajectory compensation. At last, the ensemble empirical mode decomposition (EEMD) is applied for trajectory data denoising. Our framework is tested on three aerial videos taken by an UAV on urban roads with one including intersection. The extracted vehicle trajectories are compared with manual counts. The results show that the proposed framework achieves an average Recall of 91.91% for motor vehicles, 81.98% for non-motorized vehicles and 78.13% for pedestrians in three videos.

Spatio-Temporal Processing for Automatic Vehicle Detection in Wide-Area Aerial Video

Vehicle detection in aerial videos often requires post-processing to eliminate false detections. This paper presents a spatio-temporal processing scheme to improve automatic vehicle detection performance by replacing the thresholding step of existing detection algorithms with multi-neighborhood hysteresis thresholding for foreground pixel classification. The proposed scheme also performs spatial post-processing, which includes morphological opening and closing to shape and prune the detected objects, and temporal post-processing to further reduce false detections. We evaluate the performance of the proposed spatial processing on two local aerial video datasets and one parking vehicle dataset, and the performance of the proposed spatio-temporal processing scheme on five local aerial video datasets and one public dataset. Experimental evaluation shows that the proposed schemes improve vehicle detection performance for each of the nine algorithms when evaluated on seven datasets. Overall, the use of the proposed spatio-temporal processing scheme improves average F-score to above 0.8 and achieves an average reduction of 83.8% in false positives.

Model-Guided Multi-Path Knowledge Aggregation for Aerial Saliency Prediction

As an emerging vision platform, a drone can look from many abnormal viewpoints which brings many new challenges into the classic vision task of video saliency prediction. To investigate these challenges, this paper proposes a large-scale video dataset for aerial saliency prediction, which consists of ground-truth salient object regions of 1,000 aerial videos, annotated by 24 subjects. To the best of our knowledge, it is the first large-scale video dataset that focuses on visual saliency prediction on drones. Based on this dataset, we propose a Model-guided Multi-path Network (MM-Net) that serves as a baseline model for aerial video saliency prediction. Inspired by the annotation process in eye-tracking experiments, MM-Net adopts multiple information paths, each of which is initialized under the guidance of a classic saliency model. After that, the visual saliency knowledge encoded in the most representative paths is selected and aggregated to improve the capability of MM-Net in predicting spatial saliency in aerial scenarios. Finally, these spatial predictions are adaptively combined with the temporal saliency predictions via a spatiotemporal optimization algorithm. Experimental results show that MM-Net outperforms ten state-of-the-art models in predicting aerial video saliency.

Analysis of Affine Motion-Compensated Prediction in Video Coding

Motion-compensated prediction is used in video coding standards like High Efficiency Video Coding (HEVC) as one key element of data compression. Commonly, a purely translational motion model is employed. In order to also cover non-translational motion types like rotation or scaling (zoom), e. g. contained in aerial video sequences such as captured from unmanned aerial vehicles (UAV), an affine motion model can be applied. In this work, a model for affine motion-compensated prediction in video coding is derived. Using the rate-distortion theory and the displacement estimation error caused by inaccurate affine motion parameter estimation, the minimum required bit rate for encoding the prediction error is determined. In this model, the affine transformation parameters are assumed to be affected by statistically independent estimation errors, which all follow a zero-mean Gaussian distributed probability density function (pdf). The joint pdf of the estimation errors is derived and transformed into the pdfof the location-dependent displacement estimation error in the image. The latter is related to the minimum required bit rate for encoding the prediction error. Similar to the derivations of the fully affine motion model, a four-parameter simplified affine model is investigated. Both models are of particular interest since they are considered for the upcoming video coding standard Versatile Video Coding (VVC) succeeding HEVC. Both models provide valuable information about the minimum bit rate for encoding the prediction error as a function of affine estimation accuracies.

A COMPARATIVE STUDY OF THREE CORNER FEATURE BASED MOVING OBJECT DETECTION USING AERIAL IMAGES

Corner feature based moving objects detection is an essential and fundamental research problem in the broader aspects of computer vision and pattern recognition research domain. Performance of various corner features based aerial types image processing specially for moving objects detection is still an unsolved issue due to up and down of performance which makes it difficult to choose the appropriate corner features for detection purpose. The core part mentioned in this research is to categorize significant corner characteristics of the objects using various corner features based detection methods in image extracted from aerial video. This research demonstrated three kinds of corner features, i.e. Moravec, Susan and Harris corners due to capability of these corner features to interpret high and low intensity various for aerial types of images. Standard datasets were used to evaluate each of the corner feature based detection. Based on comprehensive experimental analysis, Harris corner was observed performing efficiently comparing with Moravec and Susan corner based detection for both datasets considered by this research. Experimental results reveals the capacity of each corner characteristics based detection methodology in terms with the effectiveness using various performance metrics for moving object detection using aerial images.

High confidence detection for moving target in aerial video

The moving target detection and tracking in aerial video is a challenge task because of its moving background, smaller target sizes, lower resolution and limited onboard computing resources. In this study, a high confidence detection method based on background compensation and three-frame-difference method is designed, which can detect moving objects in a dynamic background accurately. First, the authors use local feature extraction and matching for image registration and demonstrate that speed-up robust feature key points are suitable for the stabilisation task. Then, they estimate the global camera motion parameters using affine transformation which are obtained by the random sample consensus algorithm. Finally, they detect moving object by three-frame-difference method. As the detection results of the frame-difference method generally exists ‘empty’ and noise, in order to select the two higher-quality differential images to perform the logic AND operation, they add image quality assessment to the three-frame-difference method to obtain more accurate moving objects. Moreover, the edge detection algorithm and morphological processing are integrated together to further boost the overall detecting performance. The extensive empirical evaluations on aerial videos demonstrate that the proposed detector is very promising for the various challenging scenarios.

Unmanned Aerial Vehicles: Legal Features of Aerial Video

The purpose of the study is to systematize the requirements of regulatory documents of the United States of America, the countries of the European Union and the Russian Federation regulating the procedure for aerial video recording using UAVs. As a result of the study, information that allowed UAV owners to quickly resolve issues related to obtaining permission to fly, observing the established procedure for using airspace, fulfilling requirements to ensure data confidentiality and nondisclosure of state secrets, UAV registration and certification of an external pilot activity was presented. The efficiency of resolving issues is ensured by the fact that the main provisions of regulatory documents are supplemented by links to relevant resources, which allows obtaining more detailed information depending on the specific situation. The procedure for implementing practical actions to fulfill the requirements of legal acts is also set out. The novelty of the work lies in the way of presenting the requirements of regulatory documents, ensuring their familiarization by the UAV owners in a minimum time and in the amount determined by the specifics of performing aerial video recording in each specific case.

Drone-Action: An Outdoor Recorded Drone Video Dataset for Action Recognition

Aerial human action recognition is an emerging topic in drone applications. Commercial drone platforms capable of detecting basic human actions such as hand gestures have been developed. However, a limited number of aerial video datasets are available to support increased research into aerial human action analysis. Most of the datasets are confined to indoor scenes or object tracking and many outdoor datasets do not have sufficient human body details to apply state-of-the-art machine learning techniques. To fill this gap and enable research in wider application areas, we present an action recognition dataset recorded in an outdoor setting. A free flying drone was used to record 13 dynamic human actions. The dataset contains 240 high-definition video clips consisting of 66,919 frames. All of the videos were recorded from low-altitude and at low speed to capture the maximum human pose details with relatively high resolution. This dataset should be useful to many research areas, including action recognition, surveillance, situational awareness, and gait analysis. To test the dataset, we evaluated the dataset with a pose-based convolutional neural network (P-CNN) and high-level pose feature (HLPF) descriptors. The overall baseline action recognition accuracy calculated using P-CNN was 75.92%.

Convolutional neural networks for object detection in aerial imagery for disaster response and recovery

Accurate and timely access to data describing disaster impact and extent of damage is key to successful disaster management (a process that includes prevention, mitigation, preparedness, response, and recovery). Airborne data acquisition using helicopter and unmanned aerial vehicle (UAV) helps obtain a bird’s-eye view of disaster-affected areas. However, a major challenge to this approach is robustly processing a large amount of data to identify and map objects of interest on the ground in real-time. The current process is resource-intensive (must be carried out manually) and requires offline computing (through post-processing of aerial videos). This research introduces and evaluates a series of convolutional neural network (CNN) models for ground object detection from aerial views of disaster’s aftermath. These models are capable of recognizing critical ground assets including building roofs (both damaged and undamaged), vehicles, vegetation, debris, and flooded areas. The CNN models are trained on an in-house aerial video dataset (named Volan2018) that is created using web mining techniques. Volan2018 contains eight annotated aerial videos (65,580 frames) collected by drone or helicopter from eight different locations in various hurricanes that struck the United States in 2017–2018. Eight CNN models based on You-Only-Look-Once (YOLO) algorithm are trained by transfer learning, i.e., pre-trained on the COCO/VOC dataset and re-trained on Volan2018 dataset, and achieve 80.69% mAP for high altitude (helicopter footage) and 74.48% for low altitude (drone footage), respectively. This paper also presents a thorough investigation of the effect of camera altitude, data balance, and pre-trained weights on model performance, and finds that models trained and tested on videos taken from similar altitude outperform those trained and tested on videos taken from different altitudes. Moreover, the CNN model pre-trained on the VOC dataset and re-trained on balanced drone video yields the best result in significantly shorter training time.