UAV Video Research Articles

Optical flow estimation combining with illumination adjustment and edge refinement in livestock UAV videos

The performance of optical flow estimation is affected by many factors. The impact of illumination changes and video quality degradation in unmanned aerial vehicles videos on optical flow estimation cannot be ignored. Inspired by the human retina’s visual adaptation mechanism, we propose a mechanism for illumination adjustment that imitates retinal processing in order to reduce illumination variation. We further introduce an edge refinement mechanism into optical flow estimation that is based on a weighted neighborhood filtering. The experimental results on public benchmarks inlcuding KITTI 2012, KITTI 2015, MPI Sintel and Middlebury show that the proposed approach is robust on illumination and preserves accurate motion details. Further, experiments on outdoor livestock UAV videos show that the proposed approach implements illumination robustness and preserves the accurate detection of motion edges in other types of video. The performance of the proposed method on public benchmarks and livestock UAV videos demonstrates that the proposed approach improves motion edge accuracy of optical flow fields in varying illumination.

A TOOL TO ENHANCE THE CAPACITY FOR DEEP LEARNING BASED OBJECT DETECTION AND TRACKING WITH UAV DATA

Abstract. Currently, deployment of UAV has transformed from crucial to day-to-day scenarios for various purposes such as wastage collection, live entertainment, product delivery, town mapping, etc. Object tracking based UAV applications such as traffic monitoring, wildlife monitoring and surveillance have undergone phenomenal changeover due to deep learning based methodologies. With such transformation, there is also lack of resources to practically explore the UAV images and videos with deep learning methodologies. Hence, a deep learning-based object detection and tracking tool with UAV data (DL-ODT-UAV) is proposed to fill the learning gap, especially among students. DL-ODT-UAV is a resource to acquire basic knowledge about UAV and deep learning based object detection and tracking. It integrates various object annotators, object detectors and object tracker. Single object detection and tracking is performed with YOLO as object detector and LSTM as object tracker. Faster R-CNN is adopted in multiple object detection. With exploring the tool, the ability of students to approach problems related to deep learning methodologies will improve to a greater level.

Target tracking algorithm combined part-based and redetection for UAV

In the process of target tracking for UAV video images, the performance of the tracking algorithm declines or even the tracking fails due to target occlusion and scale variation. This paper proposes an improved target tracking algorithm based on the analysis of the tracking framework of the kernel correlation filter. First, four subblocks around the center of the target center are divided. A correlation filter fusing Histogram of Oriented Gradient (HOG) feature and Color Name (CN) feature tracks separately each target subblocks. According to the spatial structure characteristics in the subblocks, the center location and scale of the target are estimated. Secondly, the correct center location of target is determined by the global filter. Then, a tracking fault detection method is proposed. When tracking fails, the target redetection module which uses the normalized cross-correlation algorithm (NCC) to obtain the candidate target set in the re-detection area is started. Besides, this algorithm uses the global filter to obtain real target from the candidate set. In the meanwhile, this algorithm adjusts sectionally the learning rate of the classifiers according to detection results. Lastly, the performance of this algorithm is verified on the UAV123 dataset. The results show that compared with several mainstream methods, that of this algorithm is significantly improved when dealing with target scale variation and occlusion.

视频流分片编码的无人机视频传输方法

视频流分片编码的无人机视频传输方法

A QoE-Oriented Network Selection Mechanism in Heterogeneous Wireless Networks

In order to guarantee the UAV video uplink in heterogeneous networks, a novel access network selection mechanism was proposed in this paper. Due to the unknown Channel State Information(CSI), the issue was first modeled as the Multi-Armed Bandit (MAB) problem. Then, the reward function was built and the Quality of Experience (QoE) of video was considered. By means of three typical solving algorithms, decisions could be made between utilization and exploration. The simulation results show that the mechanism can effectively solve the network selection problem of video transmission in heterogeneous networks.

Characteristics and runout behaviour of the disastrous 28 August 2017 rock avalanche in Nayong, Guizhou, China

On 28th August 2017, a disastrous rock avalanche occurred in Nayong, Guizhou, China. This catastrophic event killed 35 people and destroyed 23 houses. In this study, the failure process of the avalanche was analyzed using UAV video footage and seismic signal recordings obtained for the pre- and post-failure stages of the landslide. To better understand the possible mechanism of the avalanche and its dynamic characteristics, a numerical simulation was conducted using a dynamic model (DAN3D) to simulate the runout behaviour of the displaced materials. Simulated results indicate that the failure process of the Nayong rock avalanche lasted approximately 30 s; the duration of the failure process is close to that from the UAV video footage and seismic signal recordings (35 s). The duration of movement during the landslide process is estimated to be approximately 90 s with a maximum velocity of 40 m/s. The combined frictional-Voellmy model delivered the most accurate results when simulating this event. The selected models and parameters are expected to aid in clarifying the propagation processes of similar rock avalanches and thus improve the accuracy of hazard prediction in mountainous karst areas.

无人机视频图像运动目标检测算法综述

无人机视频图像运动目标检测算法综述

Vehicle Instance Segmentation From Aerial Image and Video Using a Multitask Learning Residual Fully Convolutional Network

Object detection and semantic segmentation are two main themes in object retrieval from high-resolution remote sensing images, which have recently achieved remarkable performance by surfing the wave of deep learning and, more notably, convolutional neural networks (CNNs). In this paper, we are interested in a novel, more challenging problem of vehicle instance segmentation, which entails identifying, at a pixel-level, where the vehicles appear as well as associating each pixel with a physical instance of a vehicle. In contrast, vehicle detection and semantic segmentation each only concern one of the two. We propose to tackle this problem with a semantic boundary-aware multi-task learning network. More specifically, we utilize the philosophy of residual learning (ResNet) to construct a fully convolutional network that is capable of harnessing multi-level contextual feature representations learned from different residual blocks. We theoretically analyze and discuss why residual networks can produce better probability maps for pixel-wise segmentation tasks. Then, based on this network architecture, we propose a unified multi-task learning network that can simultaneously learn two complementary tasks, namely, segmenting vehicle regions and detecting semantic boundaries. The latter subproblem is helpful for differentiating closely spaced vehicles, which are usually not correctly separated into instances. Currently, datasets with pixel-wise annotation for vehicle extraction are ISPRS dataset and IEEE GRSS DFC2015 dataset over Zeebrugge, which specializes in semantic segmentation. Therefore, we built a new, more challenging dataset for vehicle instance segmentation, called the Busy Parking Lot UAV Video dataset, and we make our dataset available at http://www.sipeo.bgu.tum.de/download so that it can be used to benchmark future vehicle instance segmentation algorithms.

SPOT Poachers in Action: Augmenting Conservation Drones With Automatic Detection in Near Real Time

The unrelenting threat of poaching has led to increased development of new technologies to combat it. One such example is the use of long wave thermal infrared cameras mounted on unmanned aerial vehicles (UAVs or drones) to spot poachers at night and report them to park rangers before they are able to harm animals. However, monitoring the live video stream from these conservation UAVs all night is an arduous task. Therefore, we build SPOT (Systematic POacher deTector), a novel application that augments conservation drones with the ability to automatically detect poachers and animals in near real time. SPOT illustrates the feasibility of building upon state-of-the-art AI techniques, such as Faster RCNN, to address the challenges of automatically detecting animals and poachers in infrared images. This paper reports (i) the design and architecture of SPOT, (ii) a series of efforts towards more robust and faster processing to make SPOT usable in the field and provide detections in near real time, and (iii) evaluation of SPOT based on both historical videos and a real-world test run by the end users in the field. The promising results from the test in the field have led to a plan for larger-scale deployment in a national park in Botswana. While SPOT is developed for conservation drones, its design and novel techniques have wider application for automated detection from UAV videos.

Human motion analysis from UAV video

PurposeThe purpose of this paper is to present a preliminary solution to address the problem of estimating human pose and trajectory by an aerial robot with a monocular camera in near real time.Design/methodology/approachThe distinguishing feature of the solution is a dynamic classifier selection architecture. Each video frame is corrected for perspective using projective transformation. Then, a silhouette is extracted as a Histogram of Oriented Gradients (HOG). The HOG is then classified using a dynamic classifier. A class is defined as a pose-viewpoint pair, and a total of 64 classes are defined to represent a forward walking and turning gait sequence. The dynamic classifier consists of a Support Vector Machine (SVM) classifier C64 that recognizes all 64 classes, and 64 SVM classifiers that recognize four classes each – these four classes are chosen based on the temporal relationship between them, dictated by the gait sequence.FindingsThe solution provides three main advantages: first, classification is efficient due to dynamic selection (4-class vs 64-class classification). Second, classification errors are confined to neighbors of the true viewpoints. This means a wrongly estimated viewpoint is at most an adjacent viewpoint of the true viewpoint, enabling fast recovery from incorrect estimations. Third, the robust temporal relationship between poses is used to resolve the left-right ambiguities of human silhouettes.Originality/valueExperiments conducted on both fronto-parallel videos and aerial videos confirm that the solution can achieve accurate pose and trajectory estimation for these different kinds of videos. For example, the “walking on an 8-shaped path” data set (1,652 frames) can achieve the following estimation accuracies: 85 percent for viewpoints and 98.14 percent for poses.

Orientation Analysis between UAV Video and Photos for 3D Measurement of Bridges

Orientation Analysis between UAV Video and Photos for 3D Measurement of Bridges

Semantic regions segmentation using a spatio-temporal model from an UAV image sequence with an optimal configuration for data acquisition

ABSTRACTUnmanned aerial vehicles (UAV) are used to conduct a variety of recognition as well as specific missions such as target tracking and safe landing. The transmitted image sequences to be interpreted at ground station usually face limited requirements of the data transmission. In this paper, on the one hand, we handle a surveillance mission with segmenting an UAV video's content into semantic regions. A multi-class image segmentation approach is proposed considering UAV video-specific characteristics. After post-processing steps on the segmentation results, a support vector-machine classifier is used to recognize regions. A Markov model is introduced to combine the results from the previous frames in order to improve the accuracy. On the other hand, this study also assesses the influences of data reduction techniques on the proposed techniques. The comparisons between the untreated configuration and control conditions under manipulations of the frame rate, spatial resolution, and compression ratio, demonstrate how these data reduction techniques adversely influence the algorithm's performance. The experiments also point out the optimal configuration in order to obtain a trade-off between the target performance and requirements of the data transmission.

Multiple Moving Object Detection From UAV Videos Using Trajectories of Matched Regional Adjacency Graphs

Image registration has been long used as a basis for the detection of moving objects. Registration techniques attempt to discover correspondences between consecutive frame pairs based on image appearances under rigid and affine transformations. However, spatial information is often ignored, and different motions from multiple moving objects cannot be efficiently modeled. Moreover, image registration is not well suited to handle occlusion that can result in potential object misses. This paper proposes a novel approach to address these problems. First, segmented video frames from unmanned aerial vehicle captured video sequences are represented using region adjacency graphs of visual appearance and geometric properties. Correspondence matching (for visible and occluded regions) is then performed between graph sequences by using multigraph matching. After matching, region labeling is achieved by a proposed graph coloring algorithm which assigns a background or foreground label to the respective region. The intuition of the algorithm is that background scene and foreground moving objects exhibit different motion characteristics in a sequence, and hence, their spatial distances are expected to be varying with time. Experiments conducted on several DARPA VIVID video sequences as well as self-captured videos show that the proposed method is robust to unknown transformations, with significant improvements in overall precision and recall compared to existing works.

Wavelet-Based Optical Flow Estimation of Instant Surface Currents From Shore-Based and UAV Videos

Instant fields of surface current are retrieved from shore-based and unmanned aerial vehicle videos by an optical flow (OF) method named “Typhoon.” This computer vision algorithm estimates dense 2-D 2-component velocity fields from the observable motion of foam patterns in the surf zone. Despite challenging image data resolution and quality, comparison of OF surface current estimates with measurements by an acoustic Doppler velocimeter reveals its ability to capture both wave-to-wave fluctuations and low-frequency variations. The method is also successfully applied to the monitoring of a “flash rip” event. This paper shows clearly the high potential of this method in the nearshore, where the rapid development of webcams and drones offers a large number of applications for swimming and surfing safety, engineering and naval security, and research purpose, by providing quantitative information.

Maritime Vessels Real-time Tracking-by-detection in UAV Videos

Background/Objectives: This paper presents a fast real-time multi-object tracking algorithm adopted for maritime vessels tracking in UAV videos. Methods/Statistical Analysis: This research applies extended multi-object tracking-bydetection framework in highly dynamic UAV-captured maritime environment. Propagation is performed using particle filter whose particle weights are updated using modified observation model that incorporates a term based upon trackers affinity. The latter is also used for trackers grouping, that handles detector inaccuracies, and preventing identity switches by means of special propagation mode that is enabled when targets approach each other and start to overlap. Findings: Our research has shown that state-of-art multi-tracking algorithm is applicable to maritime vessels real-time tracking in UAV videos, provided the use of weak and fast online classifiers, which weakness is compensated by algorithm features based on trackers affinity matrix. Improvements/Applications: This paper presents an approach to maritime vessels tracking that allows to use fast, but less accurate, simple detectors and classifiers, enabling real-time processing on board of small-sized UAVs, while keeping decent precision and accuracy.Keywords: Maritime Vessels Tracking, Multi-object Tracking, Particle Filter, Online Learning, Trackers Affinity, Trackingby-detection, UAV Surveillance

LED Panel Detection and Pattern Discrimination Using UAV’s On-Board Camera for Autoflight Control

[abstFig src='/00280003/04.jpg' width=""250"" text='UAV autoflight control system using LED panels' ] The UAV flight control method we propose uses LED panels and a video camera on the UAV. Specifically, the LED panel displays patterns related to the UAV commands and blinking patterns for panel detection. A panel detection process based in UAV video camera images uses the frequency of green blinking as a cue for panel detection, then command patterns are distinguished and the UAV performs tasks based on this pattern (command). In experiments we performed for panel detection and discrimination using the UAV, we confirm the effectiveness of the proposed method for autoflight control.

Adaptive Prediction based Diamond Search Algorithm (APDS) for Video Encoder of Solar Powered UAV

Aerial Vehicle (UAV) provide bird's eye view over an intersection or a large area, and provide real-time surveillance of area under observation. UAVs have been playing a vital role in disaster management due to the increased sensing and processing capabilities. This paper proposes a fast adaptive prediction based diamond search Motion Estimation (ME) algorithm for Sun Falcon 2, a solar powered UAV's video encoder to cope the computational complexity, low power and increased quality of ME process requirement. Results show that the proposed Adaptive Predict Diamond Search (APDS) ME algorithm performs best in the term of PSNR, MSE and number of Search Points (SP), for approximately all the video sequences. Moreover, performance of APDS is decreased a little bit in term of number of SP when compared to Hexagon search algorithm but its PSNR is still considerably high for those video sequences. The average PSNR improvement rate of APDS is 0.62, 2.67, 0.82, 0.83 and 2.31 for Diamond Search (DS), HexBS, FHS, FSS and MDS respectively, while the average SIR is 25.4404, 6.3374, 48.274 and 205.55 for DS, FHS, FSS and MDS respectively.

Using Image Registration Method to Register UAV

For the UAV’s characteristics that is the date of camera shooting and photographing and digital map contain information, a UAV target location method is presented based on image matching. Firstly, using the artificial reconnaissance registration method that has improved, the UAV aerial images that consider digital map coordinates as the target are registered to obtain registration information. Then, the UAV video frame resolution containing the target is improved by improving the method of super-resolution reconstruction. Finally, making use of the Gaussian pyramid matching algorithm based on optical flow, the video frames are registered with the aerial to complete the target. By MATLAB and C++ simulation, achieving the above algorithm. The results proved that this method achieve the target location without the UAV position and attitude information, while improving the precision and speed of target location, that is an effective method for target location in the battlefield.

Statistically tuned Gaussian background subtraction technique for UAV videos

Background subtraction is one of the efficient techniques to segment the targets from non-informative background of a video. The traditional background subtraction technique suits for videos with static background whereas the video obtained from unmanned aerial vehicle has dynamic background. Here, we propose an algorithm with tuning factor and Gaussian update for surveillance videos that suits effectively for aerial videos. The tuning factor is optimized by extracting the statistical features of the input frames. With the optimized tuning factor and Gaussian update an adaptive Gaussian-based background subtraction technique is proposed. The algorithm involves modelling, update and subtraction phases. This running Gaussian average based background subtraction technique uses updation at both model generation phase and subtraction phase. The resultant video extracts the moving objects from the dynamic background. Sample videos of various properties such as cluttered background, small objects, moving background and multiple objects are considered for evaluation. The technique is statistically compared with frame differencing technique, temporal median method and mixture of Gaussian model and performance evaluation is done to check the effectiveness of the proposed technique after optimization for both static and dynamic videos.

Mosaic of UAV Aerial Video by Integrating Optical Flow Computation and Fourier-Mellin Transformation

UAV Video is rapidly emerging as a widely used source of imagery for many applications in recent years. This paper presents our research on the mosaic of UAV video for the purpose of harbor surveillance. First, one new framework on estimating video frame transformation with Optical flow is presented in this paper. For this new framework, fewer number of Gaussian pyramid is created for implementing for the multiresolution approach and thus, more details for optical flow computation is well kept; Second, we make a discussion on using Fourier-Mellin Transformation in image frequency domain to estimate initial motion parameter of adjacent video frames and with those initial motion parameters, small displacements for optical flow computation can be achieved; The experimental results demonstrated that the mosaic image generated from aerial video shows satisfied visual quality and its surveillance application for fast response to time-critical event, e.g., flood, is descried.