Drone Video Research Articles

CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety-Oriented Research and Digital Twins

The development of safety-oriented research and applications requires fine-grain vehicle trajectories that not only have high accuracy, but also capture substantial safety-critical events. However, it would be challenging to satisfy both these requirements using the available vehicle trajectory datasets do not have the capacity to satisfy both. This paper introduces the CitySim dataset that has the core objective of facilitating safety-oriented research and applications. CitySim has vehicle trajectories extracted from 1,140-min of drone videos recorded at 12 locations. It covers a variety of road geometries including freeway basic segments, weaving segments, expressway merge/diverge segments, signalized intersections, stop-controlled intersections, and control-free intersections. CitySim was generated through a five-step procedure that ensured trajectory accuracy. The five-step procedure included video stabilization, object filtering, multivideo stitching, object detection and tracking, and enhanced error filtering. Furthermore, CitySim provides the rotated bounding box information of a vehicle, which was demonstrated to improve safety evaluations. Compared with other video-based trajectory datasets, CitySim had significantly more safety-critical events, including cut-in, merge, and diverge events, which were validated by distributions of both minimum time-to-collision and minimum post encroachment time. In addition, CitySim had the capability to facilitate digital-twin-related research by providing relevant assets, such as the recording locations’ three-dimensional base maps and signal timings.

Object Detection in Drone Video with Temporal Attention Gated Recurrent Unit Based on Transformer

Unmanned aerial vehicle (UAV) based object detection plays a pivotal role in civil and military fields. Unfortunately, the problem is more challenging than general visual object detection due to the significant appearance deterioration in images captured by drones. Considering that video contains more abundant visual features and motion information, a better idea for UAV based image object detection is to enhance target appearance in reference frame by aggregating the features in neighboring frames. However, simple feature aggregation methods will frequently introduce the interference of background into targets. To solve this problem, we proposed a more effective module, termed Temporal Attention Gated Recurrent Unit (TA-GRU), to extract effective temporal information based on recurrent neural networks and transformers. TA-GRU works as an add-on module to bring existing static object detectors to high performance video object detectors, with negligible extra computational cost. To validate the efficacy of our module, we selected YOLOv7 as baseline and carried out comprehensive experiments on the VisDrone2019-VID dataset. Our TA-GRU empowered YOLOv7 to not only boost the detection accuracy by 5.86% in the mean average precision (mAP) on the challenging VisDrone dataset, but also to reach a running speed of 24 frames per second (fps).

산불피해강도에 따른 이산화탄소 배출량 추정

In this study, carbon dioxide emission from the 2021 Andong forest fire was analyzed with burn severity, which was classified into high(crown fire), moderate(heat damage), and low(surface fire). Drone videos of the wildland fire area were taken using a multispectral sensor based on UAV and classified by grouping the differences in derived NDVI values between satellite images before the fire and UAV images immediately after the fire. Fuel load consumption and carbon dioxide emission from forest fires were estimated through field investigation of each burn severity site. According to the results of this study, the total burned areas were calculated to be 350ha, of which H(crown fire), M(heat damage), and L(surface fire) were to be 85ha, 125ha, and 141ha, respectively. 15.40ton/ha fire carbon dioxide in H(crown fire), 4.92ton/ha in M(heat damage), 4.69ton/ha in L(surface fire) were emitted from the fire. The total amount of carbon dioxide emitted by the Andong forest fire in 2021 was estimated to be 2,570 tons.

An enhanced noise-tolerant hashing for drone object detection

Drone, a.k.a. Unmanned aerial vehicle (UAV), has been pervasively applied in geological hazard monitoring, smart agriculture, and urban planning in the past decade. In this work, we fuse multiple attributes into a noise-tolerant hashing framework that can detect objects from drone pictures extremely fast. Our method can intrinsically and flexibly encode various topological structures from each target object, based on which multi-scale objects can be discovered in a view- and altitude-invariant way. Moreover, by leveraging lF and l1 norms collaboratively, the calculated hash codes are robust to low quality drone pictures and noisy semantic labels. More specifically, for each drone-borne picture, we extract visually/semantically salient object parts inside it. To characterize their topological structure, we construct a graphlet by linking the spatially adjacent object patches into a small graph. Subsequently, a binary matrix factorization (MF) is designed to hierarchically exploit the semantics of these graphlets, wherein three attributes: i) deep binary hash codes learning, ii) contaminated pictures/labels denoising, and iii) adaptive data graph updating are seamlessly incorporated. Such multi-attribute binary MF can be solved iteratively, and in turn each graphlet is transformed into the binary hash codes. Finally, the hash codes corresponding to graphlets within each drone photo are utilized for ranking-based object discovery. Comprehensive experiments on the DAC-SDC, MOHR, and our self-compiled data set have demonstrated the competitively speed and accuracy of our method. As a byproduct, we employ an elaborately-designed FPGA architecture to calculate our hash codes. On average, a 57 frames per second (fps) object detection speed is achieved on 4K drone videos (without temporal modeling).

Pedestrian walking speed monitoring at street scale by an in-flight drone.

The walking speed of pedestrians is not only a reflection of one's physiological condition and health status but also a key parameter in the evaluation of the service level of urban facilities and traffic engineering applications, which is important for urban design and planning. Currently, the three main ways to obtain walking speed are based on trails, wearable devices, and images. The first two cannot be popularized in larger open areas, while the image-based approach requires multiple cameras to cooperate in order to extract the walking speed of an entire street, which is costly. In this study, a method for extracting the pedestrian walking speed at a street scale from in-flight drone video is proposed. Pedestrians are detected and tracked by You Only Look Once version 5 (YOLOv5) and Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) algorithms in the video taken from a flying unmanned aerial vehicle (UAV). The distance that pedestrians traveled related to the ground per fixed time interval is calculated using a combined algorithm of Scale-Invariant Feature Transform (SIFT) and random sample consensus (RANSAC) followed by a geometric correction algorithm. Compared to ground truth values, it shows that 90.5% of the corrected walking speed predictions have an absolute error of less than 0.1 m/s. Overall, the method we have proposed is accurate and feasible. A particular advantage of this method is the ability to accurately predict the walking speed of pedestrians without keeping the flight speed of the UAV constant, facilitating accurate measurements by non-specialist technicians. In addition, because of the unrestricted flight range of the UAV, the method can be applied to the entire scale of the street, which assists in a better understanding of how the settings and layouts of urban affect people's behavior.

Research on Smart Construction Education Training Contents Using a Drone Simulator

Recently, application field of drone is expending throughout the growth of drone technology in the structural change of whole society which calls the 4th industrial Revolution. So, the construction field has been already using drone video shooting in construction design and efforts to introduce drone, which is one of smart construction technology, are also increasing even in facility maintenance management. It takes so much time and costs to train drone experts who are required in the smart construction field so training contents studies, which are for smart construction education, demand to acquire professional skills and expend education opportunities using a drone simulator. Therefore, we present real-world-images based on spatial information for realistic images using the flight dynamics modeling of a quad rotor drone and render a smart construction drone simulator for drone pilots education training to inspect facilities safety.

Development of an unmanned aerial vehicle for remote live streaming on web dashboard

As the prominence of autonomous aerial vehicles continues to rise globally, the ability of surveillance carried out remotely is likely to become a key aspect. In large spaces, random security checks are performed by security guards and are quite inefficient since on-foot monitoring is extremely non-productive and slow. This paper proposes a remote surveillance solution integrated into an aerial vehicle that performs random inspections and streams the live video on a web dashboard which is a faster and more reliable solution. The paper presents how to design and implement an Unmanned Aerial Vehicle (UAV) with surveillance capabilities wherein the live stream will be remotely accessible on any device connected to the same network. The main purpose of this drone is to transmit live data/video to a web dashboard and hence run inferences to analyse the data. A custom-made drone is integrated with Raspberry Pi with a Universal Serial Bus (USB) camera. An attempt to stream drone video feeds on local devices has been made in this field. The proposed framework operates in the robot operating system (ROS) integrated with the gazebo simulator and is designed to enable the software to communicate with the hardware. Two primary methodologies are utilised; simulations and experimental. The expected outcome should satisfy the need to view the drone’s live video feed on a web dashboard for remote surveillance and computer vision application. At the resolution of 640 × 480, the latency in the live stream varied from 1 to 3 s with 10 frame rates per second. The analysis of the flight logs of the drone suggests that it meets the desired stability required for smooth video streaming. The total estimate calculated to implement this system is Rs. 70,000 including contingencies and spare parts required for experimentation. The proposed prototype appears to be a viable solution for security solutions required by multiple large space owners on the study's findings.

Fish surveys on the move: Adapting automated fish detection and classification frameworks for videos on a remotely operated vehicle in shallow marine waters

Mobile underwater cameras, diver-operated or on underwater vehicles, have become popular for monitoring fisheries. Replacing divers with cameras has clear advantages, such as creating permanent records and accessing waters unavailable to divers. The use of cameras, however, typically produces large quantities of video that are time-consuming to process manually. Automated analysis of underwater videos from stationary cameras using deep learning techniques has advanced considerably in recent years, but the use of mobile cameras potentially raises new challenges for existing methods. We tested how well three automation procedures for stationary underwater cameras, taking an object-centric rather than background-centric approach, performed on surveys of fish using a mobile camera. We analyzed underwear drone videos from reef and seagrass habitat to detect and count two marine fisheries species, luderick (Girella tricuspidata) and yellowfin bream (Acanthopagrus australis). Three convolutional neural network (CNN) frameworks were compared: Detectron Faster R-CNN, Detectron2 Faster R-CNN (using a Regional Proposal Network, RPN), and YOLOv5 (a single-stage detector, SSD). Models performed well overall. Per frame, overall F1 scores ranged 81.4 - 87.3%, precision 88.2 – 96.0%, and recall 73.2 - 88.2%. For quantifying MaxN per video, overall F1 ranged 85.9 – 91.4%, precision 81.9 – 95.3%, and recall 87.1 – 91.1%. For luderick, F1 was > 80% for all frameworks per frame and 89% or higher for MaxN. For yellowfin bream, F1 scores were lower (35.0 - 73.8% for frames, 43.4 - 73.0% for MaxN). Detectron2 performed poorly, and YOLOv5 and Detectron performed similarly with advantages depending on metrics and species. For these two frameworks, performance was as good as in videos from stationary cameras. Our findings show that object detection technology is very useful for extracting fish data from mobile underwater cameras for the system tested here. There is a need now to test performance over a wider range of environments to produce generalizable models. The key steps required area to test and enhance performance: 1. for suites of species in the same habitats with different water clarity, 2. in other coastal environments, 3. trialing cameras moving at different speeds, and 4. using different frame-rates.

Using drone technology to collect school transportation data

Travel tallies are administered by elementary, middle, and high schools to collect data that measure how students arrive and leave school each day. In this study, a new data collection method using a quadcopter drone was examined. For comparison purposes, participatory student tallies and drone videos were collected on the same day, and the effectiveness of each method was then compared. The study concluded that the tally and drone results did not always yield similar modal results, so an explanation as to why these deviations occurred and what it means for researchers and practitioners is discussed.

The view from above: Examining how drone video ads elicit favorable viewer responses

ABSTRACT Marketers are increasingly adopting drone videos to advertise their products and services in a novel and engaging manner. Despite this increasing adoption, academic research on the use of drone videos for product advertising is scarce. Particularly lacking is evidence of the efficacy of drone video ads in eliciting favorable viewer responses. This study seeks to fill this gap in the literature by developing a model in which exposure to a drone video ad enhances viewers’ perceived presence and perceived novelty of the drone video ad. In turn, viewers’ perceived presence and perceived novelty of the drone video ad are hypothesized to contribute to a favorable attitude toward the brand and greater intention to share the drone video ad on social media through a favorable attitude toward the drone video ad. To test the proposed model, a sample of viewers were directed to watch a drone video ad and answered questions about the drone video ad, the advertised brand, and their intention to share the drone video ad on social media. The findings contribute to the literature by documenting the effectiveness of drone video ads in eliciting favorable viewer responses and elucidating the theoretical mechanisms by which these responses are elicited.

A Drone Video Clip Dataset and its Applications in Automated Cinematography

AbstractDrones became popular video capturing tools. Drone videos in the wild are first captured and then edited by humans to contain aesthetically pleasing camera motions and scenes. Therefore, edited drone videos have extremely useful information for cinematography and for applications such as camera path planning to capture aesthetically pleasing shots. To design intelligent camera path planners, learning drone camera motions from these edited videos is essential. However, first, this requires to filter drone clips and extract their camera motions out of these edited videos that commonly contain both drone and non‐drone content. Moreover, existing video search engines return the whole edited video as a semantic search result and cannot return only drone clips inside an edited video. To address this problem, we proposed the first approach that can automatically retrieve drone clips from an unlabeled video collection using high‐level search queries, such as “drone clips captured outdoor in daytime from rural places”. The retrieved clips also contain camera motions, camera view, and 3D reconstruction of a scene that can help develop intelligent camera path planners. To train our approach, we needed numerous examples of edited drone videos. To this end, we introduced the first large‐scale dataset composed of edited drone videos. This dataset is also used for training and validating our drone video filtering algorithm. Both quantitative and qualitative evaluations have confirmed the validity of our method.

DVS: A Drone Video Synopsis towards Storing and Analyzing Drone Surveillance Data in Smart Cities

The commercialization and advancement of unmanned aerial vehicles (UAVs) have increased in the past decades for surveillance. UAVs use gimbal cameras and LIDAR technology for monitoring as they are resource-constrained devices that are composed of limited storage, battery power, and computing capacity. Thus, the UAV’s surveillance camera and LIDAR data must be analyzed, extracted, and stored efficiently. Video synopsis is an efficient methodology that deals with shifting foreground objects in time and domain space, thus creating a condensed video for analysis and storage. However, traditional video synopsis methodologies are not applicable for making an abnormal behavior synopsis (e.g., creating a synopsis only of the abnormal person carrying a revolver). To mitigate this problem, we proposed an early fusion-based video synopsis. There is a drastic difference between the proposed and the existing synopsis methods as it has several pressing characteristics. Initially, we fused the 2D camera and 3D LIDAR point cloud data; Secondly, we performed abnormal object detection using a customized detector on the merged data and finally extracted only the meaningful data for creating a synopsis. We demonstrated satisfactory results while fusing, constructing the synopsis, and detecting the abnormal object; we achieved an mAP of 85.97%.

The effects of visual technology on price discovery for residential listings and transactions

PurposeThe purpose of this paper is to examine the effect of visual technology on the price discovery process in listings of residential properties in Singapore from 2015 to 2018.Design/methodology/approachThe authors empirically model the effects of 360 virtual tours and drone video on four dimensions in price discovery – buyers’ arrival rate, sale probability, transaction prices and time-on-market – using a comprehensive data set for the residential properties in Singapore.FindingsThe analysis shows that the availability of virtual tours or drone video in a listing increases the arrival rate from potential buyers, the probability of a successful sale and the selling price. These findings are consistent with the hypothesis that technologically enhanced tools improve the quality of information and the marketability of property. However, listings with virtual tours tend to be associated with longer marketing time, which is consistent with the prediction of the information overload hypothesis.Research limitations/implicationsThis paper extends the housing and price discovery literature by examining how technologically enabled new information affects property transactions.Originality/valueTo the best of the authors’ knowledge, this is the first paper to consider the impact of drone video on property market outcome.

Social organization and fitness response in grazing beef cows – Understanding through interactions and activity measuring

According to behavioural ecology theory, sociality evolves when the benefits of close association with conspecifics exceed the costs. Still, even when sociality evolves, the balance between costs and benefits differentiates amongst individuals in a group where each individual is selected to maximize its own fitness. In this study we aim to find correlations between fitness and social interactions and activity, and to test whether one of the prevailing social behaviour models, dominance hierarchy and self-organization, can describe social behaviour in a large group of grazing cows. Direct observations and drone video recordings were carried out in a herd of 130 ± 12 cows during two consecutive summers in a typical Mediterranean pasture in northern Israel. Observations' frequency was two to three weekly, lasting 4 ± 2 h each, during resting, grazing and feeding times. Agonistic interactions, allogrooming and grouping were recorded. Locomotion activity was monitored continuously by electronic pedometers. We calculated dominance probability, using the "Percolation and conductance" network approach whereby multiple indirect dominance pathways in the network are used to fill in missing data in the win/loss matrix. Dominance probability calculated was ambiguous (inconsistent dominance direction) with resulting average dominance index of 0.56 ± 0.029. Proxies for fitness were calving-to-conception interval and calf weaning weight. Calves were not considered as part of the group. Multiple regression analyses showed a positive effect of grouping and negative effect of oscillating grazing activity pattern on fitness response (p = 0.012; p = 0.01 for calving to conception interval and weaning weight respectively). Oscillating activity pattern was expressed in high Coefficient of Variance value of stepping activity. We suggest that in a large group of free grazing beef cows, social organization does not result from dominance hierarchy, and is better described by self-organizing systems hypothesis. Herd schedule and activity patterns were repetitive, but individuals' schedule within it was not. Individuals' morning grazing activity patterns were repetitive, but other daily hours' activity was not and individuals' nearest neighbours were changing continuously between daily activity sessions.

Drone Model Classification Using Convolutional Neural Network Trained on Synthetic Data.

We present a convolutional neural network (CNN) that identifies drone models in real-life videos. The neural network is trained on synthetic images and tested on a real-life dataset of drone videos. To create the training and validation datasets, we show a method of generating synthetic drone images. Domain randomization is used to vary the simulation parameters such as model textures, background images, and orientation. Three common drone models are classified: DJI Phantom, DJI Mavic, and DJI Inspire. To test the performance of the neural network model, Anti-UAV, a real-life dataset of flying drones is used. The proposed method reduces the time-cost associated with manually labelling drones, and we prove that it is transferable to real-life videos. The CNN achieves an overall accuracy of 92.4%, a precision of 88.8%, a recall of 88.6%, and an f1 score of 88.7% when tested on the real-life dataset.

Multiple Object Tracking of Drone Videos by a Temporal-Association Network with Separated-Tasks Structure

The task of multi-object tracking via deep learning methods for UAV videos has become an important research direction. However, with some current multiple object tracking methods, the relationship between object detection and tracking is not well handled, and decisions on how to make good use of temporal information can affect tracking performance as well. To improve the performance of multi-object tracking, this paper proposes an improved multiple object tracking model based on FairMOT. The proposed model contains a structure to separate the detection and ReID heads to decrease the influence between every function head. Additionally, we develop a temporal embedding structure to strengthen the representational ability of the model. By combing the temporal-association structure and separating different function heads, the model’s performance in object detection and tracking tasks is improved, which has been verified on the VisDrone2019 dataset. Compared with the original method, the proposed model improves MOTA by 4.9% and MOTP by 1.2% and has better tracking performance than the models such as SORT and HDHNet on the UAV video dataset.

Evaluation of unmanned aerial vehicles for surveys of lek‐mating grouse

AbstractUnmanned aerial vehicles (UAVs), or drones, are being utilized by wildlife biologists to monitor populations of birds and mammals. However, the reaction of wildlife to drones varies by species, so a preliminary evaluation must be conducted to determine if the target species can be detected via drone footage and to determine how the target species will react to drone presence. Lek‐mating prairie‐chickens (Tympanuchus spp.) seem amenable to drone‐based surveys because they are relatively large and display in groups on elevated sites with sparse vegetation. The goal of our study was to determine what flight characteristics maximize prairie‐chicken detection on drone video footage, to document the birds' reactions to the drones, and to compare how the noise produced by the drones compares to ambient sounds. We tested 3 sizes of rotary‐winged drones, flown at 3 heights, with 3 different camera angles by flying over known prairie‐chicken leks. We determined that >65% of prairie‐chickens present were detected using video footage at a height of 100 m with a 10° camera angle. Drones of different sizes had similar detectability. However, observers in blinds adjacent to leks routinely detected more birds than were detected on drone footage (detection on drone footage was 39.7 ± 9.6% of birds present). Prairie‐chickens flushed in 96% of trials when a drone was flown over display locations. Time to return to the lek following drone disturbance was similar to prairie‐chickens' reactions to natural predators. Prairie‐chickens flushed when drone sound levels were comparable to ambient noise. Developing guidelines for the ethical use of drones in wildlife research will require quantifying the amount of taxa‐specific disturbance caused by drones. Our study begins to close this knowledge gap by documenting disturbance to upland, lek‐mating grouse of conservation concern, and suggests that the focal species' reaction to aerial predators may provide clues as to how that species may react to drones.

Drone-based large-scale particle image velocimetry applied to tidal stream energy resource assessment

Resource quantification is vital in developing a tidal stream energy site but challenging in high energy areas. Drone-based large-scale particle image velocimetry (LSPIV) may provide a novel, low cost, low risk approach that improves spatial coverage compared to ADCP methods. For the first time, this study quantifies performance of the technique for tidal stream resource assessment, using three sites. Videos of the sea surface were captured while concurrent validation data were obtained (ADCP and surface drifters). Currents were estimated from the videos using LSPIV software. Variation in accuracy was attributed to wind, site geometry and current velocity. Root mean square errors (RMSEs) against drifters were 0.44 m s−1 for high winds (31 km/h) compared to 0.22 m s−1 for low winds (10 km/h). Better correlation was found for the more constrained site (r2 increased by 4%); differences between flood and ebb indicate the importance of upstream bathymetry in generating trackable surface features. Accuracy is better for higher velocities. A power law current profile approximation enables translation of surface current to currents at depth with satisfactory performance (RMSE = 0.32 m s−1 under low winds). Overall, drone video derived surface velocities are suitably accurate for “first-order” tidal resource assessments under favourable environmental conditions.

Aerial Observations and Numerical Simulations Confirm Density‐Driven Streamwise Vortices at a River Confluence

AbstractWhen rivers collide, complex three‐dimensional coherent flow structures are generated along the confluence's mixing interface. These structures mix streamborne pollutants and suspended sediment and have considerable bearing on the morphology and habitat quality of the postconfluent reach. A particular structure of interest—streamwise orientated vortices (SOVs)—were first detected in numerical simulations to form in pairs, one on each side of the mixing interface rotating in the opposite sense of the other. Since, it has proven difficult to detect SOVs in situ with conventional pointwise velocimetry instrumentation. Despite the lack of clear evidence to confirm their existence, SOVs are nevertheless considered important drivers of mixing and sediment transport processes at confluences. Additionally, their causal mechanisms are not fully known which hinders a complete conceptual understanding of these processes. To address these gaps, we analyze observations of strongly coherent SOVs filmed in aerial drone video of a mesoscale confluence with a stark turbidity contrast between its tributaries. Eddy‐resolved modeling demonstrates the SOVs' dynamics could only be accurately reproduced when a density difference (Δρ) was imposed between the tributaries (Δρ = 0.5 kg/m3)—providing compelling evidence the observed SOVs are indeed a novel density‐driven class of SOV. This work confirms that SOVs exist, expands understanding of their generative processes and highlights the important role of small density gradients (e.g., ≤0.5 kg/m3) on river confluence hydrodynamics.

Capitalizing on Drone Videos to Calibrate Simulation Models for Signalized Intersections and Roundabouts

Simulation is an indispensable tool for the assessment of highway-related capital investments and operational changes. Model calibration, a challenging task in any simulation study, is a crucial step. The model’s robustness, accuracy, and quality are directly dependent on it. Many parameters exist, and field observations are often lacking to aid in their correct specification. Recently, videos from drones have created a uniquely powerful way to aid this process. Observations of the inputs (demand), outputs (vehicles processed), processing rates (e.g., saturation flow rates), and performance results (times in system, queue dynamics, and delays) are all available simultaneously. For signalized intersections, only the signal timing events are missing, and those data can be obtained from signal timing logs. This paper illustrates how modeling teams can use drone data to calibrate model parameters pertaining to intersection operation. It shows how saturation flow rates can be adjusted for signalized intersections so that queue dynamics and delays can be matched. For roundabouts, it illustrates how critical gaps and move-up times can be adjusted to match field observations of performance. Three real-world settings with associated drone data are used as case study examples.