Raw Aerial Images Research Articles

Vision-Based HAR in UAV Videos Using Histograms and Deep Learning Techniques.

Activity recognition in unmanned aerial vehicle (UAV) surveillance is addressed in various computer vision applications such as image retrieval, pose estimation, object detection, object detection in videos, object detection in still images, object detection in video frames, face recognition, and video action recognition. In the UAV-based surveillance technology, video segments captured from aerial vehicles make it challenging to recognize and distinguish human behavior. In this research, to recognize a single and multi-human activity using aerial data, a hybrid model of histogram of oriented gradient (HOG), mask-regional convolutional neural network (Mask-RCNN), and bidirectional long short-term memory (Bi-LSTM) is employed. The HOG algorithm extracts patterns, Mask-RCNN extracts feature maps from the raw aerial image data, and the Bi-LSTM network exploits the temporal relationship between the frames for the underlying action in the scene. This Bi-LSTM network reduces the error rate to the greatest extent due to its bidirectional process. This novel architecture generates enhanced segmentation by utilizing the histogram gradient-based instance segmentation and improves the accuracy of classifying human activities using the Bi-LSTM approach. Experimental outcomes demonstrate that the proposed model outperforms the other state-of-the-art models and has achieved 99.25% accuracy on the YouTube-Aerial dataset.

Feasibility assessment of tree-level flower intensity quantification from UAV RGB imagery: A triennial study in an apple orchard

A timely and accurate spatial inventory of flowering characteristics benefits both the floral phenology monitoring in ecology and various crop management activities in agricultural systems. Recent advancement has proven the superiority of computer vision in flower classification at image level. Yet progress in the flowering intensity estimation at tree level is much less and still far from satisfactory. To tackle this problem, a novel approach was designed for the use of single raw aerial images to quantify flower intensity. With pre-prepared dataset, flower-associated pixels were extracted for individual trees using a pixel-based classification method, the color thresholding. Next, three flowering indices retrieved from unmanned aerial vehicle (UAV) were evaluated, the index percentage (IPG), index pixel (IP), and index area (IA). Finally, linear correlation of the flowering indices to flower cluster number and expert-assessed floridity recorded in the field were calculated. Results indicated that IPG yielded the highest correlation to flower cluster (R2 = 0.93, RMSE = 8) and floridity estimation (R2 = 0.78, RMSE = 0.9). A UAV-based floridity scoring method was also designed for automatic estimation tasks in practice, and a comparable and even better performance to the expert-based approach was demonstrated. Furthermore, effects of vertical (nadir) and horizontal (angular) overlapping of flower clusters within the canopy were evaluated, showing excellent potential to improve the estimation accuracy.

SCAF-Net: Scene Context Attention-Based Fusion Network for Vehicle Detection in Aerial Imagery

In recent years, deep learning methods have achieved great success for vehicle detection tasks in aerial imagery. However, most existing methods focus only on extracting latent vehicle target features, and rarely consider the scene context as vital prior knowledge. In this letter, we propose a scene context attention-based fusion network (SCAF-Net), to fuse the scene context of vehicles into an end-to-end vehicle detection network. First, we propose a novel strategy, patch cover, to keep the original target and scene context information in raw aerial images of a large scale as much as possible. Next, we use an improved YOLO-v3 network as one branch of SCAF-Net, to generate vehicle candidates on each patch. Here, a novel branch for the scene context is utilized to extract the latent scene context of vehicles on each patch without any extra annotations. Then, these two branches above are concatenated together as a fusion network, and we apply an attention-based model to further extract vehicle candidates of each local scene. Finally, all vehicle candidates of different patches, are merged by global nonmax suppress (g-NMS) to output the detection result of the whole original image. Experimental results demonstrate that our proposed method outperforms the comparison methods with both high detection accuracy and speed. Our code is released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/minghuicode/SCAF-Net</uri> .

Multilabel Aerial Image Classification With Unsupervised Domain Adaptation

Deep learning (DL) methods are promising for the multilabel aerial image classification (MAIC) task. However, current DL methods face a common problem: the need for large multilabeled datasets. Collecting and annotating raw aerial image datasets can be extremely time- and labor-consuming. To address this concern in MAIC, domain adaptation (DA) provides a novel solution by transferring the knowledge learned from a label-rich dataset (i.e., the source domain) to a label-scarce dataset (i.e., the target domain), while current DA models are mainly designed for single-labeled tasks. In this article, we propose a novel end-to-end MAIC model based on DA techniques, named DA-MAIC. To the best of our knowledge, this article for the first time integrates DA to tackle the label scarcity problem in the MAIC task. Specifically, the proposed DA-MAIC is composed of two main parts: the image classifier and the domain classifier. The image classifier captures task-discriminative features based on the graph convolutional network (GCN) to predict multiple image labels; and the domain classifier extracts domain-invariant representations, which mitigates the domain shift between two underlying distributions. We extensively evaluate the proposed DA-MAIC from different perspectives on three benchmark datasets, including the commonly used UCM dataset, the high-resolution AID dataset, and the recently proposed DFC15 dataset. Both quantitative and qualitative results support that the proposed DA-MAIC can generalize the source domain knowledge to new scenarios and substantially improve the classification performance on the target domain task.

BathyNet: A Deep Neural Network for Water Depth Mapping from Multispectral Aerial Images

Besides airborne laser bathymetry and multimedia photogrammetry, spectrally derived bathymetry provides a third optical method for deriving water depths. In this paper, we introduce BathyNet, an U-net like convolutional neural network, based on high-resolution, multispectral RGBC (red, green, blue, coastal blue) aerial images. The approach combines photogrammetric and radiometric methods: Preprocessing of the raw aerial images relies on strict ray tracing of the potentially oblique image rays, considering the intrinsic and extrinsic camera parameters. The actual depth estimation exploits the radiometric image content in a deep learning framework. 3D water surface and water bottom models derived from simultaneously captured laser bathymetry point clouds serve as reference and training data for both image preprocessing and actual depth estimation. As such, the approach highlights the benefits of jointly processing data from hybrid active and passive imaging sensors. The RGBC images and laser data of four groundwater supplied lakes around Augsburg, Germany, captured in April 2018 served as the basis for testing and validating the approach. With systematic depth biases less than 15 cm and a standard deviation of around 40 cm, the results satisfy the vertical accuracy limit Bc7 defined by the International Hydrographic Organization. Further improvements are anticipated by extending BathyNet to include a simultaneous semantic segmentation branch.

Improved Accuracy of High-Throughput Phenotyping From Unmanned Aerial Systems by Extracting Traits Directly From Orthorectified Images.

The development of high-throughput genotyping and phenotyping has provided access to many tools to accelerate plant breeding programs. Unmanned Aerial Systems (UAS)-based remote sensing is being broadly implemented for field-based high-throughput phenotyping due to its low cost and the capacity to rapidly cover large breeding populations. The Structure-from-Motion photogrammetry processes aerial images taken from multiple perspectives over a field to an orthomosaic photo of a complete field experiment, allowing spectral or morphological trait extraction from the canopy surface for each individual field plot. However, some phenotypic information observable in each raw aerial image seems to be lost to the orthomosaic photo, probably due to photogrammetry processes such as pixel merging and blending. To formally assess this, we introduced a set of image processing methods to extract phenotypes from orthorectified raw aerial images and compared them to the negative control of extracting the same traits from processed orthomosaic images. We predict that standard measures of accuracy in terms of the broad-sense heritability of the remote sensing spectral traits will be higher using the orthorectified photos than with the orthomosaic image. Using three case studies, we therefore compared the broad-sense heritability of phenotypes in wheat breeding nurseries including, (1) canopy temperature from thermal imaging, (2) canopy normalized difference vegetation index (NDVI), and (3) early-stage ground cover from multispectral imaging. We evaluated heritability estimates of these phenotypes extracted from multiple orthorectified aerial images via four statistical models and compared the results with heritability estimates of these phenotypes extracted from a single orthomosaic image. Our results indicate that extracting traits directly from multiple orthorectified aerial images yielded increased estimates of heritability for all three phenotypes through proper modeling, compared to estimation using traits extracted from the orthomosaic image. In summary, the image processing methods demonstrated in this study have the potential to improve the quality of the plant trait extracted from high-throughput imaging. This, in turn, can enable breeders to utilize phenomics technologies more effectively for improved selection.

ImageBreed: Open‐access plant breeding web–database for image‐based phenotyping

AbstractHigh‐throughput image‐phenotyping promises to accelerate the rate of genetic improvement in plant breeding through varietal selections informed by longitudinal growth models. To facilitate routine analyses and to drive breeding decisions, data integration is critical for effective management of germplasm, field experiment design, phenotyping, tissue sampling, genotyping, aerial‐phenotyping campaigns, image files, and geo‐spatial information. To this end, ImageBreed provides a software solution for end‐to‐end image‐based phenotyping integrated into the Breedbase plant breeding system. ImageBreed provides open‐source orthophotomosaic construction for raw image captures from standard color cameras and from the MicaSense RedEdge multispectral camera. Additionally, previously assembled orthophotomosaic raster images can be uploaded. Orthophotomosaic images allow for streamlined extraction of plot‐polygon images; however, ImageBreed plot‐polygon images can also be extracted directly from raw aerial image captures. A web–database interface streamlines assignment of plot‐polygon images from the orthophotomosaic or raw aerial‐captures to the field experiment design. Image processes spanning Fourier‐transform filtering, thresholding, and vegetation index masking are applied to reduce noise in extracted phenotypes. Summary‐statistic phenotypic values are extracted for every observed plot‐polygon image using a structured ontology. Plot‐polygon images are queryable against genotypic, phenotypic, and experimental design information for training of machine learning models and for driving breeding decisions in varietal advancement. ImageBreed is publicly available at http://imagebreed.org and built on the open‐source Breedbase system (https://github.com/solgenomics/sgn); all image‐processing scripts are available at https://github.com/solgenomics/DroneImageScripts and via a Docker image. All data deposited in http://imagebreed.org are publicly available for longitudinal model training and for driving future breeding decisions.

DisCountNet: Discriminating and Counting Network for Real-Time Counting and Localization of Sparse Objects in High-Resolution UAV Imagery

Recent deep-learning counting techniques revolve around two distinct features of data—sparse data, which favors detection networks, or dense data where density map networks are used. Both techniques fail to address a third scenario, where dense objects are sparsely located. Raw aerial images represent sparse distributions of data in most situations. To address this issue, we propose a novel and exceedingly portable end-to-end model, DisCountNet, and an example dataset to test it on. DisCountNet is a two-stage network that uses theories from both detection and heat-map networks to provide a simple yet powerful design. The first stage, DiscNet, operates on the theory of coarse detection, but does so by converting a rich and high-resolution image into a sparse representation where only important information is encoded. Following this, CountNet operates on the dense regions of the sparse matrix to generate a density map, which provides fine locations and count predictions on densities of objects. Comparing the proposed network to current state-of-the-art networks, we find that we can maintain competitive performance while using a fraction of the computational complexity, resulting in a real-time solution.

TERRAIN AWARE IMAGE CLIPPING FOR REAL-TIME AERIAL MAPPING

Abstract. Many remote sensing applications demand for a fast and efficient way of generating orthophoto maps from raw aerial images. One prerequisite is direct georeferencing, which allows to geolocate aerial images to their geographic position on the earth’s surface. But this is only half the story. When dealing with a large quantity of highly overlapping images, a major challenge is to select the most suitable image parts in order to generate seamless aerial maps of the captured area. This paper proposes a method that quickly determines such an optimal (rectangular) section for each single aerial image, which in turn can be used for generating seamless aerial maps. Its key approach is to clip aerial images depending on their geometric intersections with a terrain elevation model of the captured area, which is why we call it terrain aware image clipping (TAC). The method has a modest computational footprint and is therefore applicable even for rather limited embedded vision systems. It can be applied for both, real-time aerial mapping applications using data links as well as for rapid map generation right after landing without any postprocessing step. Referring to real-time applications, this method also minimizes transmission of redundant image data. The proposed method has already been demonstrated in several search-and-rescue scenarios and real-time mapping applications using a broadband data link and diffent kinds of camera and carrier systems. Moreover, a patent for this technology is pending.

Efficient and accurate set-based registration of time-separated aerial images

This paper addresses the task of time-separated aerial image registration. The ability to solve this problem accurately and reliably is important for a variety of subsequent image understanding applications. The principal challenge lies in the extent and nature of transient appearance variation that a land area can undergo, such as that caused by the change under illumination conditions, seasonal variations, or the occlusion by non-persistent objects (people, cars). Our work introduces several major novelties (i) unlike previous work on aerial image registration, we approach the problem using a set-based paradigm; (ii) we show how image space local, pair-wise constraints can be used to enforce a globally good registration using a constraints graph structure; (iii) we show how a simple holistic representation derived from raw aerial images can be used as a basic building block of the constraints graph in a manner which achieves both high registration accuracy and speed; (iv) lastly, we introduce a new and, to the best of our knowledge, the only data corpus suitable for the evaluation of set-based aerial image registration algorithms. Using this data set, we demonstrate (i) that the proposed method outperforms the state-of-the-art for pair-wise registration already, achieving greater accuracy and reliability, while at the same time reducing the computational cost of the task and (ii) that the increase in the number of available images in a set consistently reduces the average registration error, with a major difference already for a single additional image.

Landmark Fingerprinting and Matching for Aerial Positioning Systems

This paper describes a new approach for a vision-based positioning system for unmanned aerial vehicles. The proposed method is matching landmarks found in an aerial image to a reference database, and it uses the match data to estimate the current position of the vehicle. Previous research in the area has generally focused on matching raw aerial image data to a set of reference images. Although these methods can be designed to provide acceptable results in specific scenarios, they struggle with variations in lighting, seasonal changes, and changing environments. We present a new multistage method that aims to overcome these challenges by analyzing the image for key features that can be matched to known ground objects. The new approach can be divided into a set of subproblems: detection, fingerprinting, matching, and state estimation. Computational analysis shows that the system can accurately detect and match landmarks to a prior database with 70% accuracy when operating with real landmark databases, high distortion levels, and viewing angles, thus enabling the system to determine the position of the vehicle. The proposed system is more flexible than current methods as the core methods (descriptor and matcher) are independent of the sensor used for detection and can be used with any type of landmark. It is therefore expected that the system can be applied to a wide range of problems, from pure vision-based navigation for small unmanned aerial vehicles to planetary exploration, as long as a sensor capable of registering relevant landmarks is available.

Digital aerial images land cover classification based on vegetation indices

Abstract . Knowledge of how land cover has changed over time improve assessments of the changes in the future. Wide availability of remote sensed data and relatively low cost of their acquisition make them very attractive data source for Geographical Information Systems (GIS). The main goal of this paper is to prepare, run and evaluate image classification using a block of raw aerial images obtained from Digital Mapping Camera (DMC). Classification was preceded by preparation of raw images. It contained geometric and radiometric correction of every image in block. Initial images processing lead to compensate their brightness differences. It was obtained by calculating two vegetation indices: Normalized Difference Vegetation Index (NDVI) and Green Normalized Vegetation Index (gNDVI). These vegetation indices were the foundation of image classification. PCI Geomatics Geomatica 10.2 and Microimages TNT Mips software platforms were used for this purpose.