Unmanned Aerial Vehicles Datasets Research Articles

An Integrated Object and Machine Learning Approach for Tree Canopy Extraction from UAV Datasets

Unmanned aerial vehicles (UAV) have emerged as new platforms for acquiring ultra-high resolution images, which are challenging for extraction of features using conventional image processing approaches. Tree canopies are required to be constantly monitored for better planning and management. UAV is currently one way to survey canopies over a large area for precisely estimating their geometry. Conventional segmentation techniques are extensively used for image feature extraction. However, they lack in accuracy and require high computational processing when used for ultra-high resolution UAV datasets. These issues can be handled by superpixel segmentation algorithms which have good boundary adherence and are computationally efficient. Simple linear iterative clustering (SLIC) is a subset of superpixel segmentation technique which uses minimum tuning parameters making it most efficient. As the random forest is known for handling multiple attributes and robustness, it can be used for classifying and extracting features from segmented image generated using SLIC. The present study mainly focuses on the automation for extraction of tree canopies along with their object-based attributes from the UAV dataset. The data acquisition was carried out using Trimble UX5 fixed-wing UAV which was further orthorectified at a spatial resolution of 13 cm. The ortho-image was further segmented using SLIC algorithm. Canopy segments are then identified and classified using random forest, which is then merged into trees objects on the basis of their proximity. Accuracy assessment was then carried out for extracted tree canopies and was found that the aforesaid approach could achieve 93% similarity index. The current study highlights the potential of using SLIC segmentation and random forest classification method for tree canopy extraction from the ultra-high resolution ortho-image derived from UAV platforms.

Modeling the directional anisotropy of fine-scale TIR emissions over tree and crop canopies based on UAV measurements

Land surface temperature (LST) is a vital parameter for the achievement of the surface energy budget and in thorough investigations of water cycle processes. Lightweight thermal infrared (TIR) sensors onboard unmanned aerial vehicles (UAVs) are rapidly becoming key instruments for extracting high-resolution LSTs given the flexibility they offer in capturing different scales. With this expansion, there has been increasing concern regarding the growing demand to obtain a mapping of normalized LST given the directional anisotropy (DA) of surface fine-scale emissions. To date, this topic suffers from a lack of deep analysis and practical solutions for characterizing the DA of fine-scale TIR data from UAV measurements over tree and crop canopies. In this paper, the first objective was to understand the pattern of brightness temperatures (BTs) DAs at a high spatial resolution by using UAV-based multiangle observations and three-dimensional (3D) radiative transfer model simulations. This study highlighted the need for first performing an angular normalization of the BTs of fine-scale pixels prior to any application, as these were easily affected by adjacent pixels and displayed broad spatial variability from 0.5 °C to 5.0 °C due to 3D occlusion. The second objective of the present study was to appraise the reliability of a modified kernel-driven model, in comparison to the model from which it was derived, with an additional kernel designed to mimic the adjacency effect, plus, a quadratic function used to simplify the estimate of the directional emissivity kernel. The root mean square error of the best fit between the measured UAV dataset and the modified kernel-driven model was approximately 0.65 °C, which proves its efficiency since the DA indexes of the BTs were about 1.40 °C. This outlined the role of the model to normalize from directional effects the camera image pixels and thereby deliver fine-scale BTs. In addition, results from LESS simulations also demonstrated the good performance of the modified kernel-driven model for simulating the DAs of thermal emissions for both tree and row-planted scenes.Index Terms—Land surface temperature, UAV, directional anisotropy, high spatial resolution.

Reliable and Efficient UAV Image Matching via Geometric Constraints Structured by Delaunay Triangulation

Outlier removal is a crucial step in local feature-based unmanned aerial vehicle (UAV) image matching. Inspired by our previous work, this paper proposes a method for reliable and efficient outlier removal in UAV image matching. The inputs of the method are only two images without any other auxiliary data. The core idea is to design local geometric constraints within the neighboring structure via the Delaunay triangulation and use a two-stage method for outlier removal and match refinement. In the filter stage, initial matches are first organized as the Delaunay triangulation (DT) and its corresponding graph, and their dissimilarity scores are computed from the affine-invariant spatial angular order (SAO), which is used to achieve hierarchical outlier removal. In addition, by using the triangle constraint between the refined Delaunay triangulation and its corresponding graph, missed inliers are resumed from match expansion. In the verification stage, retained matches are refined using the RANSAC-based global geometric constraint. Therefore, the two-stage algorithm is termed DTSAO-RANSAC. Finally, using four datasets, DTSAO-RANSAC is comprehensively analyzed and compared with other methods in feature matching and image orientation tests. The experimental results demonstrate that compared with the LO-RANSAC algorithm, DTSAO-RANSAC can achieve efficient outlier removal with speedup ratios ranging from 4 to 16 and, it can provide reliable matching results for image orientation of UAV datasets.

UAV‐based training for fully fuzzy classification of Sentinel‐2 fluvial scenes

AbstractAn estimated 76% of global stream area is occupied by channels with widths above 30 m. Sentinel‐2 imagery with resolutions of 10 m could supply information about the composition of river corridors at national and global scales. Fuzzy classification models that infer sub‐pixel composition could further be used to compensate for small channel widths imaged at 10 m of spatial resolution. A major challenge to this approach is the acquisition of suitable training data useable in machine learning models that can predict land‐cover type information from image radiance values. In this contribution, we present a method which combines unmanned aerial vehicles (UAVs) and Sentinel‐2 imagery in order to develop a fuzzy classification approach capable of large‐scale investigations. Our approach uses hyperspatial UAV imagery in order to derive high‐resolution class information that can be used to train fuzzy classification models for Sentinel‐2 data where all bands are super‐resolved to a spatial resolution of 10 m. We use a multi‐temporal UAV dataset covering an area of 5.25 km2. Using a novel convolutional neural network (CNN) classifier, we predict sub‐pixel membership for Sentinel‐2 pixels in the fluvial corridor as divided into classes of water, vegetation and dry sediment. Our CNN model can predict fuzzy class memberships with median errors from −5% to +3% and mean absolute errors from 10% to 20%. We also show that our CNN fuzzy predictor can be used to predict crisp classes with accuracies from 95.5% to 99.9%. Finally, we use an example to show how a fuzzy CNN model trained with localized UAV data can be applied to longer channel reaches and detect new vegetation growth. We therefore argue that the novel use of UAVs as field validation tools for freely available satellite data can bridge the scale gap between local and regional fluvial studies. © 2020 The Authors. Earth Surface Processes and Landforms published by John Wiley & Sons Ltd

UAV object tracking by background cues and aberrances response suppression mechanism

Real-time object tracking for unmanned aerial vehicles (UAVs) is an essential and challenging research topic for computer vision. However, the scenarios that UAVs deal with are complicated, and the UAV tracking targets are small. Therefore, general trackers often fail to take full advantage of their performance in UAV scenarios. In this paper, we propose a tracking method for UAV scenes, which utilizes background cues and aberrances response suppression mechanism to track in 4 degrees-of-freedom. Firstly, we consider the tracking task as a similarity measurement problem. In this study, we decompose this problem into two subproblems for optimization. Secondly, to alleviate the problem of small targets in UAV scenes, we utilize background cues fully. Also, to reduce interference by background information, we employ an aberrance response suppression mechanism. Then, to obtain accurate target state information, we introduce a logarithmic polar coordinate system. We perform phase correlation calculations in logarithmic polar coordinates to obtain the rotation and scale changes of the target. Finally, target states are obtained through response fusion, which includes displacement, scale, and rotation angle. Our approach is carried out in a large number of experiments on various UAV datasets, such as UAV123, DBT70, and UAVDT2019. Compared with the current advanced trackers, our method is superior on UAV small target tracking.

UAVid: A semantic segmentation dataset for UAV imagery

Semantic segmentation has been one of the leading research interests in computer vision recently. It serves as a perception foundation for many fields, such as robotics and autonomous driving. The fast development of semantic segmentation attributes enormously to the large scale datasets, especially for the deep learning related methods. There already exist several semantic segmentation datasets for comparison among semantic segmentation methods in complex urban scenes, such as the Cityscapes and CamVid datasets, where the side views of the objects are captured with a camera mounted on the driving car. There also exist semantic labeling datasets for the airborne images and the satellite images, where the nadir views of the objects are captured. However, only a few datasets capture urban scenes from an oblique Unmanned Aerial Vehicle (UAV) perspective, where both of the top view and the side view of the objects can be observed, providing more information for object recognition. In this paper, we introduce our UAVid dataset, a new high-resolution UAV semantic segmentation dataset as a complement, which brings new challenges, including large scale variation, moving object recognition and temporal consistency preservation. Our UAV dataset consists of 30 video sequences capturing high-resolution images in oblique views. In total, 300 images have been densely labeled with 8 classes for the semantic labeling task. We have provided several deep learning baseline methods with pre-training, among which the proposed Multi-Scale-Dilation net performs the best via multi-scale feature extraction, reaching a mean intersection-over-union (IoU) score around 50%. We have also explored the influence of spatial-temporal regularization for sequence data by leveraging on feature space optimization (FSO) and 3D conditional random field (CRF). Our UAVid website and the labeling tool have been published online (https://uavid.nl/).

Predicting Biomass and Yield in a Tomato Phenotyping Experiment Using UAV Imagery and Random Forest.

Biomass and yield are key variables for assessing the production and performance of agricultural systems. Modeling and predicting the biomass and yield of individual plants at the farm scale represents a major challenge in precision agriculture, particularly when salinity and other abiotic stresses may play a role. Here, we evaluate a diversity panel of the wild tomato species (Solanum pimpinellifolium) through both field and unmanned aerial vehicle (UAV)-based phenotyping of 600 control and 600 salt-treated plants. The study objective was to predict fresh shoot mass, tomato fruit numbers, and yield mass at harvest based on a range of variables derived from the UAV imagery. UAV-based red–green–blue (RGB) imageries collected 1, 2, 4, 6, 7, and 8 weeks before harvest were also used to determine if prediction accuracies varied between control and salt-treated plants. Multispectral UAV-based imagery was also collected 1 and 2 weeks prior to harvest to further explore predictive insights. In order to estimate the end of season biomass and yield, a random forest machine learning approach was implemented using UAV-imagery-derived predictors as input variables. Shape features derived from the UAV, such as plant area, border length, width, and length, were found to have the highest importance in the predictions, followed by vegetation indices and the entropy texture measure. The multispectral UAV imagery collected 2 weeks prior to harvest produced the highest explained variances for fresh shoot mass (87.95%), fruit numbers (63.88%), and yield mass per plant (66.51%). The RGB UAV imagery produced very similar results to those of the multispectral UAV dataset, with the explained variance reducing as a function of increasing time to harvest. The results showed that predicting the yield of salt-stressed plants produced higher accuracies when the models excluded control plants, whereas predicting the yield of control plants was not affected by the inclusion of salt-stressed plants within the models. This research demonstrates that it is possible to predict the average biomass and yield up to 8 weeks prior to harvest within 4.23% of field-based measurements and up to 4 weeks prior to harvest at the individual plant level. Results from this work may be useful in providing guidance for yield forecasting of healthy and salt-stressed tomato plants, which in turn may inform growing practices, logistical planning, and sales operations.

Assessing the effect of spatial resolution on the delineation of management zones for smallholder farming in southern Brazil

Remote sensing estimates of cover-crop aboveground biomass (AGB) have been used to delineate management zones for smallholder farming in southern Brazil. In this study, we investigated the spatial resolution influence on the AGB estimates of rye, calculated from regression relationships with the Normalized Difference Vegetation Index (NDVI), and on the subsequent delineation of management zones using the Management Zone Analyst (MZA) software. Data acquired by an Unmanned Aerial Vehicle (UAV) Parrot Sequoia camera (0.20 m spatial resolution) in Brazil were compared with observations from the PlanetScope (PS) satellite constellation (3 m) and the Operational Land Imager (OLI)/Landsat-8 (30 m). A three-endmember mixture model (green vegetation, soil, and shadow) was applied to surface reflectance data of these instruments for evaluating the cover-crop development at two dates in August 2017. Because of the differences in the technical specifications of the sensors, we resampled the UAV dataset into four levels of spatial resolution (1, 3, 10, and 30 m). Using the UAV map (0.20 m) as a reference, we obtained confusion matrices for the original and resampled data. The results showed that the increasing amounts of rye AGB from the beginning to the end of August promoted significant changes in surface reflectance and in soil-green vegetation fractions calculated at variable spatial resolution. The performance of the regression models to estimate cover-crop AGB was approximately similar in the transition from the sub-metric (0.20 m) to the metric (3 m) spatial resolutions, or from the UAV camera to the PS data. For all datasets, the MZA detected two management zones with zone 2 having higher cover-crop AGB than zone 1. When compared to the UAV management zone map (reference), the PS map had a moderate-to-substantial agreement, while the OLI/Landsat-8 map had a fair-to-moderate concordance. Substantial agreements with the reference map were observed at simulated 1 m and 3 m data, as indicated by Kappa coefficients of 0.73 and 0.63 and overall accuracies of 86.40% and 81.40%, respectively. We conclude that the 3 m spatial resolution data of the PS comprise an alternative to delineate management zones for smallholder farming in southern Brazil when compared to the very-high spatial resolution observations of the UAV cameras.

Transfer Change Rules from Recurrent Fully Convolutional Networks for Hyperspectral Unmanned Aerial Vehicle Images without Ground Truth Data

Change detection (CD) networks based on supervised learning have been used in diverse CD tasks. However, such supervised CD networks require a large amount of data and only use information from current images. In addition, it is time consuming to manually acquire the ground truth data for newly obtained images. Here, we proposed a novel method for CD in case of a lack of training data in an area near by another one with the available ground truth data. The proposed method automatically entails generating training data and fine-tuning the CD network. To detect changes in target images without ground truth data, the difference images were generated using spectral similarity measure, and the training data were selected via fuzzy c-means clustering. Recurrent fully convolutional networks with multiscale three-dimensional filters were used to extract objects of various sizes from unmanned aerial vehicle (UAV) images. The CD network was pre-trained on labeled source domain data; then, the network was fine-tuned on target images using generated training data. Two further CD networks were trained with a combined weighted loss function. The training data in the target domain were iteratively updated using he prediction map of the CD network. Experiments on two hyperspectral UAV datasets confirmed that the proposed method is capable of transferring change rules and improving CD results based on training data extracted in an unsupervised way.

The Blackbird UAV dataset

This article describes the Blackbird unmanned aerial vehicle (UAV) Dataset, a large-scale suite of sensor data and corresponding ground truth from a custom-built quadrotor platform equipped with an inertial measurement unit (IMU), rotor tachometers, and virtual color, grayscale, and depth cameras. Motivated by the increasing demand for agile, autonomous operation of aerial vehicles, this dataset is designed to facilitate the development and evaluation of high-performance UAV perception algorithms. The dataset contains over 10 hours of data from our quadrotor tracing 18 different trajectories at varying maximum speeds (0.5 to 13.8 ms-1) through 5 different visual environments for a total of 176 unique flights. For each flight, we provide 120 Hz grayscale, 60 Hz RGB-D, and 60 Hz semantically segmented images from forward stereo and downward-facing photorealistic virtual cameras in addition to 100 Hz IMU, ~190 Hz motor speed sensors, and 360 Hz millimeter-accurate motion capture ground truth. The Blackbird UAV dataset is therefore well suited to the development of algorithms for visual inertial navigation, 3D reconstruction, and depth estimation. As a benchmark for future algorithms, the performance of two state-of-the-art visual odometry algorithms are reported and scripts for comparing against the benchmarks are included with the dataset. The dataset is available for download at http://blackbird-dataset.mit.edu/.

Online Semantic Subspace Learning with Siamese Network for UAV Tracking

In urban environment monitoring, visual tracking on unmanned aerial vehicles (UAVs) can produce more applications owing to the inherent advantages, but it also brings new challenges for existing visual tracking approaches (such as complex background clutters, rotation, fast motion, small objects, and realtime issues due to camera motion and viewpoint changes). Based on the Siamese network, tracking can be conducted efficiently in recent UAV datasets. Unfortunately, the learned convolutional neural network (CNN) features are not discriminative when identifying the target from the background/clutter, In particular for the distractor, and cannot capture the appearance variations temporally. Additionally, occlusion and disappearance are also reasons for tracking failure. In this paper, a semantic subspace module is designed to be integrated into the Siamese network tracker to encode the local fine-grained details of the target for UAV tracking. More specifically, the target’s semantic subspace is learned online to adapt to the target in the temporal domain. Additionally, the pixel-wise response of the semantic subspace can be used to detect occlusion and disappearance of the target, and this enables reasonable updating to relieve model drifting. Substantial experiments conducted on challenging UAV benchmarks illustrate that the proposed method can obtain competitive results in both accuracy and efficiency when they are applied to UAV videos.

Efficient match pair selection for oblique UAV images based on adaptive vocabulary tree

The primary contribution of this paper is an efficient match pair selection method for oblique unmanned aerial vehicle (UAV) images. First, high overlap degrees and spatial resolutions cause image and feature redundancies in vocabulary tree building and image indexing. To cope with this issue, an image selection strategy and a feature selection strategy are designed to decrease the total number of features. Second, by analysing the distribution of the similarity scores, an adaptive threshold selection method is implemented to determine the number of candidate match pairs for each query image, and it avoids the disadvantages of the fixed number and fixed proportion methods. Then, an algorithm, termed AVT-Expansion, is proposed for the match pair selection and simplification where the initial match pairs are first selected by using the adaptive vocabulary tree (AVT). To simplify the initial match pairs, the AVT method is integrated with our previous MST-Expansion algorithm, which is used to extract a match graph by analysing the image topological connection network. Finally, the proposed method is verified using three UAV datasets captured with different oblique multi-camera systems. Experimental results demonstrate that the efficiency of the vocabulary tree building is improved, with speed-up ratios ranging from 14 to 16, and that high image retrieval precision values are obtained for the three datasets. For match pair selection of oblique UAV images, the proposed method is an efficient solution.

Quality assessment of 3D point cloud of industrial buildings from imagery acquired by oblique and nadir UAV flights

Purpose. The main objective of this paper is to assess the quality of the 3D model of industrial buildings generated from Unmanned Aerial Vehicle (UAV) imagery datasets, including nadir (N), oblique (O), and Nadir and Oblique (N+O) UAV datasets. Methodology. The quality of a 3D model is defined by the accuracy and density of point clouds created from UAV images. For this purpose, the UAV was deployed to acquire images with both O and N flight modes over an industrial mining area containing a mine shaft tower, factory housing and office buildings. The quality assessment was conducted for the 3D point cloud model of three main objects such as roofs, facades, and ground surfaces using CheckPoints (CPs) and terrestrial laser scanning (TLS) point clouds as the reference datasets. The Root Mean Square Errors (RMSE) were calculated using CP coordinates, and cloud to cloud distances were computed using TLS point clouds, which were used for the accuracy assessment. Findings. The results showed that the point cloud model generated by the N flight mode was the most accurate but least dense, whereas that of the O mode was the least accurate but most detailed level in comparison with the others. Also, the combination of O and N datasets takes advantages of individual mode as the point clouds accuracy is higher than that of case O, and its density is much higher than that of case N. Therefore, it is optimal to build exceptional accurate and dense point clouds of buildings. Originality. The paper provides a comparative analysis in quality of point cloud of roofs and facades generated from UAV photogrammetry for mining industrial buildings. Practical value. Findings of the study can be used as references for both UAV survey practices and applications of UAV point cloud. The paper provides useful information for making UAV flight planning, or which UAV points should be integrated into TLS points to have the best point cloud.

Conditional GAN based individual and global motion fusion for multiple object tracking in UAV videos

Multiple Object Tracking (MOT) meets great challenges in videos captured by Unmanned Aerial Vehicles (UAVs). Different from traditional videos, due to high altitude and abrupt motion changes of UAVs, the sizes of target objects in UAVs videos are usually very small and the appearance information of target objects is unreliable. The motion analysis is meaningful to associate multiple objects in UAV videos. However, the traditional motion analysis models inevitably suffer from the autonomous motion of UAVs. In this paper, we proposed a Conditional Generative Adversarial Networks (GAN) based model to predict complex motions in UAV videos. We regard the objects motions and the UAV movement as the individual motions and global motions respectively. They are complementary with each other and are employed jointly to facilitate accurate motion prediction. Specifically, a social Long Short Term Memory network is exploited to estimate the individual motion of objects, and a Siamese network is constructed to generate the global motion to reflect the view changes from UAVs, and a conditional GAN is developed to generate the final motion affinity. Extensive experimental results are conducted on public UAV datasets contained various types of objects and 4 different kinds of object detection inputs. Robust motion prediction and improved MOT performance are achieved compared with state-of-the-art methods.

Image fusion employing adaptive spectral-spatial gradient sparse regularization in UAV remote sensing

Unmanned aerial vehicle (UAV) remote sensing has been widely used in vegetation phenotypes and precision agriculture. The fusion of UAV multispectral and panchromatic images has considerable research value. For example, an accurate vegetation index can be obtained. However, large geometrical distortions are observed in UAV images, contributing to the insufficiency of existing fusion algorithms. Spectrum consistency, which indicates that changes in spectral direction are always a smooth function, is investigated in this paper to solve the above problem. Spatial adaptivity is also introduced to reduce spectral distortion in the fusion process. Based on the two aspects, a multispectral and panchromatic image fusion model employing adaptive spectral-spatial gradient sparse regularization is proposed for UAV remote sensing. The separable approximation and augmented Lagrangian methods are employed to optimize this model. In the experiments, the proposed method is firstly compared with other state-of-the-art fusion algorithms, and good performance is verified by UAV datasets in terms of visual effect and objective quality analysis. Secondly, the fusion algorithm is applied in the application of a vegetation phenotype. The experiments finally demonstrate that accurate vegetation indices can be generated by adopting the proposed algorithm. This finding proves the substantial research value of the proposed algorithm in UAV remote sensing.

The Unmanned Aerial Vehicle Benchmark: Object Detection, Tracking and Baseline

With the increasing popularity of Unmanned Aerial Vehicles (UAVs) in computer vision-related applications, intelligent UAV video analysis has recently attracted the attention of an increasing number of researchers. To facilitate research in the UAV field, this paper presents a UAV dataset with 100 videos featuring approximately 2700 vehicles recorded under unconstrained conditions and 840k manually annotated bounding boxes. These UAV videos were recorded in complex real-world scenarios and pose significant new challenges, such as complex scenes, high density, small objects, and large camera motion, to the existing object detection and tracking methods. These challenges have encouraged us to define a benchmark for three fundamental computer vision tasks, namely, object detection, single object tracking (SOT) and multiple object tracking (MOT), on our UAV dataset. Specifically, our UAV benchmark facilitates evaluation and detailed analysis of state-of-the-art detection and tracking methods on the proposed UAV dataset. Furthermore, we propose a novel approach based on the so-called Context-aware Multi-task Siamese Network (CMSN) model that explores new cues in UAV videos by judging the consistency degree between objects and contexts and that can be used for SOT and MOT. The experimental results demonstrate that our model could make tracking results more robust in both SOT and MOT, showing that the current tracking and detection methods have limitations in dealing with the proposed UAV benchmark and that further research is indeed needed.

Attitude Jitter Compensation for Remote Sensing Images Using Convolutional Neural Network

Attitude jitter of satellites and unmanned aerial vehicle (UAV) platforms is a problem that degenerates the imaging quality in high-resolution remote sensing. This letter proposes a deep learning architecture that automatically learns essential scene features from a single image to estimate the attitude jitter, which is used to compensate deformed images. The proposed methodology consists of a convolutional neural network and a jitter compensation model. The neural network analyzes the deformed images and generates the attitude jitter vectors in two directions, which are utilized to correct the images through interpolation and resampling. The PatternNet and the small UAV data sets are introduced to train the neural network and to validate its effectiveness and accuracy. The compensation results on distorted remote sensing images obtained by satellites and UAVs reveal that the image distortion due to attitude jitter is clearly reduced and that the geometric quality is effectively improved. Compared to the existing methods that primarily rely on sensor data or parallax observation, no auxiliary information is required in our framework.

DiagonalNet: Confidence diagonal lines for the UAV detection

Due to the maneuverability and small size of the Unmanned aerial vehicles (UAVs), the traditional object detection method cannot meet the requirements of detection accuracy and real‐time performance. To address this dilemma, we propose an object detection method by using the improved hourglass network as its backbone network generating the confidence diagonal lines as detection result, and then getting the bounding box through this diagonal line. As a one‐stage method, it has a good real‐time performance. At the same time, due to the good performance of our improved network, its mean average precision performs well. Through experiments with the UAV dataset, compared with faster‐regions‐with‐convolutional‐neural‐network and You only look once (YOLO), we show that the proposed framework can quickly and accurately detect objects. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Online Spatial and Temporal Calibration for Monocular Direct Visual-Inertial Odometry.

Owing to the nonlinearity in visual-inertial state estimation, sufficiently accurate initial states, especially the spatial and temporal parameters between IMU (Inertial Measurement Unit) and camera, should be provided to avoid divergence. Moreover, these parameters are required to be calibrated online since they are likely to vary once the mechanical configuration slightly changes. Recently, direct approaches have gained popularity for their better performance than feature-based approaches in little-texture or low-illumination environments, taking advantage of tracking pixels directly. Based on these considerations, we perform a direct version of monocular VIO (Visual-inertial Odometry), and propose a novel approach to initialize the spatial-temporal parameters and estimate them with all other variables of interest (IMU pose, point inverse depth, etc.). We highlight that our approach is able to perform robust and accurate initialization and online calibration for the spatial and temporal parameters without utilizing any prior information, and also achieves high-precision estimates even when large temporal offset occurs. The performance of the proposed approach was verified through the public UAV (Unmanned Aerial Vehicle) dataset.

Building Extraction from UAV Images Jointly Using 6D-SLIC and Multiscale Siamese Convolutional Networks

Automatic building extraction using a single data type, either 2D remotely-sensed images or light detection and ranging 3D point clouds, remains insufficient to accurately delineate building outlines for automatic mapping, despite active research in this area and the significant progress which has been achieved in the past decade. This paper presents an effective approach to extracting buildings from Unmanned Aerial Vehicle (UAV) images through the incorporation of superpixel segmentation and semantic recognition. A framework for building extraction is constructed by jointly using an improved Simple Linear Iterative Clustering (SLIC) algorithm and Multiscale Siamese Convolutional Networks (MSCNs). The SLIC algorithm, improved by additionally imposing a digital surface model for superpixel segmentation, namely 6D-SLIC, is suited for building boundary detection under building and image backgrounds with similar radiometric signatures. The proposed MSCNs, including a feature learning network and a binary decision network, are used to automatically learn a multiscale hierarchical feature representation and detect building objects under various complex backgrounds. In addition, a gamma-transform green leaf index is proposed to truncate vegetation superpixels for further processing to improve the robustness and efficiency of building detection, the Douglas–Peucker algorithm and iterative optimization are used to eliminate jagged details generated from small structures as a result of superpixel segmentation. In the experiments, the UAV datasets, including many buildings in urban and rural areas with irregular shapes and different heights and that are obscured by trees, are collected to evaluate the proposed method. The experimental results based on the qualitative and quantitative measures confirm the effectiveness and high accuracy of the proposed framework relative to the digitized results. The proposed framework performs better than state-of-the-art building extraction methods, given its higher values of recall, precision, and intersection over Union (IoU).