3D Spatial Information Research Articles

New Strategies for Time Delay Estimation during System Calibration for UAV-Based GNSS/INS-Assisted Imaging Systems

The need for accurate 3D spatial information is growing rapidly in many of today’s key industries, such as precision agriculture, emergency management, infrastructure monitoring, and defense. Unmanned aerial vehicles (UAVs) equipped with global navigation satellite systems/inertial navigation systems (GNSS/INS) and consumer-grade digital imaging sensors are capable of providing accurate 3D spatial information at a relatively low cost. However, with the use of consumer-grade sensors, system calibration is critical for accurate 3D reconstruction. In this study, ‘consumer-grade’ refers to cameras that require system calibration by the user instead of by the manufacturer or other high-end laboratory settings, as well as relatively low-cost GNSS/INS units. In addition to classical spatial system calibration, many consumer-grade sensors also need temporal calibration for accurate 3D reconstruction. This study examines the accuracy impact of time delay in the synchronization between the GNSS/INS unit and cameras on-board UAV-based mapping systems. After reviewing existing strategies, this study presents two approaches (direct and indirect) to correct for time delay between GNSS/INS recorded event markers and actual time of image exposure. Our results show that both approaches are capable of handling and correcting this time delay, with the direct approach being more rigorous. When a time delay exists and the direct or indirect approach is applied, horizontal accuracy of 1–3 times the ground sampling distance (GSD) can be achieved without either the use of any ground control points (GCPs) or adjusting the original GNSS/INS trajectory information.

A 3D dynamic visualization method coupled with an urban drainage model

The urban drainage model is an important tool for the management and decision-making around an urban drainage network. Owing to urban drainage networks’ complex distribution, existing model analysis methods have gradually exposed problems of insufficient utilization of spatial information, weak coupling and analytical capacity of multi-source heterogeneous data, and poor human-computer interaction. By coupling an urban drainage model with 3D GIS technology and 3D visualization technology, this study presents a 3D dynamic visualization method of an urban drainage model (3DDVM-UDM). A case study from Chaohu, China, is used to demonstrate the method. The environmental information and distributions of pipe network objects in 3D space were presented. The urban drainage model can be edited and simulated in the 3D environment, and a 2D flooding simulation method was also established. Multiple model results were dynamically and simultaneously visualized in the pipe network entity models. By coupling and comparing the model results under different rainfall return periods, various information was intuitively analyzed, such as the water level and water quality differences of each pipe and node, and the spatial distribution differences of fully filled pipes, overflowed nodes and buildings influenced by node overflow. The operation state of a drainage pipe network can be systematically analyzed by intuitively combining with the attribute information of a pipe network, 3D spatial information and other relevant data. This study provides a feasible theory and method to assist in environmental management and decision-making, to facilitate communication between different fields, and to promote a shift in analysis from 2D to the combination of 2D and 3D.

Corrigendum to “Implementation of an open platform for 3D spatial information based on WebGL”

Implementation of an open platform for 3D spatial information based on WebGL https://doi.org/10.4218/etrij.2018-0352 ETRI Journal, Volume 41, Issue 3, June 2019, pp. 277–288. Page 279: The text reads “We also describe the unique 3D data format for the Oracle BI publisher resource tracking file (XDO) for 3D structural models.”. This text should have read “We also describe the unique 3D data format XDO for 3D structural models.” Page 280: The caption for Figure 4 reads “Coordinate systems: (A) Oracle BI publisher resource tracking file (XDO) and (B) WebGL.” The caption should read “Coordinate systems: (A) XDO and (B) WebGL.” The author apologizes for these errors.

Deep CNNs as a method to classify rotating objects based on monostatic RCS

Radar systems emit a time-varying signal and measure the response of a radar-reflecting surface. In the case of narrowband, monostatic radar signal domain, all spatial information is projected into a radar cross-section (RCS) scalar. The authors address the challenging problem of determining shape class using monostatic RCS estimates collected as a time series from a rotating object tumbling with unknown motion parameters under detectability limitations and signal noise. Previous shape classification methods have relied on image-like synthetic aperture radar or multistatic (multiview) radar configurations with known geometry. Convolutional neural networks (CNNs) have revolutionised learning tasks in the computer vision domain by leveraging images and video rich with high-resolution two-dimensional (2D) or 3D spatial information. They show that a feed-forward CNN can be trained to successfully classify object shape using only noisy monostatic RCS signals with unknown motion. They construct datasets containing over 100,000 simulated RCS signals belonging to different shape classes. They introduce deep neural network architectures that produce 2% classification error on testing data. They also introduce a refinement network that transforms simulated signals to appear more realistic and improve training utility. The results are a pioneering step toward the recognition of more complex targets using narrowband, monostatic radar.

Right Ventricular Segmentation from MRI Using Deep Convolutional Neural Networks.

The assessment of right ventricular (RV) function is essential in the diagnosis of many cardiac diseases. Magnetic resonance imaging (MRI) offers an excellent solution to image right ventricle non-invasively with high contrast and temporal resolution. Manual assessment of the RV function from MRI sequences is tedious and time-consuming and automating the process is of great interest. This study proposes a convolutional neural network-based machine learning approach to automate the delineation of the RV from a sequence of MRI. The architecture of the neural network differs from that of a widely-known U-Net approach. Additionally, the proposed approach used image concatenation to create and utilize 3D spatial information in the segmentation process. Quantitative evaluations were performed over 256 images acquired from 16 patients in publicly available data in comparison to manual delineations. Comparisons with the results by U-Net demonstrated that the proposed method outperforms the prior state-of-the-art method.

IMPROVING 3D MESH QUALITY USING MULTI-DIRECTIONAL UAV IMAGES

Abstract. With the increasing use of 3D spatial information, updated 3D spatial information is required and unmanned aerial vehicles (UAVs) are being used to accomplish this task. In general, 3D spatial information is constructed by generating meshes through point clouds extracted from many images. However, generating a mesh that is comparable to a real building is difficult because UAV images use relatively cheap sensors compared to aerial images. In this study, Nadir images with various overlapping degrees were obtained and point clouds generated to compare the reproducibility of actual buildings. This led to the confirmation that the use of Nadir images alone does not fully complement the occlusion even when the images are taken with high overlapping degrees. To complement such limitations of unidirectional images, oblique images were used in addition to create point clouds, and the occlusion was found to have been complemented. Furthermore, through its conversion, high quality 3D spatial information data could be obtained.

Research on Somatosensory Interaction Based on Convolutional Neural Network

Somatosensory interaction is an important part of human-computer interaction. The core of somatosensory interaction must accurately obtain the 3D spatial information of the human body. This article uses an RGBD camera to acquire both color and depth images. We Perform 2D human pose estimation in color images using hourglass network with residual structure. At the same time, we use the sparse feature points to align the color map with the depth map in order to accurately map the detection results in color images to the corresponding depth images. The experimental results show that this method can accurately and quickly obtain the 3-dimensional posture information of the human body, which is an important guarantee for somatosensory interaction.

Extraction of topographic deformation based on the 3D information of individual trees

ABSTRACT Terrestrial laser scanning (TLS) has been used widely in topographic deformation monitoring, and the issue of deformation information extraction from multi-temporal point cloud data has high importance. Here, a method is developed for the extraction of topographic deformation in a forested area based on the 3D spatial information of individual trees. Initially, ground points are removed by using a local topographic surface fitting method, then individual trees are segmented using the PTrees algorithm; finally, the motion of individual trees is computed with a combination of global and local rigid-body transformations. Experimental validation results show that the proposed method is reliable to within a 0.12° maximum tilt error and 8.7 mm maximum displacement error. In addition, some inferences were drawn based on the original topography and the topographic deformation monitoring results. For example, a mining work-face was located beyond the eastern part of the study area, and trees growing in the middle part of this region of interest may face a survival challenge.

Research on BIM Forward Design Based on Oblique Photogrammetry Reality Model

With the rapid development and steady advancement of Building Information Modeling (BIM) technology in China, traditional 2D Digital Line Graphic (DLG) data cannot visually display rich 3D spatial information, and it is difficult to meet the needs of BIM forward design. Aiming at the increasing demand for 3D spatial data in municipal project design, this paper proposes a forward design method based on oblique photogrammetry reality model. Firstly, the Quadric Error Metrics (QEM) algorithm is used to lighten the reality model. Secondly, the standard model library of municipal facilities is built in CATIA, and the lightweight reality model is integrated into CATIA for fine restoration. The refined reality model can provide BIM designers with reference to features, geomorphology, etc., avoiding unreasonable spatial planning, no external reference to model monomers, and missing model space location information. It is convenient for designers to design, display, compare and modify the scheme in the same platform to improve the efficiency of BIM design. Change the delivery form of traditional surveying and mapping data results, and encourage designers to use BIM technology for forward design from surveying and mapping, and accelerate the popularization and promotion of BIM technology in the whole life cycle of the construction industry.

Optimal flight parameters for unmanned aerial vehicles collecting spatial information for estimating large-scale waste generation

ABSTRACT Large-scale waste and debris from disasters and building construction can cause secondary damage and environmental pollution. Satellite images and the Global Positioning System are being used to calculate the amounts of such waste. However, efficiency in terms of accuracy and cost remains problematic. To solve this problem, this study used an unmanned aerial vehicle (UAV) to build three-dimensional (3D) spatial information for large-scale waste generation. Appropriate parameters were selected to quickly and accurately estimate the amount of waste. Fifty-six scenarios were distinguished according to parameters such as flight altitude, image overlap, and number of ground control points (GCPs). Images were then registered by implementing the SIFT (Scale Invariant Feature Transform) and SfM (Structure from Motion) algorithms to build 3D spatial information and estimate the amount of waste. The accuracy was generally high when flight altitude and image overlap were high, and the number of GCPs was specific. Scenario A, with the lowest flight altitude and overlap, did not yield any registrable images. The results of the flight altitude showed a different tendency from those of existing studies. This could be attributable to the difference in the distance covered on the ground by an image in the width direction, which depended on the characteristics of study sites. Among all the scenarios, D-7 had the highest accuracy, with a root-mean-square error of about 0.08 m. In this scenario, the amount of waste was estimated to be approximately 767,571.3 m3. The top 10 scenarios in terms of the accuracy of x, y, and z locations displayed similar tendencies with regard to the amount of waste. In contrast, the bottom 15 scenarios showed a variety of results. When the accuracy and flight time of each scenario were considered, scenarios H-3, H-4, and H-5 achieved the highest accuracy in a short time. Thus, these scenarios were the most effective and applicable for estimating the amount of waste. Consequently, parameters such as flight altitude, image overlap, and GCPs need to be considered when estimating large-scale waste associated with a difference in elevation using a UAV.

Implementation of an open platform for 3D spatial information based on WebGL

Implementation of an open platform for 3D spatial information based on WebGL

Auditory scene reproduction for tele-operated robot systems

ABSTRACTIn a tele-operated robot environment, reproducing auditory scenes and conveying 3D spatial information of sound sources are inevitable in order to make operators feel more realistic tele-presence. In this paper, we propose a tele-presence robot system that enables reproduction and manipulation of auditory scenes. This tele-presence system is carried out on the basis of 3D information about where targeted humans are speaking, and matching with the operator's head orientation. We employed multiple microphone arrays and human tracking technologies to localize and separate voices around a robot. In the operator site, separated sound sources are rendered using head-related transfer functions (HRTF) according to the sound sources' spatial positions and the operator's head orientation that is being tracked real time. Two-party and three-party interaction experiments indicated that the proposed system has significantly higher accuracy when perceiving direction of sounds and gains higher subjective scores in sense of presence and listenability, compared to a baseline system which uses stereo binaural sounds obtained by two microphones mounted on the humanoid robot's ears.

Comparison between point cloud and mesh models using images from an unmanned aerial vehicle

Structure from motion (SfM) is a well-known algorithm used for the generating of three-dimensional (3D) spatial information using images. The objective of this study is to compare the measurements of objects ascertained from point cloud and mesh models derived from the SfM algorithm. In particular, we analyze a single tree to determine the correlation between the number of acquired images from the UAVs and the object measurement for each model. The results indicate that the number of images does not have a critical impact on surveys and the point cloud is approximately 2% more accurate than mesh models for individual tree measurement. Our results will be useful in terms of selecting the data acquisition method as well as the data itself for measuring objects based on SfM 3D data.

Generalized method to design phase masks for 3D super-resolution microscopy

Point spread function (PSF) engineering by phase modulation is a novel approach to three-dimensional (3D) super-resolution microscopy, with different point spread functions being proposed for specific applications. It is often not easy to achieve the desired shape of engineered point spread functions because it is challenging to determine the correct phase mask. Additionally, a phase mask can either encode 3D space information or additional time information, but not both simultaneously. A robust algorithm for recovering a phase mask to generate arbitrary point spread functions is needed. In this work, a generalized phase mask design method is introduced by performing an optimization. A stochastic gradient descent algorithm and a Gauss-Newton algorithm are developed and compared for their ability to recover the phase masks for previously reported point spread functions. The new Gauss-Newton algorithm converges to a minimum at much higher speeds. This algorithm is used to design a novel stretching-lobe phase mask to encode temporal and 3D spatial information simultaneously. The stretching-lobe phase mask and other masks are fabricated in-house for proof-of-concept using multi-level light lithography and an optimized commercially sourced stretching-lobe phase mask (PM) is validated experimentally to encode 3D spatial and temporal information. The algorithms' generalizability is further demonstrated by generating a phase mask that comprises four different letters at different depths.

Biomass Prediction of Heterogeneous Temperate Grasslands Using an SfM Approach Based on UAV Imaging

An early and precise yield estimation in intensive managed grassland is mandatory for economic management decisions. RGB (red, green, blue) cameras attached on an unmanned aerial vehicle (UAV) represent a promising non-destructive technology for the assessment of crop traits especially in large and remote areas. Photogrammetric structure from motion (SfM) processing of the UAV-based images into point clouds can be used to generate 3D spatial information about the canopy height (CH). The aim of this study was the development of prediction models for dry matter yield (DMY) in temperate grassland based on CH data generated by UAV RGB imaging over a whole growing season including four cuts. The multi-temporal study compared the remote sensing technique with two conventional methods, i.e., destructive biomass sampling and ruler height measurements in two legume-grass mixtures with red clover (Trifolium pratense L.) and lucerne (Medicago sativa L.) in combination with Italian ryegrass (Lolium multiflorum Lam.). To cover the full range of legume contribution occurring in a practical grassland, pure stands of legumes and grasses contained in each mixture were also investigated. The results showed, that yield prediction by SfM-based UAV RGB imaging provided similar accuracies across all treatments (R2 = 0.59–0.81) as the ruler height measurements (R2 = 0.58–0.78). Furthermore, results of yield prediction by UAV RGB imaging demonstrated an improved robustness when an increased CH variability occurred due to extreme weather conditions. It became apparent that morphological characteristics of clover-based canopies (R2 = 0.75) allow a better remotely sensed prediction of total annual yield than for lucerne-grass mixtures (R2 = 0.64), and that these crop-specific models cannot be easily transferred to other grassland types.

Real-time 3D volumetric human body reconstruction from a single view RGB-D capture device

Recently, volumetric video based communications have gained a lot of attention, especially due to the emergence of devices that can capture scenes with 3D spatial information and display mixed reality environments. Nevertheless, capturing the world in 3D is not an easy task, with capture systems being usually composed by arrays of image sensors, which sometimes are paired with depth sensors. Unfortunately, these arrays are not easy to assembly and calibrate by non-specialists, making their use in volumetric video applications a challenge. Additionally, the cost of these systems is still high, which limits their popularity in mainstream communication applications. This work proposes a system that provides a way to reconstruct the head of a human speaker from single view frames captured using a single RGB-D camera (e.g. Microsoft?s Kinect 2 device). The proposed system generates volumetric video frames with a minimum number of occluded and missing areas. To achieve a good quality, the system prioritizes the data corresponding to the participants? face, therefore preserving important information from speakers facial expressions. Our ultimate goal is to design an inexpensive system that can be used in volumetric video telepresence applications and even on volumetric video talk-shows broadcasting applications.

A Turn-On Optoacoustic Probe for Imaging Metformin-Induced Upregulation of Hepatic Hydrogen Sulfide and Subsequent Liver Injury.

Metformin is currently the most prescribed oral agent for diabetes treatment; however the overdose or long-term use may cause some severe side effects such as liver injury. Researches indicate that metformin-induced liver injury is closely related to upregulation of hepatic H2S. Hence, monitoring hepatic H2S generation induced by metformin could be an effective approach for evaluating hepatoxicity of the drug.Methods: We present a novel turn-on and dual-mode probe for detecting and imaging metformin-induced liver injury by specifically tracking the upregulation of hepatic H2S with fluorescent and optoacoustic methods. After reaction with H2S, the strong electron-withdrawing group dinitrophenyl ether (which acts as both the recognition moiety and the fluorescence quencher) was cleaved and replaced by an electron-donating group hydroxyl. This correspondingly leads to the changes of the probe's electronic state and absorption red-shifting as well as the subsequent turn-on fluorescent and optoacoustic signals.Results: The probe was applied to the colon tumor-bearing mice model and the metformin-induced liver injury mice model to achieve tumor imaging and liver injury assessment. The biosafety of the probe was verified by histological analysis (hematoxylin and eosin staining) and serum biochemical assays.Conclusion: The probe responds quickly to H2S in tumors and the liver, and MSOT imaging with the probe offers cross-secitonal and 3D spatial information of liver injury. This study may provide an effective approach for accessing medication side effects by tracking drug-metabolism-related products.

A Deep Learning-Based Approach for the Detection and Localization of Prostate Cancer in T2 Magnetic Resonance Images.

We address the problem of prostate lesion detection, localization, and segmentation in T2W magnetic resonance (MR) images. We train a deep convolutional encoder-decoder architecture to simultaneously segment the prostate, its anatomical structure, and the malignant lesions. To incorporate the 3D contextual spatial information provided by the MRI series, we propose a novel 3D sliding window approach, which preserves the 2D domain complexity while exploiting 3D information. Experiments on data from 19 patients provided for the public by the Initiative for Collaborative Computer Vision Benchmarking (I2CVB) show that our approach outperforms traditional pattern recognition and machine learning approaches by a significant margin. Particularly, for the task of cancer detection and localization, the system achieves an average AUC of 0.995, an accuracy of 0.894, and a recall of 0.928. The proposed mono-modal deep learning-based system performs comparably to other multi-modal MR-based systems. It could improve the performance of a radiologist in prostate cancer diagnosis and treatment planning.

Quantitative assessment for the rockfall hazard in a post-earthquake high rock slope using terrestrial laser scanning

Geometric information and discontinuity characterization of rock masses are the key aspects of the analysis of the evolution and failure mechanisms of rockfalls. With the advantageous use of terrestrial laser scanning (TLS), the accurate three-dimensional (3D) spatial information of rock slopes can be obtained without contact. By conducting a fuzzy K-means algorithm in this study, the automatic identification of discontinuity sets is achieved, and dominant occurrences of rock mass discontinuities in each local region can be acquired in great detail. This automatic identification method permits the user to visually identify the discontinuity sets and acquire their spacial distribution features, e.g. occurrences, spacings, trace lengths and their geometric compounding relationships. At the same time, based on the shortest distance (SD) algorithm and the surface-to-plane volume calculation algorithm, the distributions, shapes, volumes and scars of the rockfalls can be accurately detected over the monitored time interval. These methods are able to provide adequate investigations and quantitative assessment for the rockfall failure mechanisms and evolutions of the Hongshiyan post failure rock slope after the 2014 Ludian earthquake. The topography of the landslide surface, mechanical properties, scales and the structural features of the discontinuities have a significant effect on the failure mechanisms, distributions and volumes of rockfalls. The main failure mechanisms of the rockfall investigated in this rock slope can be divided into plane failure, wedge failure and toppling failure. The plane and wedge failures mainly occurred in the landslide surface while the toppling failures are mainly observed along the edge of the post failure slope, which is obviously responded by preferred discontinuities. The reverse-dip stratified structure characteristics and the excavation disturbances during the recovering and slope treatment stage induce numerous progressive and continuous failures of rockfalls. The results are beneficial for the design and optimization of rockfall treatments.

Triboelectric Self-Powered Wearable Flexible Patch as 3D Motion Control Interface for Robotic Manipulator.

Triboelectric nanogenerators and sensors can be applied as human-machine interfaces to the next generation of intelligent and interactive products, where flexible tactile sensors exhibit great advantages for diversified applications such as robotic control. In this paper, we present a self-powered, flexible, triboelectric sensor (SFTS) patch for finger trajectory sensing and further apply the collected information for robotic control. This innovative sensor consists of flexible and environmentally friendly materials, i. e., starch-based hydrogel, polydimethylsiloxane (PDMS), and silicone rubber. The sensor patch can be divided into a two-dimensional (2D) SFTS for in-plane robotic movement control and a one-dimensional (1D) SFTS for out-of-plane robotic movement control. The 2D-SFTS is designed with a grid structure on top of the sensing surface to track the continuous sliding information on the fingertip, e. g., trajectory, velocity, and acceleration, with four circumjacent starch-based hydrogel PDMS elastomer electrodes. Combining the 2D-SFTS with the 1D-SFTS, three-dimensional (3D) spatial information can be generated and applied to control the 3D motion of a robotic manipulator, and the real-time demonstration is successfully realized. With the facile design and very low-cost materials, the proposed SFTS shows great potential for applications in robotics control, touch screens, and electronic skins.