3D Spatial Mapping Research Articles

Chromatic covalent organic frameworks enabling in-vivo chemical tomography

Covalent organic frameworks designed as chromatic sensors offer opportunities to probe biological interfaces, particularly when combined with biocompatible matrices. Particularly compelling is the prospect of chemical tomography – or the 3D spatial mapping of chemical detail within the complex environment of living systems. Herein, we demonstrate a chromic Covalent Organic Framework (COF) integrated within silk fibroin (SF) microneedles that probe plant vasculature, sense the alkalization of vascular fluid as a biomarker for drought stress, and provide a 3D in-vivo mapping of chemical gradients using smartphone technology. A series of Schiff base COFs with tunable pKa ranging from 5.6 to 7.6 enable conical, optically transparent SF microneedles with COF coatings of 120 to 950 nm to probe vascular fluid and the surrounding tissues of tobacco and tomato plants. The conical design allows for 3D mapping of the chemical environment (such as pH) at standoff distances from the plant, enabling in-vivo chemical tomography. Chromatic COF sensors of this type will enable multidimensional chemical mapping of previously inaccessible and complex environments.

Automated Characterization of Macroscale Tissue 3D Spatial Material Properties via Robotically-Directed Impedimetric Sensing

Here, we report a novel method for automated characterization of bulk tissue 3D spatial properties based on reverse engineering-driven non-planar tool path planning and robotically-directed sensing. The method incorporates information on object (e.g., tissue) and inspection tool (e.g., sensor) geometry for automated inspection of tissue mechanical and dielectric properties across macroscopic nonplanar domains as large as 44 cm <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{2}$</tex-math> </inline-formula> . The process avoids the need for manual sensor-tissue integration processes. The impact and the utility of the method were demonstrated by automated mapping of 3D spatial distributions of mechanical and dielectric properties of plant and animal tissues using multiple complementary impedimetric-based sensors of varying types and form factor, including rigid micro-electromechanical systems (MEMS) and flexible multi-functional fibers. Applications to automated characterization of food quality (e.g., type and age) are provided, including 3D spatial mapping of plant and animal tissue mechanical and dielectric property distributions. Ultimately, automated methods for 3D spatial inspection of plant and animal tissue properties are critical to agriculture, food processing, organ transplantation, and biomanufacturing industries. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article is motivated by the need for automating the inspection of soft 3D biological objects. Here, we present a method that generates 3D quality maps of tissue properties using several sensors. The proposed tool path planning program outputs a customized tissue-conforming path for inspection based on the topographical features of the tissue. Hence, this method enables high-throughput, spatially-resolved, minimally-invasive, and reliable inspection of soft 3D biological objects. Applications to inspection of food quality were provided using two impedimetric-based sensors. Practitioners can directly apply the framework to inspection of other tissue properties and soft objects. This work provides an advance in automated methods for inspection, and real-time monitoring of tissue 3D property distributions, which reduces the need for manual tissue handling and requirement for sensor-product interface prior to characterization. This work can be implemented in various biomanufacturing applications and industries, including food safety and quality control, tissue engineering, and organ transplantation, to ensure the quality and safety of macroscopic tissue-engineered medical and food products.

Placing Value in Domestic Interiors

This article explores the interplay between place and value in the display of art in domestic spaces in seventeenth-century Amsterdam, using the painting collection of Pieter de Graeff (1638-1707) and Jacoba Bicker (1640-1695) as a case study. With the aid of a schematic 3D reconstruction of their house, based on De Graeff’s inventory and other sources, this research brings these artworks back to their original domestic context, testing hypotheses regarding their placement and visual impact while investigating display patterns. The 3D spatial mapping of De Graeff’s inventory connects the paintings’ monetary values with their location, and helps evaluate the symbolic and emotional values attributed to family portraits and other artworks. The article is accompanied by a methodological section in the HTML format detailing the 3D reconstruction process and sources.

INSIHGT: A scalable and homogeneous 3D multiplexed molecular mapping platform for spatial biology and precision medicine.

e15180 Background: The dynamic interplay between cancer cells and their microenvironment is a determinant of disease evolution and response to treatment. However, the current state of three-dimensional (3D) tumor microenvironment analysis is hindered by inadequate penetration of antibodies into thick tissue sections, leading to a substantial gap in acquiring unbiased, precise 3D multiplexed immunohistochemistry (IHC) images. Methods: We developed INSIHGT to surmount these limitations by enabling multi-round, multiplexed 3D IHC in standard formalin-fixed tissues. This fully automatable, fast-track method requires a simple series of chemical cocktail exchanges, translating to a tissue-to-image time of 6 days for 1 cm³ samples and as little as 6 hours for biopsy specimens. INSIHGT offers a scalable and efficient solution for detailed 3D spatial mapping of cancerous cells and their interactions within their native context. Results: 3D IHC with INSIHGT shows ideal penetration homogeneity, which quantitatively matches reference signals in 1cm3 samples. We further validated INSIHGT with 309 commercially available primary antibodies and is compatible with simultaneous fluorescent in situ hybridization (FISH) to detect RNA transcripts or DNA chromosomal abnormalities. Notably, the quality of post-INSIHGT traditional histological analyses, such as H&amp;E and standard IHC, was not altered. We demonstrated a 3D mapping of over 20 markers in a 1 mm-thick tumor sample across seven INSIHGT cycles, and an atlas of multiple tumour samples with 3D IHC &amp; H&amp;E. The large datasets seamlessly integrates with single-cell segmentation and analysis workflows, yielding comprehensive maps that elucidate protein expression, cell morphology, spatial positioning, and potential cell-cell interactions. Conclusions: INSIHGT provides an unparalleled level of visualization and analytical detail concerning the spatial dynamics within the tumor microenvironment at a single-cell resolution. As a transformative next generation histopathology platform, it can uncover new structural and molecular details within clinical specimens, accelerating biomarker discovery and novel diagnostic assays development, thereby enriching the field of tumor biology and propelling the progress of personalized oncology treatments.

3D Radiation Mapping Using Gaussian Process Regression with Intensity Projection

This article presents a novel approach for generating a three‐dimensional radiation map using data collected by mobile robots, aimed at monitoring radiation distribution in environments such as nuclear power plants. The proposed approach leverages Gaussian process regression with a novel adaptation of the inverse square law as a kernel function, which accurately reflects the physical characteristics of radiation, enabling precise mapping from uncertain data. Additionally, a method is proposed for constructing a comprehensive radiation map in 3D environments by estimating the radiation source and project the radiation data from sparse data. The effectiveness of the methodology is validated through simulations and experiments. Utilizing Octomap, a 3D spatial mapping tool, the study not only successfully visualizes radiation distribution in complex settings with multiple sources but also quantitatively demonstrates the enhanced accuracy of our approach compared to other existing methods. This research offers contributions in managing radiation risks, providing essential insights into the field.

Laboratory-based non-destructive 3D spatial mapping of phases and orientations

Laboratory-based non-destructive 3D spatial mapping of phases and orientations

Using 3D CNN for classification of Parkinson’s disease from resting-state fMRI data

Parkinson’s disease is a chronic and progressive movement disorder caused by the degeneration of dopamine-producing neurons in the substantia nigra of the brain. Currently, there is no specific diagnostic test available for Parkinson’s disease, and physicians rely on symptoms and medical history for diagnosis. In this study, a 3D-CNN deep learning model is proposed for detecting Parkinson’s disease using 4D-fMRI data. The data is preprocessed using independent component analysis (ICA) and dual regression processes through MELODIC in FSL, which results in a sequence of 30 3D spatial maps, each with its unique time course. A reference network, referred to as an atlas, is then applied using the fslcc command in FSL to map the 3D spatial maps. Fourteen resting-state networks (RSNs) are identified successfully, while the remaining maps are rejected as noise or artifacts. The detected RSNs or 3D spatial maps are fed into the 3D-CNN model, which is trained with a 10-fold cross-validation method. The proposed model has an accuracy of 86.07% on average.

Scattering Attenuation Images of the Control of Thrusts and Fluid Overpressure on the 2016–2017 Central Italy Seismic Sequence

AbstractDeep fluid circulation likely triggered the large extensional events of the 2016–2017 Central Italy seismic sequence. Nevertheless, the connection between fault mechanisms, main crustal‐scale thrusts, and the circulation and interaction of fluids with tectonic structures controlling the sequence is still debated. Here, we show that the 3D temporal and spatial mapping of peak delays, proxy of scattering attenuation, detects thrusts and sedimentary structures and their control on fluid overpressure and release. After the mainshocks, scattering attenuation drastically increases across the hanging wall of the Monti Sibillini and Acquasanta thrusts, revealing fracturing and fluid migration. Before the sequence, low‐scattering volumes within Triassic formations highlight regions of fluid overpressure, which enhances rock compaction. Our results highlight the control of thrusts and paleogeography on the sequence and hint at the monitoring potential of the technique for the seismic hazard assessment of the Central Apennines and other tectonic regions.

Automatic detection and classification of treatment deviations in proton therapy from realistically simulated prompt gamma imaging data.

A clinical study regarding the potential of range verification in proton therapy (PT) by prompt gamma imaging (PGI) is carried out at our institution. Manual interpretation of the detected spot-wise range shift information is time-consuming, highly complex, and therefore not feasible in a broad routine application. Here, we present an approach to automatically detect and classify treatment deviations in realistically simulated PGI data for head-and-neck cancer (HNC) treatments using convolutional neural networks (CNNs) and conventional machine learning (ML) approaches. For 12 HNC patients and 1 anthropomorphic head phantom (n=13), pencil beam scanning (PBS) treatment plans were generated, and 1 field per plan was assumed to be monitored with a PGI slit camera system. In total, 386 scenarios resembling different relevant or non-relevant treatment deviations were simulated on planning and control CTs and manually classified into 7 classes: non-relevant changes (NR) and relevant changes (RE) triggering treatment intervention due to range prediction errors (±RP), setup errors in beam direction (±SE), anatomical changes (AC), or a combination of such errors (CB). PBS spots with reliable PGI information were considered with their nominal Bragg peak position for the generation of two 3D spatial maps of 16× 16×16 voxels containing PGI-determined range shift and proton number information. Three complexity levels of simulated PGI data were investigated: (I) optimal PGI data, (II) realistic PGI data with simulated Poisson noise based on the locally delivered proton number, and (III) realistic PGI data with an additional positioning uncertainty of the slit camera following an experimentally determined distribution. For each complexity level, 3D-CNNs were trained on a data subset (n=9) using patient-wise leave-one-out cross-validation and tested on an independent test cohort (n=4). Both the binary task of detecting RE and the multi-class task of classifying the underlying error source were investigated. Similarly, four different conventional ML classifiers (logistic regression, multilayer perceptron, random forest, and support vector machine) were trained using five previously established handcrafted features extracted from the PGI data and used for performance comparison. On the test data, the CNN ensemble achieved a binary accuracy of 0.95, 0.96, and 0.93 and a multi-class accuracy of 0.83, 0.81, and 0.76 for the complexity levels (I), (II), and (III), respectively. In the case of binary classification, the CNN ensemble detected treatment deviations in the most realistic scenario with a sensitivity of 0.95 and a specificity of 0.88. The best performing ML classifiers showed a similar test performance. This study demonstrates that CNNs can reliably detect relevant changes in realistically simulated PGI data and classify most of the underlying sources of treatment deviations. The CNNs extracted meaningful features from the PGI data with a performance comparable to ML classifiers trained on previously established handcrafted features. These results highlight the potential of a reliable, automatic interpretation of PGI data for treatment verification, which is highly desired for a broad clinical application and a prerequisite for the inclusion of PGI in an automated feedback loop for online adaptive PT.

CT-Loc: Cross-domain visual localization with a channel-wise transformer

We tackle the cross-domain visual localization problem of estimating camera position and orientation from real images without three-dimensional (3D) spatial mapping or modeling. Recent studies have shown suboptimal performance in this task owing to the photometric and geometric differences between synthetic and real images. In this study, we present a deep learning approach that uses a channel-wise transformer localization (CT-Loc) framework. Inspired by the human behavior of looking for structural landmarks to estimate one’s location, CT-Loc encodes the most salient features of task-relevant objects in target scenes. To evaluate the efficacy of the proposed method in a real-world application, we built a complex and large-scale dataset of the interior of the mechanical room during operations and conducted extensive performance comparisons with the publicly available state-of-the-art University of Melbourne Corridor and Virtual KITTI 2 datasets. Compared with the otherwise best-performing BIM-PoseNet indoor camera localization model, our method significantly reduces position and orientation errors through the application of attention weights and saliency maps while also learning only the visual structural patterns (e.g., floors and doors) that are most relevant to localization tasks. Our model successfully ignores uninformative objects. This approach yields higher-level robust camera-pose regression localization results without requiring prebuilt maps. The code is available at https://github.com/kdaeho27/CT-Loc.

A Novel Approach to 3D Spatial Mapping of the Human Hematopoietic Microenvironment in Normal and Diseased Bone Marrow

The bone marrow hematopoietic microenvironment (HME) plays a pivotal role in regulating normal and diseased hematopoiesis. Important cellular components of the human (HME) include the bone marrow (BM) stromal cells (Li, BioRxive 2022) and other cell types, such as vascular endothelial cells (ECs), and megakaryocytes (MKs). Perturbations of the HME can affect normal and diseased hematopoiesis. In Philadelphia-negative myeloproliferative neoplasms (MPNs), which include Essential Thrombocythemia (ET), Polycythemia Vera (PV), Primary Myelofibrosis (PMF), and its precursor, pre-fibrotic MF (pre-PMF), the HME is of critical importance for disease development and progression (Mead, Blood 2017). However, despite this important role, little is known about the spatial organization of the different HME elements in MPNs. Therefore, we aimed to develop a novel method of immunofluorescence (IF) multiplexing with markers for HME and hematopoietic cells based on sequential staining and bleaching in combination with scanning IF microscopy and standard imaging analysis (Fig.1). Herein we show that this approach allowed us to generate 3D reconstitutions of standard biobank-stored BM samples from MPN patients and age-matched hematopoietically normal controls. Furthermore, we demonstrate how the free and open-source 3D computer graphics software toolset Blender (Stichting Blender Foundation, Amsterdam) can be utilized to analyze cell distributions and spatial relationships in 3D. BM biopsies from patients with MPN and controls were stained sequentially for CD31, CD34, CD45, and CD271 with repetitive bleaching steps to realize five color images with DAPI as a nuclear stain. Twelve subsequent slides per sample were stacked to create three-dimensional BM reconstructions with the imaging program Arivis Vision 4D. Iso-surfaces for niche cells and structures were created and exported as mesh objects for spatial distribution analysis in Blender. We recapitulated the BM architecture using this approach and produced comprehensive 3D models of endosteal and perivascular BM niches. MPN bone marrows displayed considerable differences between the different MPN types, especially concerning megakaryocyte (MK) morphology and distribution, as well as CD271 staining density and CD271+ cell numbers. Whereas ET and pre-PMF were comparable to their paired normal control, we found that PMF and especially PV samples showed considerably increased numbers and densities of CD271+ cells. In PMF the generated MSC objects formed a thick fibrillary structure reminiscent of classic fibrosis staining that was particularly dense close to vessels, bone surfaces, and MKs. Measurements of the spatial relationships of MKs and hematopoietic stem and progenitor cells (HSPC) to vessels and bone structures, respectively, revealed the most pronounced differences in PV, with median distances of PV-MKs to vessels being significantly lower than in controls (PV 7.62 µm vs 8.20 µm control, p<0.05). Also, the median distance in PV of HSPCs to vessels (2,88 µm) and bones (52.23 µm) was less than in the controls (46.31 µm and 236.7 µm, respectively; p<0.01). Taken together, we report a novel sequential staining and bleaching technique for multi-color analysis of normal and diseased BMs. The three-dimensional reconstruction technique developed herein revealed important aspects of the spatial HME composition in MPNs and normal BMs and provides a platform for a deeper investigation of BM pathophysiology, for example by increasing numbers of antibodies and including visualization of the expression of relevant genes and proteins. Figure 1View largeDownload PPTFigure 1View largeDownload PPT Close modal

Ultrasound-guided microbubble-mediated brain therapy with a modular transmit/receive phased array

The development of low-cost, ultrasound-guided focused ultrasound (USgFUS) treatment platforms is expected to advance the adoption of microbubble (MB)-mediated brain therapy by improving access to the technology. Our group has designed and fabricated clinical-scale transmit/receive phased array systems for MB-mediated USgFUS brain therapy. Acoustic field simulations were carried out to optimize array element placement, and transducer scaffolds were constructed using 3D printing techniques. These devices have been employed for skull computed tomography-array registration, 3D spatial mapping of MB activity in vivo through ex-vivo human skullcaps via noninvasive aberration correction methods, and we have harnessed this spatiotemporal cavitation information to calibrate exposure levels for safe volumetric blood-brain barrier opening. At higher exposure levels, we have demonstrated the ability of 3D MB imaging data to predict the tissue damage volume distributions induced during nonthermal brain ablation. Ultrafast processing of acoustic emissions data has been shown to uncover MB dynamics hidden by conventional whole-burst temporal averaging, and can inform temporal undersampling strategies when millisecond-long tone bursts are applied. Machine learning approaches can assist with image-based classification of MB activity, which may result in finer control of the induced bioeffects. This talk will focus on our recent results obtained with a novel modular USgFUS phased array system.

An Adversarial Generative Network Designed for High-Resolution Monocular Depth Estimation from 2D HiRISE Images of Mars

In computer vision, stereoscopy allows the three-dimensional reconstruction of a scene using two 2D images taken from two slightly different points of view, to extract spatial information on the depth of the scene in the form of a map of disparities. In stereophotogrammetry, the disparity map is essential in extracting the digital terrain model (DTM) and thus obtaining a 3D spatial mapping, which is necessary for a better analysis of planetary surfaces. However, the entire reconstruction process performed with the stereo-matching algorithm can be time consuming and can generate many artifacts. Coupled with the lack of adequate stereo coverage, it can pose a significant obstacle to 3D planetary mapping. Recently, many deep learning architectures have been proposed for monocular depth estimation, which aspires to predict the third dimension given a single 2D image, with considerable advantages thanks to the simplification of the reconstruction problem, leading to a significant increase in interest in deep models for the generation of super-resolution images and DTM estimation. In this paper, we combine these last two concepts into a single end-to-end model and introduce a new generative adversarial network solution that estimates the DTM at 4× resolution from a single monocular image, called SRDiNet (super-resolution depth image network). Furthermore, we introduce a sub-network able to apply a refinement using interpolated input images to better enhance the fine details of the final product, and we demonstrate the effectiveness of its benefits through three different versions of the proposal: SRDiNet with GAN approach, SRDiNet without adversarial network, and SRDiNet without the refinement learned network plus GAN approach. The results of Oxia Planum (the landing site of the European Space Agency’s Rosalind Franklin ExoMars rover 2023) are reported, applying the best model along all Oxia Planum tiles and releasing a 3D product enhanced by 4×.

Generating 3D Geothermal Maps in Catalonia, Spain Using a Hybrid Adaptive Multitask Deep Learning Procedure

Mapping the subsurface temperatures can efficiently lead to identifying the geothermal distribution heat flow and potential hot spots at different depths. In this paper, an advanced adaptive multitask deep learning procedure for 3D spatial mapping of the subsurface temperature was proposed. As a result, predictive 3D spatial subsurface temperatures at different depths were successfully generated using geolocation of 494 exploratory boreholes data in Catalonia (Spain). To increase the accuracy of the achieved results, hybridization with a new modified firefly algorithm was carried out. Subsequently, uncertainty analysis using a novel automated ensemble deep learning approach for the predicted temperatures and generated spatial 3D maps were executed. Comparing the accuracy performances in terms of correct classification rate (CCR) and the area under the precision–recall curves for validation and whole datasets with at least 4.93% and 2.76% improvement indicated for superiority of the hybridized model. According to the results, the efficiency of the proposed hybrid multitask deep learning in 3D geothermal characterization to enhance the understanding and predictability of subsurface spatial distribution of temperatures is inferred. This implies that the applicability and cost effectiveness of the adaptive procedure in producing 3D high resolution depth dependent temperatures can lead to locate prospective geothermally hotspot active regions.

3D Spatial Mapping of the Nanomorphology of Polymer:Fullerene Blends by Highly Selective, Homogeneous Copper Staining

3D Spatial Mapping of the Nanomorphology of Polymer:Fullerene Blends by Highly Selective, Homogeneous Copper Staining

Skin‐Inspired High‐Performance Active‐Matrix Circuitry for Multimodal User‐Interaction

AbstractArtificial electronic skin (e‐skin), a network of mechanically flexible sensors which can wrap irregular surfaces conformally and quantify various stimuli sensitively, is potentially useful in healthcare monitoring and human‐machine interaction (HMI). Although various approaches have mimicked the structures and functions of the human skin, challenges remain with high‐density integration, super sensitivity, and multi‐functionality. A multimodal and comfortable skin‐inspired active‐matrix circuitry is reported here with high pixel density (&gt;100 cm–2) based on all 2D materials, which exhibits excellent performance to detect both mechanical interactions and humidity variations. The ultra‐high sensitivity (&gt;400 and ≈ 104 for strain and humidity sensing, respectively), long‐term stability (&gt;1000 cycles), and rapid response time for every pixel can fulfill simultaneous multi‐stimulus sensing. Accordingly, a respiratory monitor is constructed to realize healthcare monitoring through observing the human breath frequency, intensity, and humidity in real‐time. Moreover, the multimodal e‐skin breaks through shackles of the contact sensor medium for HMI. 3D strain and humidity spatial mapping can reflect object location information even without contact, avoiding cross‐infection of viruses effectively between users during the COVID‐19 pandemic. The reported e‐skin will broaden applications for future healthcare and human–machine interactive devices.

Unravelling Spatial Privacy Risks of Mobile Mixed Reality Data

Previously, 3D data---particularly, spatial data---have primarily been utilized in the field of geo-spatial analyses, or robot navigation (e.g. self-automated cars) as 3D representations of geographical or terrain data (usually extracted from lidar). Now, with the increasing user adoption of augmented, mixed, and virtual reality (AR/MR/VR; we collectively refer to as MR) technology on user mobile devices, spatial data has become more ubiquitous. However, this ubiquity also opens up a new threat vector for adversaries: aside from the traditional forms of mobile media such as images and video, spatial data poses additional and, potentially, latent risks to users of AR/MR/VR. Thus, in this work, we analyse MR spatial data using various spatial complexity metrics---including a cosine similarity-based, and a Euclidean distance-based metric---as heuristic or empirical measures that can signify the inference risk a captured space has. To demonstrate the risk, we utilise 3D shape recognition and classification algorithms for spatial inference attacks over various 3D spatial data captured using mobile MR platforms: i.e. Microsoft HoloLens, and Android with Google ARCore. Our experimental evaluation and investigation shows that the cosine similarity-based metric is a good spatial complexity measure of captured 3D spatial maps and can be utilised as an indicator of spatial inference risk.

Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

This study proposes a novel hands-free interaction method using multimodal gestures such as eye gazing and head gestures and deep learning for human-robot interaction (HRI) in mixed reality (MR) environments. Since human operators hold some objects for conducting tasks, there are many constrained situations where they cannot use their hands for HRI interactions. To provide more effective and intuitive task assistance, the proposed hands-free method supports coarse-to-fine interactions. Eye gazing-based interaction is used for coarse interactions such as searching and previewing of target objects, and head gesture interactions are used for fine interactions such as selection and 3D manipulation. In addition, deep learning-based object detection is applied to estimate the initial positioning of physical objects to be manipulated by the robot. The result of object detection is then combined with 3D spatial mapping in the MR environment for supporting accurate initial object positioning. Furthermore, virtual object-based indirect manipulation is proposed to support more intuitive and efficient control of the robot, compared with traditional direct manipulation (e.g., joint-based and end effector-based manipulations). In particular, a digital twin, the synchronized virtual robot of the real robot, is used to provide a preview and simulation of the real robot to manipulate it more effectively and accurately. Two case studies were conducted to confirm the originality and advantages of the proposed hands-free HRI: (1) performance evaluation of initial object positioning and (2) comparative analysis with traditional direct robot manipulations. The deep learning-based initial positioning reduces much effort for robot manipulation using eye gazing and head gestures. The object-based indirect manipulation also supports more effective HRI than previous direct interaction methods.

Deep learning-based mobile augmented reality for task assistance using 3D spatial mapping and snapshot-based RGB-D data

This paper proposes a new deep learning-based mobile AR for intelligent task assistance by conducting 3D spatial mapping without pre-registration using AR markers, which can match virtual AR objects to their corresponding physical objects automatically and accurately using single snapshot-based RGB-D data. Firstly, the proposed approach applies a deep learning-based instance segmentation method to the snapshot-based RGB-D data to detect real object instances and to segment their surrounding regions in 3D point cloud data. Then, an iterative closest point (ICP) algorithm is used to perform a 3D spatial mapping between the segmented point cloud of the real object and its corresponding virtual model. Therefore, the virtual information can be seamlessly and automatically synchronized with its corresponding real object. To prove the effectiveness of the proposed method, we performed comparative experiments quantitatively and qualitatively, which evaluated the accuracy, basic task performance, and usability. Experimental results verify that the proposed deep learning-based 3D spatial mapping approach is more accurate and more suitable for mobile AR-based visualization and interaction than previous studies. We have also implemented several applications in actual working situations, which verifies the applicability and extensibility of the proposed approach.

Characterization of orthotropic nature of cleavage planes in granitic rock

Granite naturally exhibits anisotropic elasticity because of the alignment of minerals formed in the three orthogonal cleavage planes known as rift, grain, and hardway planes. Of critical importance for studying cleavage planes is to understand how they affect the spatial distribution of anisotropy. Here we use multiple approaches which measure tensile strength, ultrasonic wave velocity, and three-dimensional (3D) X-ray computed tomography (CT) imaging to elucidate the existence of cleavage. Our results show that essential features of cleavage-induced anisotropy, e.g., orientation of the failure mode and directional dependency of elastic constants, can be identified with high precision. Furthermore, the results highlight that the proposed CT image technique can achieve 3D spatial mapping of anisotropy, providing a causal link between the cleavage planes and the spatial distribution of anisotropy in granite. This comprehensive understanding through the described techniques could have practical uses in predicting the fracture path and potential flow pathways for petroleum, groundwater, natural gas, etc.