Fine-grained 3D Research Articles

RS-TNet: point cloud transformer with relation-shape awareness for fine-grained 3D visual processing

RS-TNet: point cloud transformer with relation-shape awareness for fine-grained 3D visual processing

Fine-Grained 3D Model Classification Based on Deep Ensemble and Detail Awareness

Deep learning based 3D model classification methods have poor effectiveness in fine-grained 3D model classification. Aiming at the problem, an end-to-end fine-grained 3D model classification framework is proposed, and a network based on deep ensemble learning network and context detail awareness module (CDAM) is constructed. Inputting the multiply views of a 3D model, the global shape features are extracted through the deep ensemble learning sub-network. And the local detail features are obtained through the auxiliary sub-network based on CDAM. Based on above two sub-networks, an end-to-end weakly supervision fine-grained 3D model classification network is constructed. Experiments are conducted on three sub-datasets with different levels of difficulties, Airplane, Chair and Car, from the public dataset FG3D. The classification accuracies for above three sub-datasets are 96.31%, 85.44% and 79.62% respectively, which demonstrate the fine classification performance and more generalization of the proposed method.

3D reconstruction and characterization of reef limestone pores based on optical and acoustic microscopic images

The pore network in reef limestone provides an important resource for fluid storage and flow. Three-dimensional (3D) reconstruction and characterization of the spatial structure of reef limestone pores offers a valuable way of identifying its physical and mechanical properties. This paper reports on a study where optical and acoustic microscope images were used to combine the advantages of the high resolution of optical testing with the high penetration capabilities of acoustic testing. This overcomes existing problems with multi-scale structural imaging of the pores in reef limestone caused by the small penetration depth of traditional optical imaging and the low resolution of ultrasonic imaging. First of all, a correlation function between the shallow and deep pore structure of reef limestone is established. This draws on high-resolution data from shallow optical microscopic images and deep acoustic microscopic images with a multi-layered cross-section. The contours of the shallow and deep pore structure are then digitized. After this, horizontal section control points and vertical transition contours are constructed that enable a deep integration of information about the multi-dimensional geometric morphology of the pore structure to be realized, providing in turn for a fine-grained 3D reconstruction of the pore structure at different scales. Finally, factors governing the characterization of contour similarity and center deviation are proposed that make it possible to generate spatial representations of the pore structure across multiple dimensions. A real-world case analysis is then used to verify the feasibility and reliability of the proposed method. The results show that the method can provide abundant data for the establishment of a reef limestone pore network and physical model, making it possible to improve the quality of 3D reconstructions of the pore structure and improve the 3D imaging of reef limestone pore networks. The proposed method can underpin new approaches to analysis of the pore structure of reef limestone and support a wide range of potential applications in the field of pore visualization and characterization.

Multi-sensor profiling for precision soil-moisture monitoring

Controlling soil moisture is crucial in optimizing watering and crop performance. Traditional monitoring systems rely on a single sensor or on a column of sensors that do not allow farmers to properly capture soil moisture dynamics in the soil volume occupied by roots. In this paper we propose PLUTO, an original approach that builds fine-grained 2D and 3D soil moisture profiles by relying on a grid of sensors. Profiles are computed using both interpolation-based and machine learning approaches. Besides the technical description of the approach, the paper reports a set of original visualizations and a large set of tests computed, over two years, on real Kiwi orchards. PLUTO proved to largely overcome the accuracy of profiles obtained with traditional sensor layouts. Considering that the cost of sensors is progressively decreasing, PLUTO provides a cost-effective, operative, and precise solution to moisture monitoring.

Autonomous UAVs landing site selection from point cloud in unknown environments

Autonomous safe landing of UAVs is an important and challenging task in unknown environments, as almost no prior scene information can be leveraged for navigation. Most existing methods cannot address this issue completely, due to terrain uncertainty and system complexity. In this paper, we present a novel and complete framework for UAVs landing, which is built on point cloud in coarse-to-fine manner. Besides, our framework is designed with modularity and has four modules: point cloud preprocessing, coarse landing site selection, fine terrain evaluation, and landing optimal model. Specifically, a composite preprocessing scheme is applied to simultaneously filter noise, generate 3D Octo-map and plan the path on the raw point cloud. To balance the accuracy and real-time of the landing system, only promising coarse landing locations are automatically selected by adopting the proposed multi-stage process in grid-map. Based on the result of coarse stage, a fine-grained 3D validation is modeled by multiple terrain factors, which can further improve landing safety. Finally, a novel landing optimal model fuses terrain condition, fuel consumption, and second landing validation to determine the final landing sites during descent. Extensive experiments have been successfully conducted on different real-world and unknown environments, verifying that our method can select safe landing sites for UAVs robustly. Additionally, the system is further evaluated under normal, emergency, and rescue situations respectively to highlight different landing requirements.

Joint Learning of 3D Lesion Segmentation and Classification for Explainable COVID-19 Diagnosis.

Given the outbreak of COVID-19 pandemic and the shortage of medical resource, extensive deep learning models have been proposed for automatic COVID-19 diagnosis, based on 3D computed tomography (CT) scans. However, the existing models independently process the 3D lesion segmentation and disease classification, ignoring the inherent correlation between these two tasks. In this paper, we propose a joint deep learning model of 3D lesion segmentation and classification for diagnosing COVID-19, called DeepSC-COVID, as the first attempt in this direction. Specifically, we establish a large-scale CT database containing 1,805 3D CT scans with fine-grained lesion annotations, and reveal 4 findings about lesion difference between COVID-19 and community acquired pneumonia (CAP). Inspired by our findings, DeepSC-COVID is designed with 3 subnets: a cross-task feature subnet for feature extraction, a 3D lesion subnet for lesion segmentation, and a classification subnet for disease diagnosis. Besides, the task-aware loss is proposed for learning the task interaction across the 3D lesion and classification subnets. Different from all existing models for COVID-19 diagnosis, our model is interpretable with fine-grained 3D lesion distribution. Finally, extensive experimental results show that the joint learning framework in our model significantly improves the performance of 3D lesion segmentation and disease classification in both efficiency and efficacy.

Quantifying the Impact of Monolithic 3D (M3D) Integration on L1 Caches

Monolithic 3D (M3D) integration has been recently introduced as a viable solution for fine-grained 3D integration. Since the conventional 3D integration uses relatively large micro-scale through-silicon-vias (TSVs), which causes large TSV area overhead, it is not cost-effective for small micro architectural blocks such as L1 caches. On the contrary, the M3D integration offers nano-scale monolithic inter-tier vias (MIVs) which are much smaller than TSVs. Thus, the M3D integration is known to be even feasible for 3D stacking of small micro architectural blocks, which reduces wire length of the blocks, leading to better performance and energy-efficiency. In this paper, we quantify the architectural impact (in terms of performance, power, temperature, and area) of the M3D integration for L1 caches. In our evaluation, the 8-layer stacked M3D L1 caches show 34.1~43.2 percent shorter access time than the 2D L1 cache. As a result, the M3D L1 caches improve the performance of SPEC CPU 2006 applications by 9.9 percent (up to 43.7 percent), on average, compared to the conventional 2D L1 caches. Additionally, the 8-layer stacked M3D L1 caches reduce dynamic energy and leakage power by 58.9 percent ~60.8 percent and 57.9~59.1 percent, respectively, compared to the 2D L1 cache. Additionally, though 3D stacking inevitably causes higher temperature than 2D baseline, since the M3D integration provides better heat dissipation as well as lower power consumption than the conventional TSV-3D, it reduces peak L1 cache temperature by up to 7.6°C, compared to the TSV-3D.

A System-Level Exploration of Binary Neural Network Accelerators with Monolithic 3D Based Compute-in-Memory SRAM

Binary neural networks (BNNs) are adequate for energy-constrained embedded systems thanks to binarized parameters. Several researchers have proposed the compute-in-memory (CiM) SRAMs for XNOR-and-accumulation computations (XACs) in BNNs by adding additional transistors to the conventional 6T SRAM, which reduce the latency and energy of the data movements. However, due to the additional transistors, the CiM SRAMs suffer from larger area and longer wires than the conventional 6T SRAMs. Meanwhile, monolithic 3D (M3D) integration enables fine-grained 3D integration, reducing the 2D wire length in small functional units. In this paper, we propose a BNN accelerator (BNN_Accel), composed of a 9T CiM SRAM (CiM_SRAM), input buffer, and global periphery logic, to execute the computations in the binarized convolution layers of BNNs. We also propose CiM_SRAM with the subarray-level M3D integration (as well as the transistor-level M3D integration), which reduces the wire latency and energy compared to the 2D planar CiM_SRAM. Across the binarized convolution layers, our simulation results show that BNN_Accel with the 4-layer CiM_SRAM reduces the average execution time and energy by 39.9% and 23.2%, respectively, compared to BNN_Accel with the 2D planar CiM_SRAM.

Massive MIMO for Cellular-Connected UAV: Challenges and Promising Solutions

Massive multiple-input multiple-output (MIMO) is a promising technology for enabling cellu-lar-connected unmanned aerial vehicle (UAV) communications in the future. Equipped with full-dimensional large arrays, ground base stations (GBSs) can apply adaptive fine-grained 3D beamforming to mitigate the strong interference between high-altitude UAVs and low-altitude terrestrial users, thus significantly enhancing the network spectral efficiency. However, the performance gain of massive MIMO critically depends on accurate channel state information of both UAVs and terrestrial users at the GBSs, which is practically difficult to achieve due to UAV-in-duced pilot contamination and UAVs' high mobility in 3D. Moreover, the increasingly popular applications relying on a large group of coordinated UAVs or UAV swarm as well as the practical hybrid GBS beamforming architecture for massive MIMO further complicate the pilot contamination and channel/beam tracking problems. In this article, we provide an overview of the above challenging issues, propose new solutions to cope with them, and discuss promising directions for future research. Preliminary simulation results are also provided to validate the effectiveness of the proposed solutions.

RETRACTED: Fast 3D image reconstruction algorithm based on artificial intelligence technology

In order to improve the 3D reconstruction capability of high-resolution fine-grained 3D images, a fast 3D image reconstruction algorithm based on artificial intelligence technology is proposed. The cross-gradient sharpening detection method is used to collect features and extract information from high-resolution fine-grained three-dimensional images, and establish an edge contour feature detection model for high-resolution fine-grained three-dimensional images. Combining the salient feature analysis method and the subspace feature analysis method to cluster and analyze the high-resolution fine-grained three-dimensional image. In the artificial intelligence environment, the saliency of the three-dimensional image is detected and analyzed, and the multi-dimensional segmentation and gray histogram of the high-resolution fine-grained three-dimensional image are reconstructed through the subspace segmentation method. According to the reconstruction results of the gray histogram, fast 3D image reconstruction and image fusion processing are performed. Finally, the accurate detection and recognition of the reconstructed image is realized. The simulation results show that this method has a good effect on 3D image reconstruction, and the time cost of image reconstruction is relatively short. It improves the recognition and feature analysis capabilities of high-resolution fine-grained 3D images, and has good application value in the reconstruction, detection and recognition of high-resolution fine-grained 3D images.

Toward Fine-Grained Sketch-Based 3D Shape Retrieval.

In this paper we study, for the first time, the problem of fine-grained sketch-based 3D shape retrieval. We advocate the use of sketches as a fine-grained input modality to retrieve 3D shapes at instance-level - e.g., given a sketch of a chair, we set out to retrieve a specific chair from a gallery of all chairs. Fine-grained sketch-based 3D shape retrieval (FG-SBSR) has not been possible till now due to a lack of datasets that exhibit one-to-one sketch-3D correspondences. The first key contribution of this paper is two new datasets, consisting a total of 4,680 sketch-3D pairings from two object categories. Even with the datasets, FG-SBSR is still highly challenging because (i) the inherent domain gap between 2D sketch and 3D shape is large, and (ii) retrieval needs to be conducted at the instance level instead of the coarse category level matching as in traditional SBSR. Thus, the second contribution of the paper is the first cross-modal deep embedding model for FG-SBSR, which specifically tackles the unique challenges presented by this new problem. Core to the deep embedding model is a novel cross-modal view attention module which automatically computes the optimal combination of 2D projections of a 3D shape given a query sketch.

Monolithic 3D-Based SRAM/MRAM Hybrid Memory for an Energy-Efficient Unified L2 TLB-Cache Architecture

Monolithic 3D (M3D) integration has been emerged as a promising technology for fine-grained 3D stacking. As the M3D integration offers extremely small dimension of via in a nanometer-scale, it is beneficial for small microarchitectural blocks such as caches, register files, translation look-aside buffers (TLBs), etc. However, since the M3D integration requires low-temperature process for stacked layers, it causes lower performance for stacked transistors compared to the conventional 2D process. In contrast, non-volatile memory (NVM) such as magnetic RAM (MRAM) is originally fabricated at a low temperature, which enables the M3D integration without performance degradation. In this paper, we propose an energy-efficient unified L2 TLB-cache architecture exploiting M3D-based SRAM/MRAM hybrid memory. Since the M3D-based SRAM/MRAM hybrid memory consumes much smaller energy than the conventional 2D SRAM-only memory and 2D SRAM/MRAM hybrid memory, while providing comparable performance, our proposed architecture improves energy efficiency significantly. Especially, as our proposed architecture changes the memory partitioning of the unified L2 TLB-cache depending on the L2 cache miss rate, it maximizes the energy efficiency for parallel workloads suffering extremely high L2 cache miss rate. According to our analysis using PARSEC benchmark applications, our proposed architecture reduces the energy consumption of L2 TLB + L2 cache by up to 97.7% (53.6% on average), compared to the baseline with the 2D SRAM-only memory, with negligible impact on performance. Furthermore, our proposed technique reduces the memory access energy consumption by up to 32.8% (10.9% on average), by reducing memory accesses due to TLB misses.

Fine-Grained 3D Shape Classification With Hierarchical Part-View Attention.

Fine-grained 3D shape classification is important for shape understanding and analysis, which poses a challenging research problem. However, the studies on the fine-grained 3D shape classification have rarely been explored, due to the lack of fine-grained 3D shape benchmarks. To address this issue, we first introduce a new 3D shape dataset (named FG3D dataset) with fine-grained class labels, which consists of three categories including airplane, car and chair. Each category consists of several subcategories at a fine-grained level. According to our experiments under this fine-grained dataset, we find that state-of-the-art methods are significantly limited by the small variance among subcategories in the same category. To resolve this problem, we further propose a novel fine-grained 3D shape classification method named FG3D-Net to capture the fine-grained local details of 3D shapes from multiple rendered views. Specifically, we first train a Region Proposal Network (RPN) to detect the generally semantic parts inside multiple views under the benchmark of generally semantic part detection. Then, we design a hierarchical part-view attention aggregation module to learn a global shape representation by aggregating generally semantic part features, which preserves the local details of 3D shapes. The part-view attention module hierarchically leverages part-level and view-level attention to increase the discriminability of our features. The part-level attention highlights the important parts in each view while the view-level attention highlights the discriminative views among all the views of the same object. In addition, we integrate a Recurrent Neural Network (RNN) to capture the spatial relationships among sequential views from different viewpoints. Our results under the fine-grained 3D shape dataset show that our method outperforms other state-of-the-art methods. The FG3D dataset is available at https://github.com/liuxinhai/FG3D-Net.

Urban Outdoor Measurement Study of Phased Antenna Array Impact on Millimeter-Wave Link Opportunities and Beam Misalignment

Exploiting multi-antenna technologies for robust beamsteering to overcome the effects of blockage and beam misalignment is the key to providing seamless multi-Gbps connectivity in millimeter-wave (mm-wave) networks. In this paper, we present the first large-scale outdoor mm-wave measurement study using a phased antenna array in a typical European town. We systematically collect fine-grained 3D angle-of-arrival (AoA) and angle-of-departure (AoD) data, totaling over 50,000 received signal strength measurements. We study the impact of phased antenna arrays in terms of number of link opportunities, achievable data rate and robustness under small-scale mobility, and compare this against reference horn antenna measurements. Our results show a limited number of 2–4 distinct spatial link opportunities per receiver location, indicating that the mm-wave multipath richness in a European town is surprisingly similar to that of dense urban metropolises. The results for the phased antenna array reveal that significant losses in estimated data rate occur for beam misalignments in the order of the half-power beamwidth, with significant and irregular variations for larger misalignments. By contrast, the loss for horn antennas is monotonically increasing with the misalignment. Our results strongly suggest that the effect of non-ideal phased antenna arrays must be explicitly considered in the design of agile beamsteering algorithms.

A hardware-in-the-loop simulation platform for distributed UAV swarms

To improve the functional verification efficiency of intelligent UAV swarm systems, this paper proposes a distributed UAV swarm hardware-in-loop (HIL) simulation platform. The proposed platform uses fine-grained 3D scenes and real aircraft aerodynamic models as inputs to simulate the control and mission decision-making process of the UAV swarm by building a swarm data interaction network. The simulation platform is equipped with real on-board electrical equipments, which can fully verify the compatibility between software and hardware in the UAV swarm system and is able to shorten the research cycle effectively. The platform supports the addition and deletion of all-digital virtual simulation nodes and can be used for large-scale swarms. The tests on individual strap-down terminal guidance and multi-aircraft coordinated search show that the platform has good simulation results for features such as data interaction, autonomous decision-making, and control of UAV swarms.

Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation

Monocular 3D object detection task aims to predict the 3D bounding boxes of objects based on monocular RGB images. Since the location recovery in 3D space is quite difficult on account of absence of depth information, this paper proposes a novel unified framework which decomposes the detection problem into a structured polygon prediction task and a depth recovery task. Different from the widely studied 2D bounding boxes, the proposed novel structured polygon in the 2D image consists of several projected surfaces of the target object. Compared to the widely-used 3D bounding box proposals, it is shown to be a better representation for 3D detection. In order to inversely project the predicted 2D structured polygon to a cuboid in the 3D physical world, the following depth recovery task uses the object height prior to complete the inverse projection transformation with the given camera projection matrix. Moreover, a fine-grained 3D box refinement scheme is proposed to further rectify the 3D detection results. Experiments are conducted on the challenging KITTI benchmark, in which our method achieves state-of-the-art detection accuracy.

Towards Smart and Reconfigurable Environment: Intelligent Reflecting Surface Aided Wireless Network

IRS is a new and revolutionizing technology that is able to significantly improve the performance of wireless communication networks, by smartly reconfiguring the wireless propagation environment with the use of massive low-cost passive reflecting elements integrated on a planar surface. Specifically, different elements of an IRS can independently reflect the incident signal by controlling its amplitude and/or phase and thereby collaboratively achieve fine-grained 3D passive beamforming for directional signal enhancement or nulling. In this article, we first provide an overview of the IRS technology, including its main applications in wireless communication, competitive advantages over existing technologies, hardware architecture as well as the corresponding new signal model. We then address the key challenges in designing and implementing the new IRS-aided hybrid (with both active and passive components) wireless network, as compared to the traditional network comprising active components only. Finally, numerical results are provided to show the great performance enhancement with the use of IRS in typical wireless networks.

RealPoint3D: Generating 3D Point Clouds from a Single Image of Complex Scenarios

Generating 3D point clouds from a single image has attracted full attention from researchers in the field of multimedia, remote sensing and computer vision. With the recent proliferation of deep learning, various deep models have been proposed for the 3D point cloud generation. However, they require objects to be captured with absolutely clean backgrounds and fixed viewpoints, which highly limits their application in the real environment. To guide 3D point cloud generation, we propose a novel network, RealPoint3D, to integrate prior 3D shape knowledge into the network. Taking additional 3D information, RealPoint3D can handle 3D object generation from a single real image captured from any viewpoint and complex background. Specifically, provided a query image, we retrieve the nearest shape model from a pre-prepared 3D model database. Then, the image, together with the retrieved shape model, is fed into RealPoint3D to generate a fine-grained 3D point cloud. We evaluated the proposed RealPoint3D on the ShapeNet dataset and ObjectNet3D dataset for the 3D point cloud generation. Experimental results and comparisons with state-of-the-art methods demonstrate that our framework achieves superior performance. Furthermore, our proposed framework works well for real images in complex backgrounds (the image has the remaining objects in addition to the reconstructed object, and the reconstructed object may be occluded or truncated) with various viewing angles.

Attention-based fusion network for human eye-fixation prediction in 3D images

Human eye-fixation prediction in 3D images is important for many 3D applications, such as fine-grained 3D video object segmentation and intelligent bulletproof curtains. While the vast majority of existing 2D-based approaches cannot be applied, the main challenge lies in the inconsistency, or even conflict, between the RGB and depth saliency maps. In this paper, we propose a three-stream architecture to accurately predict human visual attention on 3D images end-to-end. First, a two-stream feature extraction network based on advanced convolutional neural networks is trained for RGB and depth, and hierarchical information is extracted from each ResNet-18. Then, these multi-level features are fed into the channel attention mechanism to suppress the feature space inconsistency and make the network focus on a significant target. The enhanced saliency map is fused step-by-step by VGG-16 to generate the final coarse saliency map. Finally, each coarse map is refined empirically through refinement blocks, and the network's own identification errors are corrected based on the acquired knowledge, thus converting the prediction saliency map from coarse to fine. The results of comparison of our model with six other state-of-the-art approaches on the NUS dataset (CC of 0.5579, KLDiv of 1.0903, AUC of 0.8339, and NSS of 2.3373) and the NCTU dataset (CC of 0.8614, KLDiv of 0.2681, AUC of 0.9143, and NSS of 2.3795) indicate that the proposed model consistently outperforms them by a considerable margin as it fully employs the channel attention mechanism.

A Review on 3D Spatial Data Analytics for Building Information Models

Except for various properties, accurate and intuitive 3D representations of building elements and spaces are embedded in Building Information Models (BIMs). Thus, better understanding of spatial features of building elements and spaces is possible, which brings new opportunity in code compliance checking, indoor navigation, visualization, etc. With large amount of fine-grained 3D spatial data collected quickly and continuously, there is an urgent need for new methods to overcome problems like improper spatial expression, insufficient mining and utilization of information that exist in traditional methods. This research reviews state-of-the-art of related research and provides a summary of achievements and challenges in this area. Then, a framework consists of management, analysis, and application of 3D spatial data of BIM and a detailed discussion of each part are proposed, which would give the readers an overview of relevant methods, technologies, and tools. Moreover, potential research directions and open issues for future work are also discussed. The paper enables researchers to get a comprehensive understanding of 3D spatial data analytics of BIM as well as suggestions for future work, and thus makes a solid contribution in this area.