Image-based 3D Research Articles

Image-based 3D model retrieval via disentangled feature learning and enhanced semantic alignment

With the development of 3D technology and the increase in 3D models, 2D image-based 3D model retrieval tasks have drawn increased attention from scholars. Previous works align cross-domain features via adversarial domain alignment and semantic alignment. However, the extracted features of previous methods are disturbed by the residual domain-specific features, and the lack of labels for 3D models makes the semantic alignment challenging. Therefore, we propose disentangled feature learning associated with enhanced semantic alignment to address these problems. On one hand, the disentangled feature learning enables decoupling the twisted raw features into the isolated domain-invariant and domain-specific features, and the domain-specific features will be dropped while performing adversarial domain alignment and semantic alignment to acquire domain-invariant features. On the other hand, we mine the semantic consistency by compacting each 3D model sample and its nearest neighbors to further enhance semantic alignment for unlabeled 3D model domain. We give comprehensive experiments on two public datasets, and the results demonstrate the superiority of the proposed method. Especially on MI3DOR-2 dataset, our method outperforms the current state-of-the-art methods with gains of 2.88% for the strictest retrieval metric NN.

STV-SC: Segmentation and Temporal Verification Enhanced Scan Context for Place Recognition in Unstructured Environment.

Place recognition is an essential part of simultaneous localization and mapping (SLAM). LiDAR-based place recognition relies almost exclusively on geometric information. However, geometric information may become unreliable when faced with environments dominated by unstructured objects. In this paper, we explore the role of segmentation for extracting key structured information. We propose STV-SC, a novel segmentation and temporal verification enhanced place recognition method for unstructured environments. It contains a range image-based 3D point segmentation algorithm and a three-stage process to detect a loop. The three-stage method consists of a two-stage candidate loop search process and a one-stage segmentation and temporal verification (STV) process. Our STV process utilizes the time-continuous feature of SLAM to determine whether there is an occasional mismatch. We quantitatively demonstrate that the STV process can trigger false detections caused by unstructured objects and effectively extract structured objects to avoid outliers. Comparison with state-of-art algorithms on public datasets shows that STV-SC can run online and achieve improved performance in unstructured environments (Under the same precision, the recall rate is 1.4∼16% higher than Scan context). Therefore, our algorithm can effectively avoid the mismatching caused by the original algorithm in unstructured environment and improve the environmental adaptability of mobile agents.

3D SOC-Net: Deep 3D reconstruction network based on self-organizing clustering mapping

Image-based 3D reconstruction from a single-view image is critical and fundamental in many areas and can be integrated into many applications to provide useful functions. However, there are several crucial difficulties and challenges in accomplishing this process. For example, the issues of self-occlusion and lack of object information in a different perspective of viewing. Thus, the quality of the generated 3D shape from a single-view image may not be satisfactory and robust, hence affecting its feasibility in further applications. Conventionally, the 3D reconstruction process requires multiple input images such that the context of the target object can be fully conveyed. In this paper, we propose a new and simple, yet powerful framework that improves the quality of the generated point cloud from a single-view image. Concretely, the significant representatives are first discovered and selected by adopting a network architecture that contains both encoder and decoder models. Finally, the resultant point clouds are obtained by extracting the mean shape using the methods of Chamfer Distance (CD), Earth Mover’s Distance (EMD), and Self-Organizing Map (SOM). As a result, the proposed algorithm is capable to demonstrate its robustness and effectiveness when compared to state-of-the-art 3D reconstruction methods. The best mean loss exhibited is 4.45 when evaluated on 12 classes in the ShapeNetCoreV1 dataset. In addition, qualitative results are presented to further verify the reliability of the proposed method.

Morphological specificity analysis of an image-based 3D model of airway filling in a difficult airway

BackgroundThe purpose of this study was to analyze position-specific morphological changes of the upper airway and to further assess the impact of these changes in difficult airway during intubation.MethodsThis observational comparative study included two groups (n = 20 patients/group): Group A had normal airway and Group B had difficult airway. Data obtained from two-dimensional magnetic resonance imaging were imported to Mimics V20.0 software for processing. We then reconstructed three-dimensional models of upper airway filling in patients in the supine and maximum extension position based on the imaging data. Those models were projected on coronal, sagittal, and horizontal planes to investigate multiple morphological features. We measured the surface area, radial length, and corner angle of the projected areas.ResultsGroup A had larger upper airway filling volumes compared to Group B The volumes for the supine position were 6,323.83 ± 156.06 mm3 for Group A and 5,336.22 ± 316.13 mm3 for Group B (p = 0.003). The volumes the maximum extension position were 9,186.58 ± 512.61 mm3 for Group A and 6,735.46 ± 794.63 mm3 for Group B (p = 0.003). Airway volume increased in the upper airway filling model as the body position varied from the supine to maximum extension position (Group A: volume increase 2,953.75 ± 524.6 mm3, rate of change 31%; Group B: volume increase 1,632.89 ± 662.66 mm3, rate of change 25%; p = 0.052).ConclusionThe three-dimensional reconstruction model developed in this study was used to digitally quantify morphological features of a difficult airway and could be used as a novel airway management assessment tool.

A Feature Transformation Framework With Selective Pseudo-Labeling for 2D Image-Based 3D Shape Retrieval

2D image-based 3D shape retrieval (2D-to-3D) aims at searching the corresponding 3D shapes (unlabeled) when given a 2D image (labeled), which is a fundamental task in computer vision and has gained a surge of attention in recent years. However, extensive prior works are limited by two settings, 1) reducing domain discrepancy while ignoring the 3D shape style, 2) 3D shapes are simply and brutally pseudo-annotated by the 2D image-supervised classifier, neglecting the structure information underlying the 3D shape domain. To remedy these issues, we propose a feature transformation framework with selective pseudo-labeling (FTSPL) for 2D-to-3D task. Specifically, we first employ CNNs to produce both 2D image and 3D shape (described as multiple views) features, then we force the inter-domain centroid alignment class-wisely to reduce the overall domain discrepancy. In addition to this, we further exploit the intra-category attribute variation (covariance) of 3D shape features to transform the 2D image features. By doing so, we can equip 2D features with 3D shape style. Since the centroid and covariance estimation of 3D shape features require accurate label predictions, we put forward a selective pseudo-labeling module, which can assign reliable pseudo-labels for 3D shapes via nearest category centroid and cluster analysis, respectively, while preserving the structure information of 3D shapes. Comprehensive experiments validate that our model surpasses the state-of-the-arts on standard 2D-to-3D benchmarks (MI3DOR and MI3DOR-2).

Comparison of in vivo visco-hyperelastic properties of uterine suspensory tissue in women with and without pelvic organ prolapse

The uterine suspensory tissue (UST) complex includes the cardinal (CL) and uterosacral “ligaments” (USL), which are mesentery-like structures that play a role in resisting pelvic organ prolapse (POP). Since there is no information on the time-dependent material properties of the whole structure in situ and in vivo, we developed and tested an intraoperative technique to quantify in vivo whether there is a significant difference in visco-hyperelastic behavior of the CL and USL between women with and without POP. Thirteen women with POP (cases) and four controls scheduled for surgery were selected from an ongoing POP study. Immediately prior to surgery, a computer-controlled linear servo-actuator with a series force transducer applied a continuous, caudally directed traction force while simultaneously recording the resulting cervical displacement in the same direction. After applying an initial 1.1 N preload, a ramp rate of 4 mm/s was used to apply a maximum force of 17.8 N in three “ramp-and-hold” test trials. A simplified bilateral four-cable biomechanical model was used to identify the material behavior of each ligament. For this, the initial cross-section areas of the CL and USL were measured on 3-T magnetic resonance image-based 3D models from each subject. The time-dependent strain energy function of CL/USL was defined with a three-parameter hyperelastic Mooney–Rivlin material model and a two-term Prony series in relaxation form. When cases were compared with controls, the estimated time-dependent material constants of CL and USL did not differ significantly. These are the first measurements that compare the in vivo and in situ visco-hyperelastic response of the tissues comprising the CL and USL to loading in women with and without prolapse. Larger sample sizes would help improve the precision of intergroup differences.

Combining Photogrammetry and Photometric Stereo to Achieve Precise and Complete 3D Reconstruction.

Image-based 3D reconstruction has been employed in industrial metrology for micro-measurements and quality control purposes. However, generating a highly-detailed and reliable 3D reconstruction of non-collaborative surfaces is still an open issue. In this paper, a method for generating an accurate 3D reconstruction of non-collaborative surfaces through a combination of photogrammetry and photometric stereo is presented. On one side, the geometric information derived with photogrammetry is used in areas where its 3D measurements are reliable. On the other hand, the high spatial resolution capability of photometric stereo is exploited to acquire a finely detailed topography of the surface. Finally, three different approaches are proposed to fuse both geometric information and high frequency details. The proposed method is tested on six different non-collaborative objects with different surface characteristics. To evaluate the accuracy of the proposed method, a comprehensive cloud-to-cloud comparison between reference data and 3D points derived from the proposed fusion methods is provided. The experiments demonstrated that, despite correcting global deformation up to an average RMSE of less than 0.1 mm, the proposed method recovers the surface topography at the same high resolution as the photometric stereo.

An image-based coal network model for simulating hydro-mechanical gas flow in coal: An application to carbon dioxide geo-sequestration

CO2 geo-sequestration is a practical approach to achieve net-zero carbon target. However, one of the main challenges for successful CO2 geo-sequestration is the reduced coal permeability and injectivity that are caused by coal swelling. Coal has complex and heterogenous internal pore and fracture structure. The processes of gases adsorbing, desorbing, and transporting within multiscale heterogeneous coal structures are more complicated compared with conventional rocks. This paper aims to gain insights about the gas transport behaviours in coal by developing a coupled model to simulate gas flow multiphysics as well as dynamic coal deformation. This work develops an image-based 3D fracture network model, called Fracture Box Model (FBNM). In this model, each fracture is described by arrays of box elements such that the regional change of fracture opening widths can be preserved. Compared with other fracture models (e.g. discrete fracture network), FBNM can simulate complicated multiphysical gas transport more efficiently, but also be able to simulate corresponding coal matrix deformation. By comparing permeability results between direct simulation method with FBNM, it is found that FBNM can effectively estimate the permeability of original fracture networks, but requiring significantly less computational cost. To study the implications of gas types, effective stress, gas adsorption, and thermal expansion on coal permeability, gas injection pressures, gas types, coal seam temperatures are varied and investigated in the simulations. In addition to the advantage of computational efficiency, FBNM is more preferable for complicated flow transport simulations where direct simulation methods are still challenging. This work provides a promising framework which could be further developed for multiphase and multicomponent flow simulations for CO2 geo-sequestration projects.

Characterization of ultrasonic induced damage on multi-scale pore/fracture in coal using gas sorption and μ-CT 3D reconstruction

Ultrasonic physical stimulation is a promising approach of coalbed methane enhancement recovery. This study is devoted to quantitatively characterizing the damage of multi-scale pores and fractures in coal induced by power ultrasonic stimulation using the gas sorption and CT image-based 3D reconstruction. The gas sorption tests show that the ultrasound treatment (3 kW, 25 Hz, 2 h) increased the maximum N2 sorption quantities (by 75.44 %–111.8 %), the average pore diameter by 12.15 %–20.81 %, and the pore surface area (by 30.99 %–56.02 %) in the tested coal. The mesopore volumes increase by 16.46 %–79.15 %, and the macropore volumes increase by 2.38 %–14.82 % after the ultrasound treatment, which reveals that the ultrasound implemented in this study is more efficient for structure modification in pore with scale of 2–50 nm. The CT image-based characterization shows that fracture volume increased by 8.11 %–11.81 %, and the fracture surface increased by 13.92 % −21.63 %, and the average equivalent diameter increased by 6.76 %–10.74 %, and the porosity increased by 8.1 %–11.86 % for the tested coal subjected to ultrasound treatment. Based on the pore network model, the fracture connectivity was quantitatively analyzed, and it shows that pore throat number increased by 17.62 %–45.76 %, and the average equal radius of pore throat increased by 3.86 %–4.83 %. The proportion of connected fractures in the total fractures was increased by a maximum of 12.78 % in the tested coal after ultrasound treatment. The results show that ultrasound stimulation can significantly improve the multi-scale structure of pores and fractures of coal, which will promote gas desorption, diffusive and Darcian flow flux in coal seam, and enhance coalbed methane production.

View-Aware Geometry-Structure Joint Learning for Single-View 3D Shape Reconstruction.

Reconstructing a 3D shape from a single-view image using deep learning has become increasingly popular recently. Most existing methods only focus on reconstructing the 3D shape geometry based on image constraints. The lack of explicit modeling of structure relations among shape parts yields low-quality reconstruction results for structure-rich man-made shapes. In addition, conventional 2D-3D joint embedding architecture for image-based 3D shape reconstruction often omits the specific view information from the given image, which may lead to degraded geometry and structure reconstruction. We address these problems by introducing VGSNet, an encoder-decoder architecture for view-aware joint geometry and structure learning. The key idea is to jointly learn a multimodal feature representation of 2D image, 3D shape geometry and structure so that both geometry and structure details can be reconstructed from a single-view image. To this end, we explicitly represent 3D shape structures as part relations and employ image supervision to guide the geometry and structure reconstruction. Trained with pairs of view-aligned images and 3D shapes, the VGSNet implicitly encodes the view-aware shape information in the latent feature space. Qualitative and quantitative comparisons with the state-of-the-art baseline methods as well as ablation studies demonstrate the effectiveness of the VGSNet for structure-aware single-view 3D shape reconstruction.

Learning Transferable and Discriminative Representations for 2D Image-Based 3D Model Retrieval

Existing research on the 2D image-based 3D model retrieval task focuses on learning transferable representations directly to narrow the domain discrepancy. However, it is not easy to achieve in practice due to the significant variations across two domains. In addition, some methods design a domain discriminator to distinguish the feature arising from source or target domains for transferable feature representations learning, which will lead to an unexpected deterioration of the feature discriminability. To settle these problems, we propose jointly learning transferable and discriminative representations for 2D image-based 3D model retrieval. Specifically, we extract features from the 2D images and 3D models (described as multiple views) by CNN. Considering the difficulty of directly narrowing the discrepancy of two domains, we are prone to connect 2D image and 3D model domains to an intermediate domain, where the domain gap aims to be eliminated. However, the feature transferability does not denote well discriminability. Based on the batch spectral penalization (BSP) theory, the feature transferability is dominated by feature vectors with higher singular values, while the feature discriminability depends on more eigenvectors with lower singular values to convey rich discriminative structures. Therefore, we penalize the largest singular values so that the feature vectors with lower singular values are appropriately enhanced, thereby strengthening feature discriminability. A series of experiments on two challenging datasets, MI3DOR and MI3DOR-2, indicate that our method can significantly improve performance.

A Survey for 3D Object Detection Algorithms from Images

Image-based 3D object detection is one of the important and difficult problems in autonomous driving and robotics, and aims to find and represent the location, dimension and orientation of the object of interest. It generates three dimensional (3D) bounding boxes with only 2D images obtained from cameras, so there is no need for devices that provide accurate depth information such as LiDAR or Radar. Image-based methods can be divided into three main categories: monocular, stereo, and multi-view 3D object detection. In this paper, we investigate the recent state-of-the-art models of the above three categories. In the multi-view 3D object detection, which appeared together with the release of the new benchmark datasets, NuScenes and Waymo, we discuss the differences from the existing monocular and stereo methods. Also, we analyze their performance and discuss the advantages and disadvantages of them. Finally, we conclude the remaining challenges and a future direction in this field.

VR Multiscale Geovisualization Based on UAS Multitemporal Data: The Case of Geological Monuments

Technological progress in Virtual Reality (VR) and Unmanned Aerial Systems (UASs) offers great advantages in the field of cartography and particularly in the geovisualization of spatial data. This paper investigates the correlation between UAS flight characteristics for data acquisition and the quality of the derived maps and 3D models of geological monuments for VR geovisualization in different scales and timeframes. In this study, we develop a methodology for mapping geoheritage monuments based on different cartographic scales. Each cartographic scale results in diverse orthophotomaps and 3D models. All orthophotomaps and 3D models provide an optimal geovisualization, combining UAS and VR technologies and thus contributing to the multitemporal 3D geovisualization of geological heritage on different cartographic scales. The study area selected was a fossilite ferrous site located in Lesvos Geopark, UNESCO. The study area contains a fossil site surrounding various findings. The three distinct scales that occur are based on the object depicted: (i) the fossilite ferrous site (1:120), (ii) the fossil root system (1:20), and (iii) individual fossils (≥1:10). The methodology followed in the present research consists of three main sections: (a) scale-variant UAS data acquisition, (b) data processing and results (2D–3D maps and models), and (c) 3D geovisualization to VR integration. Each different mapping scale determines the UAS data acquisition parameters (flight pattern, camera orientation and inclination, height of flight) and defines the resolution of the 3D models to be embedded in the VR environment. Due to the intense excavation of the study area, the location was spatiotemporally monitored on the cartographic scale of 1:120. For the continuous monitoring of the study area, four different UASs were also used. Each of them was programmed to fly and acquire images with a constant ground sampling distance (GSD). The data were processed by image-based 3D modeling and computer vision algorithms from which the 3D models and orthophotomaps were created and used in the VR environment. As a result, a VR application visualizing multitemporal data of geoheritage monuments across three cartographic scales was developed.

Improved Semantic Representation Learning by Multiple Clustering for Image-Based 3D Model Retrieval

Under the heavy management on the increasing 3D models, the topic of image-based 3D model retrieval which organizes unlabeled 3D models based on abundant knowledge learned from labeled 2D images has drawn attention. However, prior methods are limited in aligning semantically at corresponding categories of two domains due to the lack of label information in the 3D domain. To this end, this paper proposes an improved semantic representation learning by multiple clustering approach, which improves the reliability of pseudo labels for 3D models, so as to achieve class-level semantic alignment. Specifically, this paper first extracts features for 2D images and 3D models. Then it clusters combining the 3D features with the semantic information from multiple clustering on 3D model features to obtain more reliable target pseudo labels. Extensive experiments have shown that the proposed method has achieved the gain of 3.0%-205.0% averagely for popular retrieval metrics on the benchmark of monocular image-based 3D object retrieval (MI3DOR), and 1.3%-69.7% on another advanced benchmark, MI3DOR-2.

The visual measurement system in the parachute used on the Tianwen-1 Mars exploration mission

ABSTRACT The parachute of the probe plays an important role in the deceleration of the Mars landing mission. This paper provides an overview of the visual measurement system of the parachute in Tianwen-1 exploration mission. Using two monitoring cameras, the system has captured stereo images during the parachute deployment. The morphological and motion parameters of the parachute are reconstructed by image-based feature tracking and 3D reconstruction methods. To improve the image quality, a super-resolution (SR) reconstruction method based on sparse dictionary coding is proposed. In addition, we use a multi-scale feature tracking method to track the feature points and update the motion parameters. The system has been verified by the test on the Earth and successfully applied on the onboard images. The reconstructed parachute parameters are important in the post-flight performance assessment, and the research results can be used as a regression analysis means for the structural design.

Erratum for “Image-Based 3D Reconstruction of Granular Grains via Hybrid Algorithm and Level Set with Convolution Kernel” by Pin Zhang, Zhen-Yu Yin, and Qiushi Chen

Erratum for “Image-Based 3D Reconstruction of Granular Grains via Hybrid Algorithm and Level Set with Convolution Kernel” by Pin Zhang, Zhen-Yu Yin, and Qiushi Chen

Oblique View Selection for Efficient and Accurate Building Reconstruction in Rural Areas Using Large-Scale UAV Images

3D building models are widely used in many applications. The traditional image-based 3D reconstruction pipeline without using semantic information is inefficient for building reconstruction in rural areas. An oblique view selection methodology for efficient and accurate building reconstruction in rural areas is proposed in this paper. A Mask R-CNN model is trained using satellite datasets and used to detect building instances in nadir UAV images. Then, the detected building instances and UAV images are directly georeferenced. The georeferenced building instances are used to select oblique images that cover buildings by using nearest neighbours search. Finally, precise match pairs are generated from the selected oblique images and nadir images using their georeferenced principal points. The proposed methodology is tested on a dataset containing 9775 UAV images. A total of 4441 oblique images covering 99.4% of all the buildings in the survey area are automatically selected. Experimental results show that the average precision and recall of the oblique view selection are 0.90 and 0.88, respectively. The percentage of robustly matched oblique-oblique and oblique-nadir image pairs are above 94% and 84.0%, respectively. The proposed methodology is evaluated for sparse and dense reconstruction. Experimental results show that the sparse reconstruction based on the proposed methodology reduces 68.9% of the data processing time, and it is comparably accurate and complete. Experimental results also show high consistency between the dense point clouds of buildings reconstructed by the traditional pipeline and the pipeline based on the proposed methodology.

What does it look like? An artificial neural network model to predict the physical dense 3D appearance of a large-scale object

Object 3D reconstruction is a well-known ill-posed problem that has been extensively studied and making compelling progress, especially in recent years. This is owing to the rise of computational capability in enabling the efficient processing of neural networks. This article presents a benchmark for image-based 3D reconstruction in a realistic condition. Particularly, a novel pipeline is developed to localize the dense surface of a large-scale object at different twisting angles. A shallow artificial neural network with a single hidden layer is devised to learn the correlation between the simulated frame and ground-truth data points. As a result, the proposed framework demonstrates the robustness of the model by providing a valid and reasonable prediction performance in practical problems. Notably, remarkably low RMSE of 8 and a high R2 of 1 are yielded when evaluated in a dataset of 211 sample data. Specifically, a curvature dataset is constructed by twisting a 90 kg metal board at several angles, using two six-axis articulated industrial robots.

PERF: Performant, Explicit Radiance Fields

We present a novel way of approaching image-based 3D reconstruction based on radiance fields. The problem of volumetric reconstruction is formulated as a non-linear least-squares problem and solved explicitly without the use of neural networks. This enables the use of solvers with a higher rate of convergence than what is typically used for neural networks, and fewer iterations are required until convergence. The volume is represented using a grid of voxels, with the scene surrounded by a hierarchy of environment maps. This makes it possible to get clean reconstructions of 360° scenes where the foreground and background is separated. A number of synthetic and real scenes from well-known benchmark-suites are successfully reconstructed with quality on par with state-of-the-art methods, but at significantly reduced reconstruction times.

Gradual adaption with memory mechanism for image-based 3D model retrieval

With the development of 3D modeling technology and its wide application in different fields, the number of 3D models increases rapidly, making 3D model retrieval a hot topic in current research. Compared with other 3D model retrieval methods, 2D image-based unsupervised 3D model retrieval takes the 2D images which have rich labels and are easy to obtain as the queries, and also takes into account the difficulties of labeling 3D models. 2D image-based unsupervised 3D model retrieval is a retrieval task involving cross-domain adaptation problem, which main challenge is the excessive domain gap. In this paper, we propose a cross-domain 3D model retrieval method of memory mechanism based on disentangled feature learning. The disentangled feature learning enables to disentangle the twisted original features into the isolated domain-invariant features and domain-specific features, where the former is to be aligned to narrow the domain gap. On this basis, the memory mechanism selects feature vectors from class memory modules constructed by class representative features of the opposite domain for every sample, which are used to update the domain-invariant features with gradient weight. The memory mechanism can gradually improve the adaptability of the model to the very different two domains. Experiments are conducted on the public datasets MI3DOR and MI3DOR-2 to verify the feasibility and the superiority of the proposed method. Especially on MI3DOR-2 dataset, our method outperforms the current state-of-the-art methods with gains of 7.71% for the strictest retrieval metric NN. • An end-to-end unsupervised 2D image-based 3D model retrieval framework. • Transfering knowledge from labeled 2D images to unlabeled 3D models. • Domain-invariant features are disentangled from the original features. • Memory module enhances domain-invariant features by representative features. • Experiments on MI3DOR and MI3DOR-2 verified the superiority of the method.