Image-based 3D Modeling Research Articles

Adaptive CLIP for open-domain 3D model retrieval

In order to effectively enhance the practicality of 3D model retrieval, we adopt a single real image as the query sample for retrieving 3D models. However, the significant differences between 2D images and 3D models in terms of lighting conditions, textures and backgrounds, posing a great challenge for accurate retrieval. Existing work on 3D model retrieval mainly focuses on closed-domain research, while the open-domain condition where the category relationship between the query image and the 3D model is unknown is more in line with the needs of real scenarios. CLIP shows significant promise in comprehending open-world visual concepts, facilitating effective zero-shot image recognition. Based on this multimodal pre-training large language model, we introduce Adaptive Open-domain Semantic Nearest-neighbor Contrast (AOSNC), a method for learning and aligning multi-modal text, image, and 3D model. In order to solve the issue of inconsistent cross-domain categories and difficult sample correlation in open-domain, we construct a cross-modal bridge using CLIP. This model utilizes textual features to bridge the gap between 2D images and 3D model views. Additionally, we design an adaptive network layer to address the limitations of the pre-training model for 3D model views and enhance cross-modal alignment. We propose a mutual nearest-neighbor semantic alignment loss to address the challenge of aligning features from disparate modalities (text, images, and 3D models). This loss function enhances cross-modal learning by effectively associating and distinguishing features, improving retrieval accuracy. We conducted comprehensive experiments using the image-based 3D model retrieval dataset MI3DOR and the cross-domain 3D model retrieval dataset NTU-PSB to validate the superiority of the proposed method. Our results show significant improvements in several evaluation metrics, underscoring the efficacy of our method in augmenting cross-modal feature alignment and retrieval performance.

TAJ-Net: a two-stage clustered cell segmentation network with adaptive joint learning of spatial and spectral information.

Pulmonary adenocarcinoma is the primary cause of cancer-related death worldwide and pathological diagnosis is the "golden standard" based on the regional distribution of cells. Thus, regional cell segmentation is a key step while it is challenging due to the following reasons: 1) It is hard for pure semantic and instance segmentation methods to obtain a high-quality regional cell segmentation result; 2) Since the spatial appearances of pulmonary cells are very similar which even confuse pathologists, annotation errors are usually inevitable. Considering these challenges, we propose a two-stage 3D adaptive joint training framework (TAJ-Net) to segment-then-classify cells with extra spectral information as the supplementary information of spatial information. Firstly, we propose to leverage a few-shot method with limited data for cell mask acquisition to avoid the disturbance of cluttered backgrounds. Secondly, we introduce an adaptive joint training strategy to remove noisy samples through two 3D networks and one 1D network for cell type classification rather than segmentation. Subsequently, we propose a patch mapping method to map classification results to the original images to obtain regional segmentation results. In order to verify the effectiveness of TAJ-Net, we build two 3D hyperspectral datasets, i.e., pulmonary adenocarcinoma (3,660 images) and thyroid carcinoma (4623 images) with 40 bands. The first dataset will be released for further research. Experiments show that TAJ-Net achieves much better performance in clustered cell segmentation, and it can regionally segment different kinds of cells with high overlap and blurred edges, which is a difficult task for the state-of-the-art methods. Compared to 2D models, the hyperspectral image-based 3D model reports a significant improvement of up to 11.5% in terms of the Dice similarity coefficient in the pulmonary adenocarcinoma dataset.

Mathematical Camera Array Optimization for Face 3D Modeling Application.

Camera network design is a challenging task for many applications in photogrammetry, biomedical engineering, robotics, and industrial metrology, among other fields. Many driving factors are found in the camera network design including the camera specifications, object of interest, and type of application. One of the interesting applications is 3D face modeling and recognition which involves recognizing an individual based on facial attributes derived from the constructed 3D model. Developers and researchers still face difficulty in reaching the required high level of accuracy and reliability needed for image-based 3D face models. This is caused among many factors by the hardware limitations and imperfection of the cameras and the lack of proficiency in designing the ideal camera-system configuration. Accordingly, for precise measurements, we still need engineering-based techniques to ascertain the specific level of deliverables quality. In this paper, an optimal geometric design methodology of the camera network is presented by investigating different multi-camera system configurations composed of four up to eight cameras. A mathematical nonlinear constrained optimization technique is applied to solve the problem and each camera system configuration is tested for a facial 3D model where a quality assessment is applied to conclude the best configuration. The optimal configuration is found to be a 7-camera array, comprising a pentagon shape enclosing two additional cameras, offering high accuracy. For those who prioritize point density, a 9-camera array with a pentagon and quadrilateral arrangement in the X-Z plane is a viable choice. However, a 5-camera array offers a balance between accuracy and the number of cameras.

Adaptive semantic transfer network for unsupervised 2D image-based 3D model retrieval

Unsupervised 2D image-based 3D model retrieval has been a highlighted research topic to enable flexible retrieval from 2D photos to 3D shapes. Although what methods we have so far have been great progress in this aspect, it still exists some issues in learning discriminative features and to well align the distribution diversity from various domains because of the huge cross-domain interval. According to our paper, we propose an adaptive semantic transfer network (ASTN) to improve the discrimination of feature representations and conveniently narrow the discrepancy of different domains by utilizing the intermediate domain to conduct semantic alignment. Our ASTN composes of the adaptive feature encoding module (AFE) and the dynamic semantic alignment module (DSA). To improve the quality of feature representation, the AFE module deploys a new strategy that trains the learnable parameters on multiple convolutional layers, which can adaptively pay different attentions to these layers. The DSA module dynamically constructs an intermediate domain that aims to convert the familiar direct alignment into the sum of two alignments which are source-intermediate and target-intermediate alignments, effectively narrowing the domain gap and further realizing the semantic alignment. On two arduous datasets, MI3DOR-1 and MI3DOR-2, we design abundant experiments that have demonstrated the effectiveness of our suggested method.

View-target relation-guided unsupervised 2D image-based 3D model retrieval via transformer

Unsupervised 2D image-based 3D model retrieval aims at retrieving images from the gallery of 3D models by the given 2D images. Despite the encouraging progress made in this task, there are still two significant limitations: (1) feature alignment of 2D images and 3D model gallery is still difficult due to the huge gap between the two modalities. (2) The important view information in the 3D model gallery was ignored by the prior arts, which led to inaccurate results. To alleviate these limitations, inspired by the success of vision transformers (ViT) in a great variety of vision tasks, in this paper, we propose an end-to-end 3D model retrieval architecture on top of ViT, termly transformer-based 3D model retrieval network (T3DRN). In addition, to take advantage of the valuable view information of 3D models, we present an attentive module in T3DRN named shared view-guided attentive module (SVAM) to guide the learning of the alignment features. The proposed method is tested on the challenging dataset, MI3DOR-1. The extensive experimental results have proved the superiority of our proposed method to state-of-the-art methods.

Hierarchical deep semantic alignment for cross-domain 3D model retrieval

With the development of deep learning and the widespread application of 3D modeling technology, image-based cross-domain 3D model retrieval has attracted more and more researchers’ attention. Existing methods have achieved success by aligning the feature distributions from different domains. However, previous methods just statistically align the domain-level or class-level feature distributions, leaving sample discriminability a margin to be improved for retrieval. To address this issue, this paper proposes a Hierarchical Deep Semantic Alignment Network (HDSAN) for cross-domain 3D model retrieval, which combines the proposed sample-level semantic enhancement with global domain alignment and class semantic alignment. Concretely, we adopt adversarial domain adaptation at the domain level and dynamically align the class centers of two domains at the class level. To further improve sample discriminability, we design intra-domain and cross-domain triplet center alignment to enhance the semantic representation ability at the sample level. Experiments on two commonly-used cross-domain 3D model retrieval datasets MI3DOR-1 and MI3DOR-2 demonstrate the effectiveness of the proposed method.

Finite element analysis (FEA) of femur to predict biomechanical properties and its validation

The aim of this preliminary study is to predict the biomechanical properties as strain of the femur using computed tomography medical image-based finite element analysis. Further, the finite element results were compared with the experimental results for validation purpose. The main objective of the study was to establish and validate a finite element model used to analyse biomechanical properties. For this, four sheep femur specimens were used to predict normal strain in x-x direction. Computed tomography scans were taken from each femur sample. Subsequently, an accurate image-based 3D model of each femur specimen was segmented and reconstructed with its material properties. Furthermore, the experimental test was performed at 8° bone inclination in the coronal plane and a compressive load was applied at displacement rate of 0.2 mm/s to the femur head in a vertical downward direction until fracture. Consequently, the numerical results of predicted normal strain were compared with the experimental measured normal strain. The average deviation between finite element and experimental results was found to be −11.11%–20.10% in strains. Hence, it is concluded that the results obtained using finite element analysis are in good agreement with the experimental results.

Domain-specific modeling and semantic alignment for image-based 3D model retrieval

Image-based 3D model retrieval aims to search for 3D models according to 2D image queries, which provides a convenient way for the management of large 3D model datasets. Most of the related works put the emphasis on bridging the modality gap between 2D images and 3D models, which faces a lot of challenges due to the huge domain discrepancy. In this paper, we explicitly model and eliminate the domain-specific features of 2D images and 3D models. To alleviate the negative effect of complex background of natural images, we adopt semantic focus loss to constrain networks to learn the most semantically relevant feature representations for both 2D images and 3D models. We conduct extensive experiments on two cross-domain 3D model retrieval datasets, MI3DOR and MI3DOR-2, to show the effectiveness of the proposed method.

Unsupervised self-training correction learning for 2D image-based 3D model retrieval

Existing 2D image-based 3D model retrieval (IBMR) methods usually use the pseudo labels as semantic guidance to reduce the domain-wise and class-wise feature distribution difference across annotated 2D images and unlabeled 3D models. However, they cannot entirely guarantee the quality of pseudo labels, which will decrease prediction discriminability and diversity to affect feature distribution alignment. Therefore, we propose a novel unsupervised self-training correction learning (USTCL) network for the IBMR task. Specifically, we first utilize a CNN to encode 2D images and 3D models (described as multi-view images). Then, we design a noise-corrected self-training learning module (NCST) to denoise pseudo labels in an adversarial manner to make the predicted categories more easily discriminated to improve prediction discriminability. Besides, we employ a target-guided pseudo label refining strategy (TPLR) to progressively refine generated pseudo labels to prevent minority categories from being pushed into majority categories, thereby enhancing prediction diversity. Comprehensive experiments on popular IBMR benchmarks validate the effectiveness and robustness of USTCL, e.g., it can achieve the average gains of 54.20%/22.30%, 62.17%/43.21%, 54.21%/31.18%, 63.48%/43.28%, 59.87%/44.49% in terms of NN, FT, ST, F-measure, DCG and the decrease of 29.50%/35.15% corresponding to ANMRR on MI3DOR (21,000 2D images and 7,690 3D models) and MI3DOR-2 (19,694 2D images and 3,982 3D models), respectively.

Instance-prototype similarity consistency for unsupervised 2D image-based 3D model retrieval

The unsupervised 3D model retrieval is designed to joint the information of well-labeled 2D domain and unlabeled 3D domain to learn collaborative representations. Most existing methods adopted semantic alignment, but were inevitably affected by false pseudo-label. In this paper, we design a novel Instance-Prototype Similarity Consistency Network (IPSC) to guide domain alignment with similarity consistency, which can simultaneously suppress the impact of false pseudo-label information and well reduce the domain discrepancy. IPSC contains two similarity strategies, named Single instance vs Multiple prototypes and Instance-pair vs Single prototype. The first strategy utilizes a single instance as an anchor, and measures the similarities between the anchor and multiple prototypes with the same category but from different domains. The minimization between these similarities can better align the cross-domain prototypes with Kullback–Leibler (KL) divergence than traditional Euclidean similarities. The second strategy utilizes a single prototype as an anchor, and measures the similarities between this anchor and an instance-pair with the same category but from different domains. The minimization between these similarities can conduct the instance-level alignment with KL divergence, which can better suppress the negative effect of noisy pseudo-labels. We conduct various experiments on two datasets, MI3DOR-1 (21000 2D images and 7690 3D models) and MI3DOR-2 (19694 2D images and 3982 3D models), to verify the superiority of our algorithm.

Self-supervised Image-based 3D Model Retrieval

Image-based 3D model retrieval aims at organizing unlabeled 3D models according to the relevance to the labeled 2D images. With easy accessibility of 2D images and wide applications of 3D models, image-based 3D model retrieval attracts more and more attentions. However, it is still a challenging problem due to the modality gap between 2D images and 3D models. In spite of the remarkable progress brought by domain adaptation techniques for this research topic, which usually propose to align the global distribution statistics of two domains, these methods are limited in learning discriminative features for target samples due to the lack of label information in target domain. In this article, besides utilizing the label information of 2D image domain and the adversarial domain alignment, we additionally incorporate self-supervision to address cross-domain 3D model retrieval problem. Specifically, we simultaneously optimize the adversarial adaptation for both domains based on visual features and the contrastive learning for unlabeled 3D model domain to help the feature extractor to learn discriminative feature representations. The contrastive learning is used to map view representations of the identical model nearby while view representations of different models far apart. To guarantee adequate and high-quality negative samples for contrastive learning, we design a memory bank to store and update representative view for each 3D model based on entropy minimization principle. Comprehensive experimental results on the public image-based 3D model retrieval datasets, i.e., MI3DOR and MI3DOR-2, have demonstrated the effectiveness of the proposed method.

Designing And Modeling Of De Havilland Canada Dash 8 Q300 Base on Simpleplanes

Mathematics development is aided by image-based 3D modeling. In certain ways, visual 3D modeling should be considered a branch and application of mathematical morphology. Airplanes are sometimes required in the scene due to storyline requirements. Due to the complexity and time-consuming characteristics of traditional modeling methods, this paper put forward a model making scheme using SimplePlanes software was proposed. After development and testing based on SimplePlanes software, the model is similar to the real aircraft, and using SimplePlanes will save a lot of time and meet the use requirements. In this paper, the method of modeling De Havilland Canada Dash 8 Q300 using SimplePlanes is explained in detail together with the basic modeling logic of this software. A user study is performed to demonstrates the overall good quality of the result and a significant improvement in terms of time efficiency.

Efficient and Accurate Hierarchical SfM Based on Adaptive Track Selection for Large-Scale Oblique Images

Image-based 3D modeling has been widely used in many areas. Structure from motion is the key to image-based reconstruction. However, the rapid growth of data poses challenges to current SfM solutions. A hierarchical SfM reconstruction methodology for large-scale oblique images is proposed. Firstly, match pairs are selected using positioning and orientation (POS) data and the terrain of the survey area. Then, images are divided to image groups by traversing the selected match pairs. After pairwise image matching, tracks are decimated using an adaptive track selection method. Thirdly, submaps are reconstructed from the image groups in parallel based on incremental SfM in the object space. A novel method based on statistics of the positional difference between common tracks is proposed to detect the outliers in submap merging. Finally, the reconstructed submaps are incrementally merged and optimized. The proposed methodology was used on a large oblique image set. The proposed methodology was compared with the state-of-the-art image-based reconstruction systems COLMAP and Metashape for SfM reconstruction. Experimental results show that the proposed methodology achieved the highest accuracy on the experimental dataset, i.e., about 22.37, and 3.52 times faster than COLMAP and Metashape, respectively. The experimental results demonstrate that the proposed hierarchical SfM methodology is accurate and efficient for large-scale oblique images.

CRBeDaSet: A Benchmark Dataset for High Accuracy Close Range 3D Object Reconstruction

This paper presents the CRBeDaSet—a new benchmark dataset designed for evaluating close range, image-based 3D modeling and reconstruction techniques, and the first empirical experiences of its use. The test object is a medium-sized building. Diverse textures characterize the surface of elevations. The dataset contains: the geodetic spatial control network (12 stabilized ground points determined using iterative multi-observation parametric adjustment) and the photogrammetric network (32 artificial signalized and 18 defined natural control points), measured using Leica TS30 total station and 36 terrestrial, mainly convergent photos, acquired from elevated camera standpoints with non-metric digital single-lens reflex Nikon D5100 camera (ground sample distance approx. 3 mm), the complex results of the bundle block adjustment with simultaneous camera calibration performed in the Pictran software package, and the colored point clouds (ca. 250 million points) from terrestrial laser scanning acquired using the Leica ScanStation C10 and post-processed in the Leica Cyclone™ SCAN software (ver. 2022.1.1) which were denoized, filtered, and classified using LoD3 standard (ca. 62 million points). The existing datasets and benchmarks were also described and evaluated in the paper. The proposed photogrammetric dataset was experimentally tested in the open-source application GRAPHOS and the commercial suites ContextCapture, Metashape, PhotoScan, Pix4Dmapper, and RealityCapture. As the first experience in its evaluation, the difficulties and errors that occurred in the software used during dataset digital processing were shown and discussed. The proposed CRBeDaSet benchmark dataset allows obtaining high accuracy (“mm” range) of the photogrammetric 3D object reconstruction in close range, based on a multi-image view uncalibrated imagery, dense image matching techniques, and generated dense point clouds.

Building 3D Object Model for Tourism in Monas using CityEngine

A prominent monument in DKI Jakarta is the Monument Nasional (abbreviated as Monas). This moniker serves as both a symbol for the city of Jakarta and a representation of the people of Jakarta and Indonesia. Apart from being a monument, Monas also has a history of the Indonesian nation. In this study, 3D models are designed to display visualizations of Monas and surroundings in 3D form with the aim of helping tourists who want to visit Monas in preparing and increasing their understanding of the tourism in Monas. This study was conducted with the help of CityEngine software that created the 3D models of Monas implemented and displayed on a website with a web scene format. Using image-based 3D modelling, the main geomatics techniques is photogrammetry, it allows for new methods to create virtual 3D. To do this, various components including route visualizations must collaborate. To do this work we have a framework that leverages the technology and data sources already available to reduce the time and work required for prospective 3D applications. With the existence of a Monas 3D visualization that can be accessed through web browser, it expected that it will make tourists not complicated to obtain various tourism information and increase the tourism in Jakarta.

Unsupervised Cross-Media Graph Convolutional Network for 2D Image-Based 3D Model Retrieval

With the rapid development of 3D construction technology, 3D models have been implemented in many applications. In particular, the fields of virtual and augmented reality have created a considerable demand for rapid access to large sets of 3D models in recent years. An effective method for addressing the demand is to search 3D models based on 2D images because 2D images can be easily captured by smartphones or other lightweight vision sensors. In this paper, we propose a novel unsupervised cross-media graph convolutional network (UCM-GCN) for 3D model retrieval based on 2D images. Here, we render views from 3D models to construct a graph model based on 3D model structural information. Then, we utilize the 2D image's visual information to bridge the gap between cross-modality data. Then, the proposed UCM-GCN is utilized to update the feature vector of the 2D image and the 3D model. Here, we introduce correlation loss to mitigate the distribution discrepancy across different modalities, which can fully consider the structural and visual similarities between the 2D image and 3D model to embed the final different modalities into the same feature space. To demonstrate the performance of our approach, we conducted a series of experiments on the MI3DOR dataset, which is utilized in SHREC19. We also compared it with other similar methods on the 3D-FUTURE dataset. The experimental results demonstrate the superiority of our proposed method over state-of-the-art methods.

Image-based 3D model retrieval via disentangled feature learning and enhanced semantic alignment

With the development of 3D technology and the increase in 3D models, 2D image-based 3D model retrieval tasks have drawn increased attention from scholars. Previous works align cross-domain features via adversarial domain alignment and semantic alignment. However, the extracted features of previous methods are disturbed by the residual domain-specific features, and the lack of labels for 3D models makes the semantic alignment challenging. Therefore, we propose disentangled feature learning associated with enhanced semantic alignment to address these problems. On one hand, the disentangled feature learning enables decoupling the twisted raw features into the isolated domain-invariant and domain-specific features, and the domain-specific features will be dropped while performing adversarial domain alignment and semantic alignment to acquire domain-invariant features. On the other hand, we mine the semantic consistency by compacting each 3D model sample and its nearest neighbors to further enhance semantic alignment for unlabeled 3D model domain. We give comprehensive experiments on two public datasets, and the results demonstrate the superiority of the proposed method. Especially on MI3DOR-2 dataset, our method outperforms the current state-of-the-art methods with gains of 2.88% for the strictest retrieval metric NN.

Morphological specificity analysis of an image-based 3D model of airway filling in a difficult airway

BackgroundThe purpose of this study was to analyze position-specific morphological changes of the upper airway and to further assess the impact of these changes in difficult airway during intubation.MethodsThis observational comparative study included two groups (n = 20 patients/group): Group A had normal airway and Group B had difficult airway. Data obtained from two-dimensional magnetic resonance imaging were imported to Mimics V20.0 software for processing. We then reconstructed three-dimensional models of upper airway filling in patients in the supine and maximum extension position based on the imaging data. Those models were projected on coronal, sagittal, and horizontal planes to investigate multiple morphological features. We measured the surface area, radial length, and corner angle of the projected areas.ResultsGroup A had larger upper airway filling volumes compared to Group B The volumes for the supine position were 6,323.83 ± 156.06 mm3 for Group A and 5,336.22 ± 316.13 mm3 for Group B (p = 0.003). The volumes the maximum extension position were 9,186.58 ± 512.61 mm3 for Group A and 6,735.46 ± 794.63 mm3 for Group B (p = 0.003). Airway volume increased in the upper airway filling model as the body position varied from the supine to maximum extension position (Group A: volume increase 2,953.75 ± 524.6 mm3, rate of change 31%; Group B: volume increase 1,632.89 ± 662.66 mm3, rate of change 25%; p = 0.052).ConclusionThe three-dimensional reconstruction model developed in this study was used to digitally quantify morphological features of a difficult airway and could be used as a novel airway management assessment tool.

Comparison of in vivo visco-hyperelastic properties of uterine suspensory tissue in women with and without pelvic organ prolapse

The uterine suspensory tissue (UST) complex includes the cardinal (CL) and uterosacral “ligaments” (USL), which are mesentery-like structures that play a role in resisting pelvic organ prolapse (POP). Since there is no information on the time-dependent material properties of the whole structure in situ and in vivo, we developed and tested an intraoperative technique to quantify in vivo whether there is a significant difference in visco-hyperelastic behavior of the CL and USL between women with and without POP. Thirteen women with POP (cases) and four controls scheduled for surgery were selected from an ongoing POP study. Immediately prior to surgery, a computer-controlled linear servo-actuator with a series force transducer applied a continuous, caudally directed traction force while simultaneously recording the resulting cervical displacement in the same direction. After applying an initial 1.1 N preload, a ramp rate of 4 mm/s was used to apply a maximum force of 17.8 N in three “ramp-and-hold” test trials. A simplified bilateral four-cable biomechanical model was used to identify the material behavior of each ligament. For this, the initial cross-section areas of the CL and USL were measured on 3-T magnetic resonance image-based 3D models from each subject. The time-dependent strain energy function of CL/USL was defined with a three-parameter hyperelastic Mooney–Rivlin material model and a two-term Prony series in relaxation form. When cases were compared with controls, the estimated time-dependent material constants of CL and USL did not differ significantly. These are the first measurements that compare the in vivo and in situ visco-hyperelastic response of the tissues comprising the CL and USL to loading in women with and without prolapse. Larger sample sizes would help improve the precision of intergroup differences.

Learning Transferable and Discriminative Representations for 2D Image-Based 3D Model Retrieval

Existing research on the 2D image-based 3D model retrieval task focuses on learning transferable representations directly to narrow the domain discrepancy. However, it is not easy to achieve in practice due to the significant variations across two domains. In addition, some methods design a domain discriminator to distinguish the feature arising from source or target domains for transferable feature representations learning, which will lead to an unexpected deterioration of the feature discriminability. To settle these problems, we propose jointly learning transferable and discriminative representations for 2D image-based 3D model retrieval. Specifically, we extract features from the 2D images and 3D models (described as multiple views) by CNN. Considering the difficulty of directly narrowing the discrepancy of two domains, we are prone to connect 2D image and 3D model domains to an intermediate domain, where the domain gap aims to be eliminated. However, the feature transferability does not denote well discriminability. Based on the batch spectral penalization (BSP) theory, the feature transferability is dominated by feature vectors with higher singular values, while the feature discriminability depends on more eigenvectors with lower singular values to convey rich discriminative structures. Therefore, we penalize the largest singular values so that the feature vectors with lower singular values are appropriately enhanced, thereby strengthening feature discriminability. A series of experiments on two challenging datasets, MI3DOR and MI3DOR-2, indicate that our method can significantly improve performance.