3D Retrieval Research Articles

Supervised Contrastive Learning for 3D Cross-Modal Retrieval

Interoperability between different virtual platforms requires the ability to search and transfer digital assets across platforms. Digital assets in virtual platforms are represented in different forms or modalities, such as images, meshes, and point clouds. The cross-modal retrieval of three-dimensional (3D) object representations is challenging due to data representation diversity, making common feature space discovery difficult. Recent studies have been focused on obtaining feature consistency within the same classes and modalities using cross-modal center loss. However, center features are sensitive to hyperparameter variations, making cross-modal center loss susceptible to performance degradation. This paper proposes a new 3D cross-modal retrieval method that uses cross-modal supervised contrastive learning (CSupCon) and the fixed projection head (FPH) strategy. Contrastive learning mitigates the influence of hyperparameters by maximizing feature distinctiveness. The FPH strategy prevents gradient updates in the projection network, enabling the focused training of the backbone networks. The proposed method shows a mean average precision (mAP) increase of 1.17 and 0.14 in 3D cross-modal object retrieval experiments using ModelNet10 and ModelNet40 datasets compared to state-of-the-art (SOTA) methods.

LSTM-based DEM generation in riverine environment

In the broad field of sensors and 3D information retrieval, bathymetric reconstruction from side-scan sonar imaging is associated with unique technical hurdles. Neural Networks have recently led to promising new solutions in this field, but the available methods tend to be complex and data-intensive in a way typically making their use in a riverine environment impossible. Throughout our work, we have focused on simplifying the problem-handling and treating compatibility with a riverine environment as priority. In our work, Long Short-Term Memory proved to be effective in a surprisingly simple form. Combined with traditional post-processing techniques in the GIS environment, like median filtered focal statistics, our workflow ultimately results in ∼0.259 m median of error on the evaluation dataset of the Dráva River.

Formation of Inclusion Complex of Curcumin and Tetrahydrocurcumin Prevents Angiogenesis by Inhibiting VEGF Activity: An in-Silico Study

Curcumin and tetrahydrocurcumin (THC) are known for their anticancer properties, but limited solubility in water hinders their effectiveness against cancer. In this study, we conducted an in silico exploration of β-cyclodextrin's potential to form inclusion complexes with curcumin or THC. The aim of this study was to assess the potential of curcumin and THC inclusion complexes to inhibit vascular endothelial growth factor (VEGF) signaling pathway, a key element in carcinogenesis. The in silico analysis involved multiple stages, such as bioactive compound preparation, biological activity prediction, 3D structure retrieval of VEGF and VEGFR, protein-ligand docking, and visualization. The results of the study demonstrated that both the curcumin- and THC-inclusion complexes exhibit a lower requirement for binding free energy to interact with VEGFR compared to curcumin or THC molecules alone. When VEGFR binds with curcumin, the curcumin-inclusion complex, or the THC-inclusion complex before interacting with VEGF, there is a notable increase in the binding free energy for the VEGF-VEGFR interaction. Specifically, the presence of THC-inclusion complex demonstrates the highest binding free energy for the VEGF-VEGFR interaction. The molecular dynamic simulation study shows that when VEGFR binds with curcumin, curcumin-inclusion complex, or THC-inclusion complex, the fluctuation of amino acid residues in VEGFR decreases compared to the VEGFR protein structure before binding with these molecules. In conclusion, this study suggests that the formation of inclusion complexes holds considerable promise for enhancing the anticancer potential of curcumin and THC by augmenting their anti-angiogenic activity.

Application of computer vision techniques for 3D matching and retrieval of archaeological objects

Background As cultural institutions embark in projects oriented to digitise art and archaeological collections in three dimensions, the need for developing means to access the resulting 3D models has become imperative. Shape recognition techniques developed in the field of computer vision can help in this task. Methods This paper describes the implementation of three shape descriptors, specifically shape distributions, reflective symmetry and spherical harmonics as part of the development of a search engine that retrieves 3D models from an archaeological database without the need of using keywords as query criteria. Use case The usefulness of this system is obvious in the context of cultural heritage museums, where it is essential to provide automatic access to archaeological and art collections. The prototype described in this paper uses, as study case, 3D models of archaeological objects belonging to Museo del Templo Mayor, a Mexican institution that preserves one of the largest collections of Aztec cultural heritage. Conclusions This work is part of an ongoing project focused on creating generic methodologies and user-friendly computational tools for shape analysis for the benefit of scholars and students interested in describing, interpreting and disseminating new knowledge about the morphology of cultural objects.

Improved biharmonic kernel signature for 3D non-rigid shape matching and retrieval

Improved biharmonic kernel signature for 3D non-rigid shape matching and retrieval

Balanced Class-Incremental 3D Object Classification and Retrieval

Most existing 3D object classification and retrieval algorithms rely on one-off supervised learning on closed 3D object sets and tend to provide rigid convolutional neural networks with little scalability. Such limitations substantially restrict their potential to learn newly emerged 3D object classes continually in the real world. Aiming to go beyond these limitations, we innovatively propose two new and challenging tasks: class-incremental 3D object classification ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOC</i> ) and class-incremental 3D object retrieval ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOR</i> ), the key to which is class-incremental 3D representation learning. It expects the network to update continually to learn new 3D class representations without forgetting the previously learned ones. To this end, we design a novel balanced distillation network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(BDNet)</i> that uses a dual supervision mechanism to balance between consolidating old knowledge (stability) and adapting to new 3D object classes (plasticity) carefully. On the one hand, we employ stability-based supervision to retain the stable and discriminative information of old classes that greatly benefit both classification and retrieval tasks. On the other hand, we use plasticity-based supervision to improve the network's generalization for learning new class 3D representations by transferring knowledge from a temporary teacher network to the current model. By properly handling the relationship between the two modules, we achieve a surprising performance improvement. Furthermore, considering there is no available dataset for evaluation, we build two 3D datasets, INOR-1 and INOR-2, to evaluate these two new tasks. Extensive experimental results demonstrate that our method can significantly outperform other state-of-the-art class-incremental learning methods. Even if we store 500-1000 fewer 3D objects than SOTA methods, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BDNet</i> still achieves comparable performance.

Walk in views: Multi-view path aggregation graph network for 3D shape analysis

The graph-based multi-view methods have achieved state-of-the-art results in 3D shape analysis tasks by taking advantage of graph convolutional networks (GCN) to process discrete data. However, the homogeneity of the traditional GCN aggregation operator leads to a problem in aggregating neighborhood information, i.e., if several views have the same neighbors, the same node embeddings will be generated, resulting in feature redundancy. To address this problem, we propose a Multi-view Path Aggregation Graph Network (MVPNet) for 3D shape analysis, which aims to extract a particular path from a graph composed of multiple views and aggregate it into an effective 3D shape descriptor. Specifically, we first extract a path in the graph through dynamic walking, and update the path status while searching for new nodes during the walking. Then we embed the position information of the nodes in the order of the nodes in the path. Finally, we propose to aggregate the features of a path employing a Path Transformer that is capable of handling ordered sequences. A path contains richer semantic and structural information than a traditional subgraph. To demonstrate the effectiveness of our proposed method, we conduct extensive experiments on three benchmark datasets, namely ModelNet, ShapeNetCore55 and MCB, and these experiments prove that the method outperforms the current methods in 3D shape classification and retrieval tasks.

PVFAN: Point-view fusion attention network for 3D shape recognition

3D shape recognition is a critical research topic in the field of computer vision, attracting substantial attention. Existing approaches mainly focus on extracting distinctive 3D shape features; however, they often neglect the model’s robustness and lack refinement in deep features. To address these limitations, we propose the point-view fusion attention network that aims to extract a concise, informative, and robust 3D shape descriptor. Initially, our approach combines multi-view features with point cloud features to obtain accurate and distinguishable fusion features. To effectively handle these fusion features, we design a dual-attention convolutional network which consists of a channel attention module and a spatial attention module. This dual-attention mechanism greatly enhances the generalization ability and robustness of 3D recognition models. Notably, we introduce a strip-pooling layer in the channel attention module to refine the features, resulting in improved fusion features that are more compact. Finally, a classification process is performed on the refined features to assign appropriate 3D shape labels. Our extensive experiments on the ModelNet10 and ModelNet40 datasets for 3D shape recognition and retrieval demonstrate the remarkable accuracy and robustness of the proposed method.

Self-supervised Multi-view Learning via Auto-encoding 3D Transformations

3D object representation learning is a fundamental challenge in computer vision to infer about the 3D world. Recent advances in deep learning have shown their efficiency in 3D object recognition, among which view-based methods have performed best so far. However, feature learning of multiple views in existing methods is mostly performed in a supervised fashion, which often requires a large amount of data labels with high costs. In contrast, self-supervised learning aims to learn multi-view feature representations without involving labeled data. To this end, we propose a novel self-supervised framework to learn Multi-View Transformation Equivariant Representations (MV-TER), exploring the equivariant transformations of a 3D object and its projected multiple views that we derive. Specifically, we perform a 3D transformation on a 3D object and obtain multiple views before and after the transformation via projection. Then, we train a representation encoding module to capture the intrinsic 3D object representation by decoding 3D transformation parameters from the fused feature representations of multiple views before and after the transformation. Experimental results demonstrate that the proposed MV-TER significantly outperforms the state-of-the-art view-based approaches in 3D object classification and retrieval tasks and show the generalization to real-world datasets. The code is available at https://github.com/gyshgx868/mvter .

Rethinking Rotation Invariance with Point Cloud Registration

Recent investigations on rotation invariance for 3D point clouds have been devoted to devising rotation-invariant feature descriptors or learning canonical spaces where objects are semantically aligned. Examinations of learning frameworks for invariance have seldom been looked into. In this work, we review rotation invariance (RI) in terms of point cloud registration (PCR) and propose an effective framework for rotation invariance learning via three sequential stages, namely rotation-invariant shape encoding, aligned feature integration, and deep feature registration. We first encode shape descriptors constructed with respect to reference frames defined over different scales, e.g., local patches and global topology, to generate rotation-invariant latent shape codes. Within the integration stage, we propose an Aligned Integration Transformer (AIT) to produce a discriminative feature representation by integrating point-wise self- and cross-relations established within the shape codes. Meanwhile, we adopt rigid transformations between reference frames to align the shape codes for feature consistency across different scales. Finally, the deep integrated feature is registered to both rotation-invariant shape codes to maximize their feature similarities, such that rotation invariance of the integrated feature is preserved and shared semantic information is implicitly extracted from shape codes. Experimental results on 3D shape classification, part segmentation, and retrieval tasks prove the feasibility of our framework. Our project page is released at: https://rotation3d.github.io/.

Learning View-Based Graph Convolutional Network for Multi-View 3D Shape Analysis.

View-based approach that recognizes 3D shape through its projected 2D images has achieved state-of-the-art results for 3D shape recognition. The major challenges are how to aggregate multi-view features and deal with 3D shapes in arbitrary poses. We propose two versions of a novel view-based Graph Convolutional Network, dubbed view-GCN and view-GCN++, to recognize 3D shape based on graph representation of multiple views. We first construct view-graph with multiple views as graph nodes, then design two graph convolutional networks over the view-graph to hierarchically learn discriminative shape descriptor considering relations of multiple views. Specifically, view-GCN is a hierarchical network based on two pivotal operations, i.e., feature transform based on local positional and non-local graph convolution, and graph coarsening based on a selective view-sampling operation. To deal with rotation sensitivity, we further propose view-GCN++ with local attentional graph convolution operation and rotation robust view-sampling operation for graph coarsening. By these designs, view-GCN++ achieves invariance to transformations under the finite subgroup of rotation group SO(3). Extensive experiments on benchmark datasets (i.e., ModelNet40, ScanObjectNN, RGBD and ShapeNet Core55) show that view-GCN and view-GCN++ achieve state-of-the-art results for 3D shape classification and retrieval tasks under aligned and rotated settings.

EFSCNN: Encoded Feature Sphere Convolution Neural Network for fast non-rigid 3D models classification and retrieval

Traditional methods of classification and retrieval for non-rigid 3D models, such as topological structure-based methods and spectral analysis-based methods, high rely on manual interventions, which leads to the fail of generalization. Recently, methods based on deep learning have been proposed for rigid 3D models, but they are usually not suitable for non-rigid 3D models. Due to excessive redundant calculations, point clouds-based methods and multi-view-based methods are also inefficient. In this paper, a novel deep learning framework for non-rigid 3D model classifications is presented. Different from other point clouds-based methods and multi-view-based method, this framework utilize a 3D sphere view and a new convolution way. First, an encoded feature sphere (EFS) is constructed to express the non-rigid 3D model through feature encoding, spherical projection, and crucial points extraction. Then, loop convolution and longitudinal–latitudinal loop convolution are combined into EFSCNN to extract the discriminative feature descriptor of the non-rigid 3D model. EFSCNN is a hierarchical network based on multi-direction convolution, which expands the receptive field and enhances the ability to capture local features. With the single view input EFS, EFSCNN achieves competitive performance on major non-rigid model classification and retrieval benchmarks with significantly faster runtime than previous studies.

MVContrast: Unsupervised Pretraining for Multi-view 3D Object Recognition

3D shape recognition has drawn much attention in recent years. The view-based approach performs best of all. However, the current multi-view methods are almost all fully supervised, and the pretraining models are almost all based on ImageNet. Although the pretraining results of ImageNet are quite impressive, there is still a significant discrepancy between multi-view datasets and ImageNet. Multi-view datasets naturally retain rich 3D information. In addition, large-scale datasets such as ImageNet require considerable cleaning and annotation work, so it is difficult to regenerate a second dataset. In contrast, unsupervised learning methods can learn general feature representations without any extra annotation. To this end, we propose a three-stage unsupervised joint pretraining model. Specifically, we decouple the final representations into three fine-grained representations. Data augmentation is utilized to obtain pixel-level representations within each view. And we boost the spatial invariant features from the view level. Finally, we exploit global information at the shape level through a novel extract-and-swap module. Experimental results demonstrate that the proposed method gains significantly in 3D object classification and retrieval tasks, and shows generalization to cross-dataset tasks.

FuS-GCN: Efficient B-rep based graph convolutional networks for 3D-CAD model classification and retrieval

Performing 3-dimensional computer-aided design (3D-CAD) model classification, retrieval, and reuse is of vital importance in industrial manufacturing, as it considerably shortens the engineering development cycle and reduces development costs. Although existing 3D model classification and retrieval methods achieve satisfactory performance when operating on meshes or point clouds, they cannot be applied directly to 3D-CAD models, which are generally represented by the boundary representation (B-rep). To address this issue, and to fully exploit the topology of B-rep, a graph structure descriptor called B-rep graph is proposed to pre-process B-rep data, and to extract the precise topological and geometric features from 3D-CAD models. Meanwhile, a novel efficient neural network called FuS-GCN, based on graph convolutional networks (GCNs), is designed to handle this graph data. To better extract the graph features and to improve the effect of pooling, the self-attention mechanism and feature fusion are incorporated into the pooling layer, yielding the proposed fusion self-attention graph pooling (FuSPool) algorithm. Finally, we demonstrate the effectiveness of FuS-GCN on 3D-CAD model data, while outperforming alternative 3D shape descriptors such as point clouds, voxels, and meshes.

Toward 3D Retrieval of Exoplanet Atmospheres: Assessing Thermochemical Equilibrium Estimation Methods

Characterizing exoplanetary atmospheres via Bayesian retrievals requires assuming some chemistry model, such as thermochemical equilibrium or parameterized abundances. The higher-resolution data offered by upcoming telescopes enable more complex chemistry models within retrieval frameworks. Yet many chemistry codes that model more complex processes like photochemistry and vertical transport are computationally expensive, and directly incorporating them into a 1D retrieval model can result in prohibitively long execution times. Additionally, phase-curve observations with upcoming telescopes motivate 2D and 3D retrieval models, further exacerbating the lengthy runtime for retrieval frameworks with complex chemistry models. Here we compare thermochemical equilibrium approximation methods based on their speed and accuracy with respect to a Gibbs energy-minimization code. We find that, while all methods offer orders-of-magnitude reductions in computational cost, neural network surrogate models perform more accurately than the other approaches considered, achieving a median absolute dex error of <0.03 for the phase space considered. While our results are based on a 1D chemistry model, our study suggests that higher-dimensional chemistry models could be incorporated into retrieval models via this surrogate modeling approach.

Application of computer vision techniques for 3D matching and retrieval of archaeological objects.

As cultural institutions embark in projects oriented to digitise art and archaeological collections in three dimensions, the need for developing means to access the resulting 3D models has become imperative. Shape recognition techniques developed in the field of computer vision can help in this task. This paper describes the implementation of three shape descriptors, specifically shape distributions, reflective symmetry and spherical harmonics as part of the development of a search engine that retrieves 3D models from an archaeological database without the need of using keywords as query criteria. The usefulness of this system is obvious in the context of cultural heritage museums, where it is essential to provide automatic access to archaeological and art collections. The prototype described in this paper uses, as study case, 3D models of archaeological objects belonging to Museo del Templo Mayor, a Mexican institution that preserves one of the largest collections of Aztec cultural heritage. This work is part of an ongoing project focused on creating generic methodologies and user-friendly computational tools for shape analysis for the benefit of scholars and students interested in describing, interpreting and disseminating new knowledge about the morphology of cultural objects.

KSS-ICP: Point Cloud Registration based on Kendall Shape Space.

Point cloud registration is a popular topic that has been widely used in 3D model reconstruction, location, and retrieval. In this paper, we propose a new registration method, KSS-ICP, to address the rigid registration task in Kendall shape space (KSS) with Iterative Closest Point (ICP). The KSS is a quotient space that removes influences of translations, scales, and rotations for shape feature-based analysis. Such influences can be concluded as the similarity transformations that do not change the shape feature. The point cloud representation in KSS is invariant to similarity transformations. We utilize such property to design the KSS-ICP for point cloud registration. To tackle the difficulty to achieve the KSS representation in general, the proposed KSS-ICP formulates a practical solution that does not require complex feature analysis, data training, and optimization. With a simple implementation, KSS-ICP achieves more accurate registration from point clouds. It is robust to similarity transformation, non-uniform density, noise, and defective parts. Experiments show that KSS-ICP has better performance than the state-of-the-art. Code1 and executable files2 are made public.

Enhancing Brain Tumor Diagnosis: A 3D Auto-Encoding Approach for Accurate Classification

The brain’s capacity to control and coordinate the body’s other organs makes it an integral part of the nervous system. Brain tumours, which form when abnormal cells in the brain grow uncontrolled, may be deadly if not diagnosed and treated promptly. The use of image processing technology is essential in the quest to identify malignancies in medical imaging. Because of this, the final depiction will be more complete. Only a tiny percentage of 3D form instances will be viable for feature learning because of its complex spatial structure. These issues have inspired the development of some potential solutions, such as automatic encoders for learning properties from 2D images and the translation of 3D shapes into 2D space. With the help of camera images and state-space structures, the suggested 3D-based Spatial Auto Encoder method may automatically learn a representation of the state. Autoencoders can be taught to use the prototypes they generate to rebuild a picture, with the resulting learned coefficients being put to use in 3D form matching and retrieval. It’s possible that the learned coefficients may serve this function. The auto-encoder’s impressive results in image retrieval have been attributed, at least in part, to the ease with which it can learn new features from existing ones.

Deep-Learning-Based 3D Reconstruction: A Review and Applications.

In recent years, deep learning models have been widely used in 3D reconstruction fields and have made remarkable progress. How to stimulate deep academic interest to effectively manage the explosive augmentation of 3D models has been a research hotspot. This work shows mainstream 3D model retrieval algorithm programs based on deep learning currently developed remotely, and further subdivides their advantages and disadvantages according to the behavior evaluation of the algorithm programs obtained by trial. According to other restoration applications, the main 3D model retrieval algorithms can be divided into two categories: (1) 3D standard restoration methods supported by the model, i.e., both the restored object and the recalled object are 3D models. It can be further divided into voxel-based, point coloring-based, and appearance-based methods, and (2) cross-domain 3D model recovery methods supported by 2D replicas, that is, the retrieval motivation is 2D images, and the recovery appearance is 3D models, including retrieval methods supported by 2D display, 2D depiction-based realistic replication and 3D mold recovery methods. Finally, the work proposed novel 3D fashion retrieval algorithms supported by deep science that are analyzed and ventilated, and the unaccustomed directions of future development are prospected.

GPROF-NN: a neural-network-based implementation of the Goddard Profiling Algorithm

Abstract. The Global Precipitation Measurement (GPM) mission measures global precipitation at a temporal resolution of a few hours to enable close monitoring of the global hydrological cycle. GPM achieves this by combining observations from a spaceborne precipitation radar, a constellation of passive microwave (PMW) sensors, and geostationary satellites. The Goddard Profiling Algorithm (GPROF) is used operationally to retrieve precipitation from all PMW sensors of the GPM constellation. Since the resulting precipitation rates serve as input for many of the level 3 retrieval products, GPROF constitutes an essential component of the GPM processing pipeline. This study investigates ways to improve GPROF using modern machine learning methods. We present two neural-network-based, probabilistic implementations of GPROF: GPROF-NN 1D, which (just like the current GPROF implementation) processes pixels individually, and GPROF-NN 3D, which employs a convolutional neural network to incorporate structural information into the retrieval. The accuracy of the retrievals is evaluated using a test dataset consistent with the data used in the development of the GPROF and GPROF-NN retrievals. This allows for assessing the accuracy of the retrieval method isolated from the representativeness of the training data, which remains a major source of uncertainty in the development of precipitation retrievals. Despite using the same input information as GPROF, the GPROF-NN 1D retrieval improves the accuracy of the retrieved surface precipitation for the GPM Microwave Imager (GMI) from 0.079 to 0.059 mm h−1 in terms of mean absolute error (MAE), from 76.1 % to 69.5 % in terms of symmetric mean absolute percentage error (SMAPE) and from 0.797 to 0.847 in terms of correlation. The improvements for the Microwave Humidity Sounder (MHS) are from 0.085 to 0.061 mm h−1 in terms of MAE, from 81 % to 70.1 % for SMAPE, and from 0.724 to 0.804 in terms of correlation. Comparable improvements are found for the retrieved hydrometeor profiles and their column integrals, as well as the detection of precipitation. Moreover, the ability of the retrievals to resolve small-scale variability is improved by more than 40 % for GMI and 29 % for MHS. The GPROF-NN 3D retrieval further improves the MAE to 0.043 mm h−1; the SMAPE to 48.67 %; and the correlation to 0.897 for GMI and 0.043 mm h−1, 63.42 %, and 0.83 for MHS. Application of the retrievals to GMI observations of Hurricane Harvey shows moderate improvements when compared to co-located GPM-combined and ground-based radar measurements indicating that the improvements at least partially carry over to assessment against independent measurements. Similar retrievals for MHS do not show equally clear improvements, leaving the validation against independent measurements for future investigation. Both GPROF-NN algorithms make use of the same input and output data as the original GPROF algorithm and thus may replace the current implementation in a future update of the GPM processing pipeline. Despite their superior accuracy, the single-core runtime required for the operational processing of an orbit of observations is lower than that of GPROF. The GPROF-NN algorithms promise to be a simple and cost-efficient way to increase the accuracy of the PMW precipitation retrievals of the GPM constellation and thus improve the monitoring of the global hydrological cycle.