Multimodal 3D Research Articles

Multimodal 2D Ferroelectric Transistor with Integrated Perception-and-Computing-in-Memory Functions for Reservoir Computing.

Emerging neuromorphic hardware promises energy-efficient computing by colocating multiple essential functions at the individual component level. The implementation is challenging due to mismatch between the characteristics of multifunctional devices and neural networks. Here, we demonstrate an artificial synapse based on a 2D α-phase indium selenide that exhibits integrated perception-and-computing-in-memory functions in a single-transistor setup, serving as a basic building block for reservoir computing. Extending to the array architecture enables concurrent image-sensing and memory. Further, we implement multimode deep-reservoir computing with adjustable nonlinear transformation and multisensory fusion using this core device. In the lane-keeping-assistance task for an unmanned vehicle, the system demonstrates ∼104 times lower energy consumption and significantly boosted data throughput compared to the state-of-the-art graphics processors. The demonstrated perception-and-computing-in-memory (PCIM) functions at a single-transistor level shows the feasibility of implementing ultrascalable, resource-efficient hardware for brain-inspired computing.

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

Identification of mild cognitive impairment using multimodal 3D imaging data and graph convolutional networks

Abstract Objective. Mild cognitive impairment (MCI) is a precursor stage of dementia characterized by mild cognitive decline in one or more cognitive domains, without meeting the criteria for dementia. MCI is considered a prodromal form of Alzheimer's disease (AD). Early identification of MCI is crucial for both intervention and prevention of AD. To accurately identify MCI, a novel multimodal 3D imaging data integration graph convolutional network model is designed in this paper. Approach. The proposed model utilizes 3D-VGGNet to extract three-dimensional features from multimodal imaging data (such as sMRI and FDG-PET), which are then fused into feature vectors as the node features of a population graph. Non-imaging features of participants are combined with the multimodal imaging data to construct a population sparse graph. Additionally, in order to optimize the connectivity of the graph, we employed the Pairwise Attribute Estimation (PAE) method to compute the edge weights based on non-imaging data, thereby enhancing the effectiveness of the graph structure. Subsequently, a population-based graph convolutional network (GCN) integrates the structural and functional features of different modal images into the features of each participant for MCI classification. Main results. Experiments on the ADNI demonstrated accuracies of 98.57%, 96.03%, and 96.83% for the NC-EMCI, NC-LMCI, and EMCI-LMCI classification tasks, respectively. The AUC, specificity, sensitivity, and F1-score are also superior to state-of-the-art models, demonstrating the effectiveness of the proposed model. Furthermore, the proposed model is applied to the ABIDE dataset for autism diagnosis, achieving an accuracy of 91.43% and outperforming the state-of-the-art models, indicating excellent generalization capabilities of the proposed model. Significance. This study demonstrate the proposed model’s ability to integrate multimodal imaging data and its excellent ability to recognize MCI. This will help achieve early warning for AD and intelligent diagnosis of other brain neurodegenerative diseases.

Multimodal 3D quantification of particle stimulated nucleation in industrially manufactured aluminium AA5182 sheet

Particle stimulated nucleation is a dominant recrystallisation mechanism observed in many industrially relevant aluminium alloys during thermomechanical processing. In this work, we quantify particle stimulated nucleation in 3D in an aluminium AA5182 alloy sheet cold-rolled to 75 % thickness reduction. Second phase particles and nuclei are mapped in the same sample volume by conventional laboratory absorption X-ray tomography and synchrotron X-ray Laue micro-diffraction. The large second phase particles are classified as Fe- and Mg-rich phases. It is found that84 % of the nuclei are particle stimulated and 40 % of the particles stimulate nucleation. The critical particle diameter is found to be 4μm. Deviatoric elastic strains are derived from micro-diffraction data and it is found that elastic strains are present in the recrystallised nuclei. The effects of the different particle types, particle clustering, particle size and aspect ratio as well as strain inheritance are discussed. This work provides a full 3D quantification of particle stimulated nucleation behaviour in AA5182 alloy sheet deformed to high strain.

GAN-based image generation

Generative Adversarial Networks (GANs), as a deep learning model, have made significant progress in the field of image generation and style migration. This study aims to methodically investigate GAN-based image generation methods. First, this paper outlines the basic principles of GAN and its application in image generation, focusing on analyzing the structure and performance of representative models such as DCGAN, ProGAN and StyleGAN. This paper summarizes the improvement methods such as WGAN and LSGAN, and evaluates their efficacy in improving the stability of the model and the quality of the generated images in view of the problems of pattern collapse and instability faced by GANs in the training process. Finally, this paper discusses the limitations of current techniques and possible future directions, and suggests research prospects in the field of multimodal fusion and 3D image generation. The research in this paper provides a theoretical framework and useful suggestions for enhancing the application of GAN methods in image generation.

Application of Multimodal Reconstruction Technology and 3D Printing Technology in MVD Surgery.

Microvascular decompression (MVD) plays a pivotal role in the treatment of cranial neurovascular compression syndromes, yet the safety and precision of the surgery remain a focus of clinical attention. This article delves into the application of multimodal reconstruction and 3D printing technologies in MVD surgeries, evaluating their effectiveness in preoperative planning. Multimodal reconstruction, by integrating various imaging techniques such as magnetic resonance imaging (MRI) and computed tomography (CT), provides high-resolution anatomical information, offering comprehensive data support for preoperative planning and intraoperative navigation. Complementing this, 3D printing technology presents patients' anatomical structures as individualized physical models, enabling surgeons to fabricate corresponding skin templates for surgical needs, offering intuitive and practical references. Case studies presented in this article demonstrate the application and efficacy of these technologies in actual MVD surgeries. The results suggest that multimodal reconstruction and 3D printing technologies aid surgical teams in better understanding patients' anatomical structures during preoperative planning, enhancing surgical accuracy, reducing operative time, and shortening hospital stays. Despite notable advancements in MVD surgeries, challenges such as data accuracy, technological complexity, and cost persist. Future research should aim to address these issues, further optimizing the technologies and promoting their widespread application in neurosurgical procedures. Through in-depth investigation and understanding of these advanced technologies, we hope to pave new paths for improving surgical outcomes and patients' quality of life.

Multimodal Image Confidence: A Novel Method for Tumor and Organ Boundary Representation

BackgroundThe ambiguous boundaries of tumors and organs at risk (OARs) seen in medical images pose challenges in treatment planning and other tasks in radiotherapy. MethodsThis study introduces an innovative analytical algorithm, Multi-Modal Image Confidence (MMC), which exploits the collective strengths of complementary multi-modal medical images to determine a confidence measure for each voxel belonging to the region of interest (ROI). MMC facilitates the creation of modality-specific ROI-enhanced images, enabling a detailed representation of the ROI's boundaries and internal features. By employing an interpretable mathematical model that propagates voxel confidence based on inter-voxel correlations, MMC avoids the need for model training, distinguishing it from deep learning (DL)-based methods. ResultsThe performance of the proposed algorithm was qualitatively and quantitatively evaluated using 156 nasopharyngeal carcinoma cases and 1251 glioma cases. Qualitative assessments underscored the accuracy of MMC and ROI-enhanced images in estimating lesion boundaries and capturing internal tumor characteristics. Quantitative analyses revealed strong agreement between MMC and manual delineations. ConclusionThis paper introduces a novel analytical algorithm to identify and depict ROI boundaries based on complementary multi-modal 3D medical images. The applicability of the proposed method can extend to both targets and OARs at diverse anatomical sites across multiple image modalities, amplifying its potential for augmenting radiotherapy-related tasks.

Ultracompact 3D Splitter for Single‐Core to Multi‐Core Optical Fiber Connections Fabricated Through Direct Laser Writing in Polymer

AbstractThe deployment and advancement of high‐bandwidth communication networks, quantum information systems, and sensing platforms relying on multi‐core optical fibers (MCFs) are challenged by the scarcity of cost‐effective, compact, and efficient optical interfacing components. This study introduces an unprecedented 3D‐printed 1 × 4 splitter for MCFs fabricated with 2‐photon polymerization‐based direct laser writing. The pivotal element is a triangular cross‐section 3D multimode interference (MMI) coupler, supplemented with S‐bends and adiabatic tapers to facilitate the splitting of a signal from a single core of a single‐mode fiber into four cores of a multi‐core fiber. All components are initially designed and assessed to minimize loss and polarization dependence across the C‐ and L‐bands using optical simulation. Subsequently, a proof‐of‐concept model of the splitter, compactly integrated within the fiber volume, featuring a remarkably short length of 180 µm and insertion loss of ≈−3 dB, is fabricated. The manufacturing speed, minimal loss, component compactness, and flexibility of the approach, collectively present promising avenues for pioneering developments in MCF‐coupling components.

From coin to 3D face sculpture portraits in the round of Roman emperors

Representing historical figures on visual media has always been a crucial aspect of political communication in the ancient world, as it is in modern society. A great example comes from ancient Rome, when the emperor’s portraits were serially replicated on visual media to disseminate his image across the countries ruled by the Romans and to assert the power and authority that he embodied by making him universally recognizable. In particular, one of the most common media through which ancient Romans spread the imperial image was coinage, which showed a bi-dimensional projection of his portrait on the very low relief produced by the impression of the coin-die. In this work, we propose a new method that uses a multi-modal 2D and 3D approach to reconstruct the full portrait in the round of Roman emperors from their images adopted on ancient coins. A well-defined pipeline is introduced from the digitization of coins using 3D scanning techniques to the estimation of the 3D model of the portrait represented by a polygonal mesh. A morphable model trained on real 3D faces is exploited to infer the morphological (i.e., geometric) characteristics of the Roman emperor from the contours extracted from a coin portrait using a model fitting procedure. We present examples of face reconstruction of different emperors from coins produced in Rome as well as in the imperial provinces, which sometimes showed local variations of the official portraits centrally designed.

IMG-23. AN MRI-BASED RADIOMIC SIGNATURE TO PREDICT LONG-TERM SURVIVAL IN PATIENTS WITH DIFFUSE INTRINSIC PONTINE GLIOMAS

Abstract BACKGROUND Diffuse intrinsic pontine gliomas are characterized by a short life expectancy after diagnosis, with a mean overall survival of less than 12 months. To stratify patients and possibly better adapt therapy, we are proposing a radiomic signature that predicts, at the time of diagnosis, which patients might survive beyond 2 years. METHODS A retrospective database comprising 80 patients who had undergone 3D multimodal MRI at the time of diagnosis was compiled. All images were acquired at a single center, and the protocol included T1-weighted images acquired before and after contrast injection, T2-weighted and FLAIR images. Depending on the patient, some modalities were missing (around 10% per modality). After manual tumor delineation, a dedicated radiomic pipeline extracted 79 features (first-order statistics and texture parameters) per imaging modality. In addition, gender, age and 14 tumor shape features were collected without missing data. After feature selection (maximum of 4 features per model), five logistic regression models were defined, one per modality and another for clinical and shape variables. Each patient was eligible for 2 to 5 models, with an average score calculated over these models. RESULTS Based on a leave-one-out approach, the multi-model was able to define patients who survive more than 2 years correctly, (sensitivity=100%, specificity =79%). A prediction could be made for each patient despite missing MR modalities. Defining a radiomic signature based on the multi-model score (cut-off=0.5), Kaplan-Meier analysis shows a good separation of the two groups (p-value<0.001). This result is superior to that obtained with the TP53 mutation (available for 61 patients, with p-value<0.01). CONCLUSIONS We have demonstrated the value of multimodal 3D MRI in predicting which DIPG patients might survive beyond 2 years. The robustness of this radiomic signature has yet to be tested using multicentric images.

Multimodal 2D and 3D microscopic mapping of growth cartilage by computational imaging techniques – a short review including new research

Being able to image the microstructure of growth cartilage is important for understanding the onset and progression of diseases such as osteochondrosis and osteoarthritis, as well as for developing new treatments and implants. Studies of cartilage using conventional optical brightfield microscopy rely heavily on histological staining, where the added chemicals provide tissue-specific colours. Other microscopy contrast mechanisms include polarization, phase- and scattering contrast, enabling non-stained or ‘label-free’ imaging that significantly simplifies the sample preparation, thereby also reducing the risk of artefacts. Traditional high-performance microscopes tend to be both bulky and expensive. Computational imaging denotes a range of techniques where computers with dedicated algorithms are used as an integral part of the image formation process. Computational imaging offers many advantages like 3D measurements, aberration correction and quantitative phase contrast, often combined with comparably cheap and compact hardware. X-ray microscopy is also progressing rapidly, in certain ways trailing the development of optical microscopy. In this study, we first briefly review the structures of growth cartilage and relevant microscopy characterization techniques, with an emphasis on Fourier ptychographic microscopy (FPM) and advanced x-ray microscopies. We next demonstrate with our own results computational imaging through FPM and compare the images with hematoxylin eosin and saffron (HES)-stained histology. Zernike phase contrast, and the nonlinear optical microscopy techniques of second harmonic generation (SHG) and two-photon excitation fluorescence (TPEF) are explored. Furthermore, X-ray attenuation-, phase- and diffraction-contrast computed tomography (CT) images of the very same sample are presented for comparisons. Future perspectives on the links to artificial intelligence, dynamic studies and in vivo possibilities conclude the article.

Development and Evaluation of a Two-Staged 3D Keypoint Based Workflow for the Co-Registration of Unstructured Multi-Temporal and Multi-Modal 3D Point Clouds

Abstract. Robust and automated point cloud registration methods are required in many geoscience applications using multi-temporal and multi-modal 3D point clouds. Therefore, a 3D keypoint-based coarse registration workflow has been implemented, utilizing the ISS keypoint detector and 3DSmoothNet descriptor. This paper contributes to keypoint-based registration research through variations of the standard workflow proposed in the literature, applying a two-staged strategy of global and local keypoint matching as well as prototypical keypoint projection and fine registration based on ICP. Further, by testing the utilized detector and descriptor on unstructured, multi-temporal and multi-source point clouds with variations in point cloud density, generalization ability is tested outside benchmark data. Therefore, data of the Bøverbreen glacier in Jotunheimen, Norway has been acquired in 2022 and 2023, deploying UAV-based image matching and terrestrial laser scanning. The results show good performance of the implemented robust matching algorithm PROSAC, requiring fewer iterations than the well-known RANSAC approach, but solving the rigid body transformation with TEASER++ is faster and more robust to outliers without demanding pre-knowledge of the data. Further, the results identify the keypoint detection as most limiting factor in speed and accuracy. Summarizing, keypoint-based coarse registration on low density point clouds, applying a global and local matching strategy and transformation estimation using TEASER++ is recommended. Keypoint projection shows potential, increasing number and precision in low density clouds, but has to be more robust. Further research needs to be carried out, focusing on identifying a fast and robust keypoint detector.

MMFG: Multimodal-based Mutual Feature Gating 3D Object Detection

To address the problem that image and point cloud features are fused in a coarse fusion way and cannot achieve deep fusion, this paper proposes a multimodal 3D object detection architecture based on a mutual feature gating mechanism. First, since the feature aggregation approach based on the set abstraction layer cannot obtain fine-grained features, a point-based self-attention mechanism module is designed. This module is added to the extraction branch of point cloud features to achieve fine-grained feature aggregation while maintaining accurate location information. Second, a new gating mechanism is designed for the deep fusion of image and point cloud. Deep fusion is achieved by mutual feature weighting between the image and the point cloud. The newly fused features are then fed into a feature refinement network to extract classification confidence and 3D target bounding boxes. Finally, a multi-scale detection architecture is proposed to obtain a more complete object shape. The location-based encoding feature algorithm is also designed to focus the interest points in the region of interest adaptively. The whole architecture shows outstanding performance on the KITTI3D and nuSenece datasets, especially at the difficult level. It shows that the framework solves the problem of low detection rates in LiDAR mode due to the low number of surface points obtained from distant objects.

Relative Entropy Gradient Sampler for Unnormalized Distribution

We propose a relative entropy gradient sampler (REGS) for sampling from unnormalized distributions. REGS is a particle method that seeks a sequence of simple nonlinear transforms iteratively pushing the initial samples from a reference distribution into the samples from an unnormalized target distribution. To determine the nonlinear transforms at each iteration, we consider the Wasserstein gradient flow of relative entropy. This gradient flow determines a path of probability distributions that interpolates the reference distribution and the target distribution. It is characterized by an ordinary differential equation (ODE) system with velocity fields depending on the density ratios of the density of evolving particles and the unnormalized target density. To sample with REGS, we need to estimate the density ratios and simulate the ODE system with particle evolution. We propose a novel nonparametric approach to estimating the logarithmic density ratio using neural networks. Extensive simulation studies on challenging multimodal 1D and 2D mixture distributions and Bayesian logistic regression on real datasets demonstrate that REGS has reasonable performance compared with popular samplers based on Wasserstein gradient flows. Supplementary materials for this article are available online.

SCA-PVNet: Self-and-cross attention based aggregation of point cloud and multi-view for 3D object retrieval

To address 3D object retrieval, substantial efforts have been made to generate highly discriminative descriptors for 3D objects represented by a single modality, such as voxels, point clouds, or multiview images. It is promising to leverage complementary information from multimodal representations of 3D objects to further improve retrieval performance. However, multimodal 3D object retrieval has rarely been developed or analyzed for large-scale datasets. In this paper, we propose a self-and-cross-attention-based aggregation of point clouds and multiview images (SCA-PVNet) for 3D object retrieval. With deep features extracted from point clouds and multi-view images, we design two types of feature aggregation modules, namely the in-modality aggregation module (IMAM) and the cross-modality aggregation module (CMAM ), for effective feature fusion. IMAM leverages a self-attention mechanism to aggregate multiview features, whereas CMAM exploits a cross-attention mechanism to interact with point-cloud and multiview features. The final descriptor of a 3D object for object retrieval can be obtained by concatenating the aggregated feature outputs of both modules. Extensive experiments and analyses were conducted on four datasets ranging from small to large scales to demonstrate the superiority of the proposed SCA-PVNet over state-of-the-art methods. In addition to achieving state-of-the-art retrieval performance, our method is more robust in challenging scenarios when views or points are missing during inference.

DenseSphere: Multimodal 3D object detection under a sparse point cloud based on spherical coordinate

Multimodal 3D object detection has gained significant attention due to the fusion of light detection and range (LiDAR) and RGB data. Existing 3D detection models in autonomous driving are typically trained on dense point cloud data from high-specification LiDAR sensors. However, budgetary constraints often lead to adopting low point-per-second (PPS) LiDAR sensors in real-world scenarios. The low PPS specification can generate a sparse point cloud. In this case, the existing models trained on dense data with a high PPS specification cannot achieve optimal performance under a sparse point cloud. To address this problem, we propose DenseSphere for robust multimodal 3D object detection under a sparse point cloud. Considering the data acquisition process of LiDAR sensors, DenseSphere involves the spherical coordinate-based point upsampler. Specifically, points are interpolated in the horizontal or vertical direction using bilateral interpolation. The interpolated points are refined using dilated pyramid blocks with various receptive fields. For efficient fusion with generated dense point cloud, we use a graph-based detector and hierarchical layers. Then, we demonstrate the performance of DenseSphere by comparing it with other multimodal 3D object detection models through experiments. The visual results and source code with the pretrained models are available at https://github.com/Jung-jongwon/DenseSphere.

Fusion of Multimodal Imaging and 3D Digitization Using Photogrammetry.

Multimodal sensors capture and integrate diverse characteristics of a scene to maximize information gain. In optics, this may involve capturing intensity in specific spectra or polarization states to determine factors such as material properties or an individual's health conditions. Combining multimodal camera data with shape data from 3D sensors is a challenging issue. Multimodal cameras, e.g., hyperspectral cameras, or cameras outside the visible light spectrum, e.g., thermal cameras, lack strongly in terms of resolution and image quality compared with state-of-the-art photo cameras. In this article, a new method is demonstrated to superimpose multimodal image data onto a 3D model created by multi-view photogrammetry. While a high-resolution photo camera captures a set of images from varying view angles to reconstruct a detailed 3D model of the scene, low-resolution multimodal camera(s) simultaneously record the scene. All cameras are pre-calibrated and rigidly mounted on a rig, i.e., their imaging properties and relative positions are known. The method was realized in a laboratory setup consisting of a professional photo camera, a thermal camera, and a 12-channel multispectral camera. In our experiments, an accuracy better than one pixel was achieved for the data fusion using multimodal superimposition. Finally, application examples of multimodal 3D digitization are demonstrated, and further steps to system realization are discussed.

Computational limits to the legibility of the imaged human brain

Our knowledge of the organisation of the human brain at the population-level is yet to translate into power to predict functional differences at the individual-level, limiting clinical applications and casting doubt on the generalisability of inferred mechanisms. It remains unknown whether the difficulty arises from the absence of individuating biological patterns within the brain, or from limited power to access them with the models and compute at our disposal.Here we comprehensively investigate the resolvability of such patterns with data and compute at unprecedented scale. Across 23 810 unique participants from UK Biobank, we systematically evaluate the predictability of 25 individual biological characteristics, from all available combinations of structural and functional neuroimaging data. Over 4526 GPU*hours of computation, we train, optimize, and evaluate out-of-sample 700 individual predictive models, including fully-connected feed-forward neural networks of demographic, psychological, serological, chronic disease, and functional connectivity characteristics, and both uni- and multi-modal 3D convolutional neural network models of macro- and micro-structural brain imaging.We find a marked discrepancy between the high predictability of sex (balanced accuracy 99.7%), age (mean absolute error 2.048 years, R2 0.859), and weight (mean absolute error 2.609Kg, R2 0.625), for which we set new state-of-the-art performance, and the surprisingly low predictability of other characteristics. Neither structural nor functional imaging predicted an individual's psychology better than the coincidence of common chronic disease (p < 0.05). Serology predicted chronic disease (p < 0.05) and was best predicted by it (p < 0.001), followed by structural neuroimaging (p < 0.05).Our findings suggest either more informative imaging or more powerful models will be needed to decipher individual level characteristics from the human brain. We make our models and code openly available.

GraphAlign++: An Accurate Feature Alignment by Graph Matching for Multi-Modal 3D Object Detection

GraphAlign++: An Accurate Feature Alignment by Graph Matching for Multi-Modal 3D Object Detection

Hypergraph-Based Multi-Modal Representation for Open-Set 3D Object Retrieval.

The traditional 3D object retrieval (3DOR) task is under the close-set setting, which assumes the categories of objects in the retrieval stage are all seen in the training stage. Existing methods under this setting may tend to only lazily discriminate their categories, while not learning a generalized 3D object embedding. Under such circumstances, it is still a challenging and open problem in real-world applications due to the existence of various unseen categories. In this paper, we first introduce the open-set 3DOR task to expand the applications of the traditional 3DOR task. Then, we propose the Hypergraph-Based Multi-Modal Representation (HGM 2 R) framework to learn 3D object embeddings from multi-modal representations under the open-set setting. The proposed framework is composed of two modules, i.e., the Multi-Modal 3D Object Embedding (MM3DOE) module and the Structure-Aware and Invariant Knowledge Learning (SAIKL) module. By utilizing the collaborative information of modalities derived from the same 3D object, the MM3DOE module is able to overcome the distinction across different modality representations and generate unified 3D object embeddings. Then, the SAIKL module utilizes the constructed hypergraph structure to model the high-order correlation among 3D objects from both seen and unseen categories. The SAIKL module also includes a memory bank that stores typical representations of 3D objects. By aligning with those memory anchors in the memory bank, the aligned embeddings can integrate the invariant knowledge to exhibit a powerful generalized capacity toward unseen categories. We formally prove that hypergraph modeling has better representative capability on data correlation than graph modeling. We generate four multi-modal datasets for the open-set 3DOR task, i.e., OS-ESB-core, OS-NTU-core, OS-MN40-core, and OS-ABO-core, in which each 3D object contains three modality representations: multi-view, point clouds, and voxel. Experiments on these four datasets show that the proposed method can significantly outperform existing methods. In particular, the proposed method outperforms the state-of-the-art by 12.12%/12.88% in terms of mAP on the OS-MN40-core/OS-ABO-core dataset, respectively. Results and visualizations demonstrate that the proposed method can effectively extract the generalized 3D object embeddings on the open-set 3DOR task and achieve satisfactory performance.