Point Cloud Representation Research Articles

Bi-directional information interaction for multi-modal 3D object detection in real-world traffic scenes

Multimodal 3D object detection methods are poorly adapted to real-world traffic scenes due to sparse distribution of point clouds and misalignment multimodal data during actual collection. Among the existing methods, they focus on high-quality open-source datasets, with performance relying on the accurate structural representation of point clouds and the precise mapping relationship between point clouds and images. To solve the above challenges, this paper proposes a multimodal feature-level fusion method based on the bi-directional interaction between image and point cloud. To overcome the sparsity issue in asynchronous multi-modal data, a point cloud densification scheme based on visual guidance and point cloud density guidance is proposed. This scheme can generate object-level virtual point clouds even when the point cloud and image are misaligned. To deal with the unalignment issue between point cloud and image, a bi-directional interaction module based on image-guided interaction with key points of point clouds and point cloud-guided interaction with image context information is proposed. It achieves effective feature fusion even when the point cloud and image are misaligned. The experiments on the VANJEE and KITTI datasets demonstrated the effectiveness of the proposed method, with average precision improvements of 6.20% and 1.54% compared to the baseline.

A Specialized Pipeline for Efficient and Reliable 3D Semantic Model Reconstruction of Buildings from Indoor Point Clouds.

Recent advances in laser scanning systems have enabled the acquisition of 3D point cloud representations of scenes, revolutionizing the fields of Architecture, Engineering, and Construction (AEC). This paper presents a novel pipeline for the automatic generation of 3D semantic models of multi-level buildings from indoor point clouds. The architectural components are extracted hierarchically. After segmenting the point clouds into potential building floors, a wall detection process is performed on each floor segment. Then, room, ground, and ceiling extraction are conducted using the walls 2D constellation obtained from the projection of the walls onto the ground plan. The identification of the openings in the walls is performed using a deep learning-based classifier that separates doors and windows from non-consistent holes. Based on the geometric and semantic information from previously detected elements, the final model is generated in IFC format. The effectiveness and reliability of the proposed pipeline are demonstrated through extensive experiments and visual inspections. The results reveal high precision and recall values in the extraction of architectural elements, ensuring the fidelity of the generated models. In addition, the pipeline's efficiency and accuracy offer valuable contributions to future advancements in point cloud processing.

Artificial intelligence-aided semi-automatic joint trace detection from textured three-dimensional models of rock mass

It is of great importance to obtain precise trace data, as traces are frequently the sole visible and measurable parameter in most outcrops. The manual recognition and detection of traces on high-resolution three-dimensional (3D) models are relatively straightforward but time-consuming. One potential solution to enhance this process is to use machine learning algorithms to detect the 3D traces. In this study, a unique pixel-wise texture mapper algorithm generates a dense point cloud representation of an outcrop with the precise resolution of the original textured three-dimensional (3D) model. A virtual digital image rendering was then employed to capture virtual images of selected regions. This technique helps to overcome limitations caused by the surface morphology of the rock mass, such as restricted access, lighting conditions, and shading effects. After AI-powered trace detection on two-dimensional (2D) images, a 3D data structuring technique was applied to the selected trace pixels. In the 3D data structuring, the trace data were structured through 2D thinning, 3D reprojection, clustering, segmentation, and segment linking. Finally, the linked segments were exported as 3D polylines, with each polyline in the output corresponding to a trace. The efficacy of the proposed method was assessed using a 3D model of a real-world case study, which was used to compare the results of artificial intelligence (AI)-aided and human intelligence trace detection. Rosette diagrams, which visualize the distribution of trace orientations, confirmed the high similarity between the automatically and manually generated trace maps. In conclusion, the proposed semi-automatic method was easy to use, fast, and accurate in detecting the dominant jointing system of the rock mass.

Automated Three-Dimensional Imaging and Pfirrmann Classification of Intervertebral Disc Using a Graphical Neural Network in Sagittal Magnetic Resonance Imaging of the Lumbar Spine.

This study aimed to develop a graph neural network (GNN) for automated three-dimensional (3D) magnetic resonance imaging (MRI) visualization and Pfirrmann grading of intervertebral discs (IVDs), and benchmark it against manual classifications. Lumbar IVD MRI data from 300 patients were retrospectively analyzed. Two clinicians assessed the manual segmentation and grading for inter-rater reliability using Cohen's kappa. The IVDs were then processed and classified using an automated convolutional neural network (CNN)-GNN pipeline, and their performance was evaluated using F1 scores. Manual Pfirrmann grading exhibited moderate agreement (κ = 0.455-0.565) among the clinicians, with higher exact match frequencies at lower lumbar levels. Single-grade discrepancies were prevalent except at L5/S1. Automated segmentation of IVDs using a pretrained U-Net model achieved an F1 score of 0.85, with a precision and recall of 0.83 and 0.88, respectively. Following 3D reconstruction of the automatically segmented IVD into a 3D point-cloud representation of the target intervertebral disc, the GNN model demonstrated moderate performance in Pfirrmann classification. The highest precision (0.81) and F1 score (0.71) were observed at L2/3, whereas the overall metrics indicated moderate performance (precision: 0.46, recall: 0.47, and F1 score: 0.46), with variability across spinal levels. The integration of CNN and GNN offers a new perspective for automating IVD analysis in MRI. Although the current performance highlights the need for further refinement, the moderate accuracy of the model, combined with its 3D visualization capabilities, establishes a promising foundation for more advanced grading systems.

Generative Variational-Contrastive Learning for Self-Supervised Point Cloud Representation.

Self-supervised representation learning for 3D point clouds has attracted increasing attention. However, existing methods in the field of 3D computer vision generally use fixed embeddings to represent the latent features, and impose hard constraints on the embeddings to make the latent feature values of the positive samples converge to consistency, which limits the ability of feature extractors to generalize over different data domains. To address this issue, we propose a Generative Variational-Contrastive Learning (GVC) model, where Gaussian distribution is used to construct a continuous, smoothed representation of the latent features. A distribution constraint and cross-supervision are constructed to improve the transfer ability of the feature extractor over synthetic and real-world data. Specifically, we design a variational contrastive module to constrain the feature distribution instead of feature values corresponding to each sample in the latent space. Moreover, a generative cross-supervision module is introduced to preserve the invariance features and promote the consistency of feature distribution among positive samples. Experimental results demonstrate that GVC achieves SOTA on different downstream tasks. In particular, with only pre-training on the synthetic dataset, GVC achieves a lead of 8.4% and 14.2% when transferring to the real-world dataset in the linear classification and few-shot classification.

URINet: Unsupervised point cloud rotation invariant representation learning via semantic and structural reasoning

In recent years, many rotation-invariant networks have been proposed to alleviate the interference caused by point cloud arbitrary rotations. These networks have demonstrated powerful representation learning capabilities. However, most of those methods rely on costly manually annotated supervision for model training. Moreover, they fail to reason the structural relations and lose global information. To address these issues, we present an unsupervised method for achieving comprehensive rotation invariant representations without human annotation. Specifically, we propose a novel encoder–decoder architecture named URINet, which learns a point cloud representation by combining local semantic and global structural information, and then reconstructs the input without rotation perturbation. In detail, the encoder is a two-branch network where the graph convolution based structural branch models the relationships among local regions to learn global structural knowledge and the semantic branch learns rotation invariant local semantic features. The two branches derive complementary information and explore the point clouds comprehensively. Furthermore, to avoid the self-reconstruction ambiguity brought by uncertain poses, a bidirectional alignment is proposed to measure the quality of reconstruction results without orientation knowledge. Extensive experiments on downstream tasks show that the proposed method significantly surpasses existing state-of-the-art methods on both synthetic and real-world datasets.

Fast supervoxel segmentation of connectivity median simulation based on Manhattan distance

Supervoxels provide a more natural and compact 3D point cloud representation, which is a fundamental problem in point cloud processing and has attracted extensive attention from many scholars. At present, although the supervoxel segmentation has achieved significant results, the processing efficiency still restricts its further processing in the face of large-scale point cloud processing. Therefore, this paper proposes a new supervoxel segmentation method for point clouds to achieve fast supervoxel over-segmentation of large-scale point clouds while maintaining good boundaries and accuracy. Firstly, the supervoxel segmentation is formulated as a subset selection problem. With the idea of greedy strategy, the interval quick sorting and region growing methods is designed to realize the rapid merging of subsets in the iterative process. Secondly, the Manhattan distance metric and the seed point median simulation method are proposed to enhance the precision of iterative optimization and improve the efficiency of point cloud supervoxel segmentation. Experimental results show that, compared with state-of-the-art supervoxel segmentation method, the proposed method achieves the best results under UE, GCE, BR and other indicators, while reducing the running time of the algorithm by 10.6% to 28.3%.

Wave-PCT: Wavelet point cloud transformer for point cloud quality assessment

Point cloud representation for real-world objects has seen a surge in interest recently, finding widespread applications in augmented reality, virtual reality, and autonomous driving. However, the process of acquiring, compressing, transmitting, and processing point cloud data often introduces unwanted distortions, highlighting the need for effective no-reference quality assessment methods for distorted point clouds. In this paper, we propose a Wavelet Point Cloud Transformer (Wave-PCT) method to assess the quality of point clouds without reference data. We start by performing patch sampling on a source point cloud, generating an extensive set of patches from the point cloud. These point cloud patches and their 2D projection images are processed using wavelets to produce multiscale local spectral features; moreover, we also extract global features of these patches using Pointnet++. Finally, a self-attention module is performed to fuse these local and global features, resulting in a distortion-sensitive quality score metric. We evaluate the Wave-PCT method on several point cloud quality assessment datasets using various metrics, and the experimental results indicate that the proposed framework exhibits outstanding and reliable performance when compared to various point cloud quality assessment methods.

Introducing an indoor object classification dataset including sparse point clouds from mmWave radar

This document introduces the RadIOCD, which is a dataset that contains sparse point cloud representations of indoor objects, collected by subjects wearing a commercial off-the-shelf mmWave radar. In particular, RadIOCD includes the recordings of 10 volunteers moving towards 5 different objects (i.e., backpack, chair, desk, human, and wall), placed in 3 different environments. RadIOCD includes sparse 3D point cloud data, together with their doppler velocity and intensity provided by the mmWave radar. A total of 5,776 files are available, with each one having an approximate duration of 8s. The scope of RadIOCD is the availability of data for the recognition of objects solely recorded by the mmWave radar, to be used in applications were the vision-based classification is cumbersome though critical (e.g., in search and rescue operation where there is smoke inside a building). Furthermore, we showcase that this dataset after being segmented into 76,821 samples contains enough data to apply Machine Learning-based techniques, ensuring that they could generalize in different environments and “unseen“ subjects.

Modeling 3D Cardiac Contraction and Relaxation With Point Cloud Deformation Networks.

Global single-valued biomarkers, such as ejection fraction, are widely used in clinical practice to assess cardiac function. However, they only approximate the heart's true 3D deformation process, thus limiting diagnostic accuracy and the understanding of cardiac mechanics. Metrics based on 3D shape have been proposed to alleviate these shortcomings. In this work, we present the Point Cloud Deformation Network (PCD-Net) as a novel geometric deep learning approach for direct modeling of 3D cardiac mechanics of the biventricular anatomy between the extreme ends of the cardiac cycle. Its encoder-decoder architecture combines a low-dimensional latent space with recent advances in point cloud deep learning for effective multi-scale feature learning directly on flexible and memory-efficient point cloud representations of the cardiac anatomy. We first evaluate the PCD-Net's predictive capability for both cardiac contraction and relaxation on a large UK Biobank dataset of over 10,000 subjects and find average Chamfer distances between the predicted and ground truth anatomies below the pixel resolution of the underlying image acquisition. We then show the PCD-Net's ability to capture subpopulation-specific differences in 3D cardiac mechanics between normal and myocardial infarction (MI) subjects and visualize abnormal phenotypes between predicted normal 3D shapes and corresponding observed ones. Finally, we demonstrate that the PCD-Net's learned 3D deformation encodings outperform multiple clinical and machine learning benchmarks by 11% in terms of area under the receiver operating characteristic curve for the tasks of prevalent MI detection and incident MI prediction and by 7% in terms of Harrell's concordance index for MI survival analysis.

Wigner kernels: Body-ordered equivariant machine learning without a basis.

Machine-learning models based on a point-cloud representation of a physical object are ubiquitous in scientific applications and particularly well-suited to the atomic-scale description of molecules and materials. Among the many different approaches that have been pursued, the description of local atomic environments in terms of their discretized neighbor densities has been used widely and very successfully. We propose a novel density-based method, which involves computing "Wigner kernels." These are fully equivariant and body-ordered kernels that can be computed iteratively at a cost that is independent of the basis used to discretize the density and grows only linearly with the maximum body-order considered. Wigner kernels represent the infinite-width limit of feature-space models, whose dimensionality and computational cost instead scale exponentially with the increasing order of correlations. We present several examples of the accuracy of models based on Wigner kernels in chemical applications, for both scalar and tensorial targets, reaching an accuracy that is competitive with state-of-the-art deep-learning architectures. We discuss the broader relevance of these findings to equivariant geometric machine-learning.

A Comparative Literature Review of Machine Learning and Image Processing Techniques Used for Scaling and Grading of Wood Logs

This literature review assesses the efficacy of image-processing techniques and machine-learning models in computer vision for wood log grading and scaling. Four searches were conducted in four scientific databases, yielding a total of 1288 results, which were narrowed down to 33 relevant studies. The studies were categorized according to their goals, including log end grading, log side grading, individual log scaling, log pile scaling, and log segmentation. The studies were compared based on the input used, choice of model, model performance, and level of autonomy. This review found a preference for images over point cloud representations for logs and an increase in camera use over laser scanners. It identified three primary model types: classical image-processing algorithms, deep learning models, and other machine learning models. However, comparing performance across studies proved challenging due to varying goals and metrics. Deep learning models showed better performance in the log pile scaling and log segmentation goal categories. Cameras were found to have become more popular over time compared to laser scanners, possibly due to stereovision cameras taking over for laser scanners for sampling point cloud datasets. Classical image-processing algorithms were consistently used, deep learning models gained prominence in 2018, and other machine learning models were used in studies published between 2010 and 2018.

Impact of LiDAR point cloud compression on 3D object detection evaluated on the KITTI dataset

The rapid growth on the amount of generated 3D data, particularly in the form of Light Detection And Ranging (LiDAR) point clouds (PCs), poses very significant challenges in terms of data storage, transmission, and processing. Point cloud (PC) representation of 3D visual information has shown to be a very flexible format with many applications ranging from multimedia immersive communication to machine vision tasks in the robotics and autonomous driving domains. In this paper, we investigate the performance of four reference 3D object detection techniques, when the input PCs are compressed with varying levels of degradation. Compression is performed using two MPEG standard coders based on 2D projections and octree decomposition, as well as two coding methods based on Deep Learning (DL). For the DL coding methods, we used a Joint Photographic Experts Group (JPEG) reference PC coder, that we adapted to accept LiDAR PCs in both Cartesian and cylindrical coordinate systems. The detection performance of the four reference 3D object detection methods was evaluated using both pre-trained models and models specifically trained using degraded PCs reconstructed from compressed representations. It is shown that LiDAR PCs can be compressed down to 6 bits per point with no significant degradation on the object detection precision. Furthermore, employing specifically trained detection models improves the detection capabilities even at compression rates as low as 2 bits per point. These results show that LiDAR PCs can be coded to enable efficient storage and transmission, without significant object detection performance loss.

2D Representations of 3D Point Clouds Via the Stereographic Projection with Encryption Applications

Using the topological equivalence between the Riemann sphere S\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathbb {S}$$\\end{document} and the extended complex plane C¯=C∪{∞}\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\overline{\\mathbb {C}} = \\mathbb {C} \\cup \\{\\infty \\}$$\\end{document}, where C\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathbb {C}$$\\end{document} is the field of complex numbers, we establish 2D-bijective representations of 3D point clouds. Points of 3D point clouds are mapped into the Riemann sphere S\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathbb {S}$$\\end{document}, and a stereographic projection is implemented to map the points into the complex plane C\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathbb {C}$$\\end{document}. The way the 3D objects are mapped into S\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathbb {S}$$\\end{document} may be varied for various applications. To prove the accuracy and efficiency of the proposed 2D representation of 3D objects, we apply this correspondence to 3D point cloud encryption. We utilize chaotic permutations, chaotic circuits, and Latin cubes in addition to the stereographic projection representation to construct our scheme. The permutation steps using chaotic maps and Latin cubes are carried out on the object data points in both S\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathbb {S}$$\\end{document} and C¯\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\overline{\\mathbb {C}}$$\\end{document}, while the chaotic circuits are applied to 2D projections of the 3D objects. To the best of our knowledge, no earlier work employed stereographic projections for 3D object encryption. Experimental simulations of this method show high encryption strength and strong confusion and diffusion properties based on quantitative and statistical measures.

PSVMLP: Point and Shifted Voxel MLP for 3D deep learning

We propose a high-performance 3D feature extraction deep learning network based on point cloud and shifted voxel, named Point and Shifted Voxel MLP (PSVMLP). The main component of PSVMLP is simple Multi-Layer Perceptron (MLP) structure. PSVMLP achieves effective extraction of multi-scale features from 3D data. Specifically, we combine point cloud and voxel-based feature extraction methods. In voxel representation learning, we propose a wide-range geometric feature extraction method based on axial shifting operations and simple MLP structure. The axial shifting operations allow shifting voxels in the depth, height, and width directions, capturing more geometric information. In point cloud representation learning, we use simple MLP structure to extract local features, and we also extract global features by combining transformer structure. By combining point cloud and voxel feature extraction methods, we obtain rich feature representations from different scales, enhancing the model’s expressive power and generalization performance. Applying our designed model to basic geometric feature learning tasks, we achieve excellent results. Despite being built primarily on a simple MLP framework, our model demonstrates remarkable performance on both shape classification and shape part segmentation tasks. Our code is available at https://github.com/hitxraz/psvmlp.

Adaptive recognition of machining features in sheet metal parts based on a graph class-incremental learning strategy

The integration of computer-aided design (CAD), computer-aided process planning (CAPP), and computer-aided manufacturing (CAM) systems is significantly enhanced by employing deep learning-based automatic feature recognition (AFR) methods. These methods outperform traditional, rule-based approaches, particularly in handling the complexities of intersecting features. However, existing deep learning-based AFR methods face two major challenges. The initial challenge stems from the frequent utilization of voxelized or point-cloud representations of CAD models, resulting in the unfortunate loss of valuable geometric and topological information inherent in original Boundary representation (B-Rep) models. The second challenge involves the limitation of supervised deep learning methods in identifying machining features that are not present in the predefined dataset. This constraint renders them suboptimal for the continually evolving datasets of real industrial scenarios. To address the first challenge, this study introduces a graph-structured language, Multidimensional Attributed Face-Edge Graph (maFEG), crafted to encapsulate the intricate geometric and topological details of CAD models. Furthermore, a graph neural network, Sheet-metalNet, is proposed for the efficient learning and interpretation of maFEGs. To tackle the second challenge, a three-component incremental learning strategy is proposed: an initial phase of pre-training and fine-tuning, a prototype sampling-based replay, and a stage employing knowledge distillation for parameter regularization. The effectiveness of Sheet-metalNet and its complementary incremental learning strategy is evaluated using the open-source MFCAD++ dataset and the newly created SMCAD dataset. Experimental results show that Sheet-metalNet surpasses state-of-the-art AFR methods in machining feature recognition accuracy. Moreover, Sheet-metalNet demonstrates adaptability to dynamic dataset changes, maintaining high performance when encountering newly introduced features, thanks to its innovative incremental learning strategy.

Automatic tooth arrangement with joint features of point and mesh representations via diffusion probabilistic models

Tooth arrangement is a crucial step in orthodontics treatment, in which aligning teeth could improve overall well-being, enhance facial aesthetics, and boost self-confidence. To improve the efficiency of tooth arrangement and minimize errors associated with unreasonable designs by inexperienced practitioners, some deep learning-based tooth arrangement methods have been proposed. Currently, most existing approaches employ MLPs to model the nonlinear relationship between tooth features and transformation matrices to achieve tooth arrangement automatically. However, the limited datasets (which to our knowledge, have not been made public) collected from clinical practice constrain the applicability of existing methods, making them inadequate for addressing diverse malocclusion issues. To address this challenge, we propose a general tooth arrangement neural network based on the diffusion probabilistic model. Conditioned on the features extracted from the dental model, the diffusion probabilistic model can learn the distribution of teeth transformation matrices from malocclusion to normal occlusion by gradually denoising from a random variable, thus more adeptly managing real orthodontic data. To take full advantage of effective features, we exploit both mesh and point cloud representations by designing different encoding networks to extract the tooth (local) and jaw (global) features, respectively. In addition to traditional metrics ADD, PA-ADD, CSA, and MErot, we propose a new evaluation metric based on dental arch curves to judge whether the generated teeth meet the individual normal occlusion. Experimental results demonstrate that our proposed method achieves state-of-the-art tooth alignment results and satisfactory occlusal relationships between dental arches. We will publish the code and dataset.

Terrain point cloud inpainting via signal decomposition

The rapid development of 3D acquisition technology has made it possible to obtain point clouds of real-world terrains. However, due to limitations in sensor acquisition technology or specific requirements, point clouds often contain defects such as holes with missing data. Inpainting algorithms are widely used to patch these holes. However, existing traditional inpainting algorithms rely on precise hole boundaries, which limits their ability to handle cases where the boundaries are not well-defined. On the other hand, learning-based completion methods often prioritize reconstructing the entire point cloud instead of solely focusing on hole filling. Based on the fact that real-world terrain exhibits both global smoothness and rich local detail, we propose a novel representation for terrain point clouds. This representation can help to repair the holes without clear boundaries. Specifically, it decomposes terrains into low-frequency and high-frequency components, which are represented by B-spline surfaces and relative height maps respectively. In this way, the terrain point cloud inpainting problem is transformed into a B-spline surface fitting and 2D image inpainting problem. By solving the two problems, the highly complex and irregular holes on the terrain point clouds can be well-filled, which not only satisfies the global terrain undulation but also exhibits rich geometric details. The experimental results also demonstrate the effectiveness of our method.

X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-Modal Knowledge Transfer

The field of 4D point cloud understanding is rapidly developing with the goal of analyzing dynamic 3D point cloud sequences. However, it remains a challenging task due to the sparsity and lack of texture in point clouds. Moreover, the irregularity of point cloud poses a difficulty in aligning temporal information within video sequences. To address these issues, we propose a novel cross-modal knowledge transfer framework, called X4D-SceneFormer. This framework enhances 4D-Scene understanding by transferring texture priors from RGB sequences using a Transformer architecture with temporal relationship mining. Specifically, the framework is designed with a dual-branch architecture, consisting of an 4D point cloud transformer and a Gradient-aware Image Transformer (GIT). The GIT combines visual texture and temporal correlation features to offer rich semantics and dynamics for better point cloud representation. During training, we employ multiple knowledge transfer techniques, including temporal consistency losses and masked self-attention, to strengthen the knowledge transfer between modalities. This leads to enhanced performance during inference using single-modal 4D point cloud inputs. Extensive experiments demonstrate the superior performance of our framework on various 4D point cloud video understanding tasks, including action recognition, action segmentation and semantic segmentation. The results achieve 1st places, i.e., 85.3% (+7.9%) accuracy and 47.3% (+5.0%) mIoU for 4D action segmentation and semantic segmentation, on the HOI4D challenge, outperforming previous state-of-the-art by a large margin. We release the code at https://github.com/jinglinglingling/X4D.

Voxel or Pillar: Exploring Efficient Point Cloud Representation for 3D Object Detection

Efficient representation of point clouds is fundamental for LiDAR-based 3D object detection. While recent grid-based detectors often encode point clouds into either voxels or pillars, the distinctions between these approaches remain underexplored. In this paper, we quantify the differences between the current encoding paradigms and highlight the limited vertical learning within. To tackle these limitations, we propose a hybrid detection framework named Voxel-Pillar Fusion (VPF), which synergistically combines the unique strengths of both voxels and pillars. To be concrete, we first develop a sparse voxel-pillar encoder that encodes point clouds into voxel and pillar features through 3D and 2D sparse convolutions respectively, and then introduce the Sparse Fusion Layer (SFL), facilitating bidirectional interaction between sparse voxel and pillar features. Our computationally efficient, fully sparse method can be seamlessly integrated into both dense and sparse detectors. Leveraging this powerful yet straightforward representation, VPF delivers competitive performance, achieving real-time inference speeds on the nuScenes and Waymo Open Dataset.