3D Object Classification Research Articles

Open-Pose 3D zero-shot learning: Benchmark and challenges

With the explosive 3D data growth, the urgency of utilizing zero-shot learning to facilitate data labeling becomes evident. Recently, methods transferring language or language-image pre-training models like Contrastive Language-Image Pre-training (CLIP) to 3D vision have made significant progress in the 3D zero-shot classification task. These methods primarily focus on 3D object classification with an aligned pose; such a setting is, however, rather restrictive, which overlooks the recognition of 3D objects with open poses typically encountered in real-world scenarios, such as an overturned chair or a lying teddy bear. To this end, we propose a more realistic and challenging scenario named open-pose 3D zero-shot classification, focusing on the recognition of 3D objects regardless of their orientation. First, we revisit the current research on 3D zero-shot classification and propose two benchmark datasets specifically designed for the open-pose setting. We empirically validate many of the most popular methods in the proposed open-pose benchmark. Our investigations reveal that most current 3D zero-shot classification models suffer from poor performance, indicating a substantial exploration room towards the new direction. Furthermore, we study a concise pipeline with an iterative angle refinement mechanism that automatically optimizes one ideal angle to classify these open-pose 3D objects. In particular, to make validation more compelling and not just limited to existing CLIP-based methods, we also pioneer the exploration of knowledge transfer based on Diffusion models. While the proposed solutions can serve as a new benchmark for open-pose 3D zero-shot classification, we discuss the complexities and challenges of this scenario that remain for further research development. The code is available publicly at https://github.com/weiguangzhao/Diff-OP3D

Deep learning for 3D object recognition: A survey

With the growing availability of extensive 3D datasets and the rapid progress in computational power, deep learning (DL) has emerged as a highly promising approach for learning from 3D data, addressing critical tasks like object detection, segmentation, and recognition. Despite the unique challenges in processing geometry data with deep neural networks, recent advancements in DL for 3D object recognition have shown remarkable success, with various methods proposed to tackle different issues. This paper aims to stimulate future research by providing a comprehensive review of recent progress in DL techniques for 3D object recognition, which are systematically categorized based on their learning behavior. We discuss the advantages, limitations, and application of each approach, highlighting their performance in 3D object classification on benchmark datasets such as ModelNet, ScanObjectNN, and Sydney Urban Object. The survey offers insightful observations and inspires future research directions.

Recognition and Classification of 3D Objects of Different Details

Recognition and Classification of 3D Objects of Different Details

Beyond local patches: Preserving global–local interactions by enhancing self-attention via 3D point cloud tokenization

Transformer-based architectures have recently shown impressive performance on various point cloud understanding tasks such as 3D object shape classification and semantic segmentation. Particularly, this can be attributed to their self-attention mechanism, which has the ability to capture long-range dependencies. However, current methods have constrained it to operate in local patches due to its quadratic memory constraints. This hinders their generalization ability and scaling capacity due to the loss of non-locality in early layers. To tackle this issue, we propose a window-based transformer architecture that captures long-range dependencies while aggregating information in the local patches. We do this by interacting each window with a set of global point cloud tokens — a representative subset of the entire scene — and augmenting the local geometry through a 3D Histogram of Oriented Gradients (HOG) descriptor. Through a series of experiments on segmentation and classification tasks, we show that our model exceeds the state-of-the-art on S3DIS semantic segmentation (+1.67% mIoU), ShapeNetPart part segmentation (+1.03% instance mIoU) and performs competitively on ScanObjectNN 3D object classification.11The code and trained models shall be made publicly available.

ESTATE: A Large Dataset of Under-Represented Urban Objects for 3D Point Cloud Classification

Abstract. Cityscapes contain a variety of objects, each with a particular role in urban administration and development. With the rapid growth and implementation of 3D imaging technology, urban areas are increasingly surveyed with high-resolution point clouds. This technical advancement extensively improves our ability to capture and analyse urban environments and their small objects. Deep learning algorithms for point cloud data have shown considerable capacity in 3D object classification but still face problems with generally under-represented objects (such as light poles or chimneys). This paper introduces the ESTATE dataset (https://github.com/3DOM-FBK/ESTATE), which combines available datasets of various sensors, densities, regions, and object types. It includes 13 classes featuring intensity and/or colour attributes. Tests using ESTATE demonstrate that the dataset improves the classification performance of deep learning techniques and could be a game-changer to advance in the 3D classification of urban objects.

ShapeAXI: Shape Analysis Explainability and Interpretability.

ShapeAXI represents a cutting-edge framework for shape analysis that leverages a multi-view approach, capturing 3D objects from diverse viewpoints and subsequently analyzing them via 2D Convolutional Neural Networks (CNNs). We implement an automatic N-fold cross-validation process and aggregate the results across all folds. This ensures insightful explainability heat-maps for each class across every shape, enhancing interpretability and contributing to a more nuanced understanding of the underlying phenomena. We demonstrate the versatility of ShapeAXI through two targeted classification experiments. The first experiment categorizes condyles into healthy and degenerative states. The second, more intricate experiment, engages with shapes extracted from CBCT scans of cleft patients, efficiently classifying them into four severity classes. This innovative application not only aligns with existing medical research but also opens new avenues for specialized cleft patient analysis, holding considerable promise for both scientific exploration and clinical practice. The rich insights derived from ShapeAXI's explainability images reinforce existing knowledge and provide a platform for fresh discovery in the fields of condyle assessment and cleft patient severity classification. As a versatile and interpretative tool, ShapeAXI sets a new benchmark in 3D object interpretation and classification, and its groundbreaking approach hopes to make significant contributions to research and practical applications across various domains. ShapeAXI is available in our GitHub repository https://github.com/DCBIA-OrthoLab/ShapeAXI.

Improved MLP Point Cloud Processing with High-Dimensional Positional Encoding

Multi-Layer Perceptron (MLP) models are the bedrock of contemporary point cloud processing. However, their complex network architectures obscure the source of their strength. We first develop an “abstraction and refinement” (ABS-REF) view for the neural modeling of point clouds. This view elucidates that whereas the early models focused on the ABS stage, the more recent techniques devise sophisticated REF stages to attain performance advantage in point cloud processing. We then borrow the concept of “positional encoding” from transformer literature, and propose a High-dimensional Positional Encoding (HPE) module, which can be readily deployed to MLP based architectures. We leverage our module to develop a suite of HPENet, which are MLP networks that follow ABS-REF paradigm, albeit with a sophisticated HPE based REF stage. The developed technique is extensively evaluated for 3D object classification, object part segmentation, semantic segmentation and object detection. We establish new state-of-the-art results of 87.6 mAcc on ScanObjectNN for object classification, and 85.5 class mIoU on ShapeNetPart for object part segmentation, and 72.7 and 78.7 mIoU on Area-5 and 6-fold experiments with S3DIS for semantic segmentation. The source code for this work is available at https://github.com/zouyanmei/HPENet.

Unsupervised Point Cloud Representation Learning by Clustering and Neural Rendering

Data augmentation has contributed to the rapid advancement of unsupervised learning on 3D point clouds. However, we argue that data augmentation is not ideal, as it requires a careful application-dependent selection of the types of augmentations to be performed, thus potentially biasing the information learned by the network during self-training. Moreover, several unsupervised methods only focus on uni-modal information, thus potentially introducing challenges in the case of sparse and textureless point clouds. To address these issues, we propose an augmentation-free unsupervised approach for point clouds, named CluRender, to learn transferable point-level features by leveraging uni-modal information for soft clustering and cross-modal information for neural rendering. Soft clustering enables self-training through a pseudo-label prediction task, where the affiliation of points to their clusters is used as a proxy under the constraint that these pseudo-labels divide the point cloud into approximate equal partitions. This allows us to formulate a clustering loss to minimize the standard cross-entropy between pseudo and predicted labels. Neural rendering generates photorealistic renderings from various viewpoints to transfer photometric cues from 2D images to the features. The consistency between rendered and real images is then measured to form a fitting loss, combined with the cross-entropy loss to self-train networks. Experiments on downstream applications, including 3D object detection, semantic segmentation, classification, part segmentation, and few-shot learning, demonstrate the effectiveness of our framework in outperforming state-of-the-art techniques.

LATFormer: Locality-Aware Point-View Fusion Transformer for 3D shape recognition

Recently, 3D shape understanding has achieved significant progress due to the advances of deep learning models on various data formats like images, voxels, and point clouds. Among them, point clouds and multi-view images are two complementary modalities of 3D objects, and learning representations by fusing both of them has been proven to be fairly effective. While prior works typically focus on exploiting global features of the two modalities, herein we argue that more discriminative features can be derived by modeling “where to fuse”. To investigate this, we propose a novel Locality-Aware Point-View Fusion Transformer (LATFormer) for 3D shape retrieval and classification. The core component of LATFormer is a module named Locality-Aware Fusion (LAF) which integrates the local features of correlated regions across the two modalities based on the co-occurrence scores. We further propose to filter out scores with low values to obtain salient local co-occurring regions, which reduces redundancy for the fusion process. In our LATFormer, we utilize the LAF module to fuse the multi-scale features of the two modalities both bidirectionally and hierarchically to obtain more informative features. Comprehensive experiments on four popular 3D shape benchmarks covering 3D object retrieval and classification validate its effectiveness.

PU-CTG: A Point Cloud Upsampling Network Using Transformer Fusion and GRU Correction

Point clouds are widely used in remote sensing applications, e.g., 3D object classification, semantic segmentation, and building reconstruction. Generating dense and uniformly distributed point clouds from low-density ones is beneficial to 3D point cloud applications. The traditional methods mainly focus on the global shape of 3D point clouds, thus ignoring detailed representations. The enhancement of detailed features is conducive to generating dense and uniform point clouds. In this paper, we propose a point cloud upsampling network to improve the detail construction ability, named PU-CTG. The proposed method is implemented by a cross-transformer-fused module and a GRU-corrected module. The aim of the cross-transformer module is to enable the interaction and effective fusion between different scales of features so that the network can capture finer features. The purpose of the gated recurrent unit (GRU) is to reconstruct fine-grained features by rectifying the feedback error. The experimental results demonstrate the effectiveness of our method. Furthermore, the ModelNet40 dataset is upsampled by PU-CTG, and the classification experiment is applied to PointNet to verify the promotion ability of this network.

Analysis of the PointNet neural network architecture

Objective. Most researchers convert point cloud data into ordinary three-dimensional voxel grids or image collections, which makes the data unnecessarily voluminous and causes problems when processing them. The purpose of the study is to analyze the architecture of the PointNet neural network. Method. A unified approach has been applied to solving various 3D recognition problems, ranging from object classification, detail segmentation to semantic scene analysis. Result. A comparative analysis of the classification of 2d and 3d objects was carried out, the layers and functions through which classification occurs were studied in detail. A type of neural network is considered that directly uses point clouds, which takes into account the invariance of permutations of points in the input data. The network is determined to provide a unified architecture for applications ranging from object classification, part segmentation, and scene semantics. For semantic segmentation, the input data can be either a single object from the part area segmentation or a small part of the 3D scene. A neural network that is widely used for raster image editing, graphic design, and digital art is a deep point cloud architecture called PointNet. Conclusion. A new deep point cloud architecture, PointNet, is introduced. For object classification task, the input point cloud is directly selected from the shape or pre-segmented from the scene point cloud. To obtain a virtual model of the real world, neural network solutions are used, based on the assumption that there is an RGB point cloud obtained by an RGB-D camera from one or several angles.

Balanced Class-Incremental 3D Object Classification and Retrieval

Most existing 3D object classification and retrieval algorithms rely on one-off supervised learning on closed 3D object sets and tend to provide rigid convolutional neural networks with little scalability. Such limitations substantially restrict their potential to learn newly emerged 3D object classes continually in the real world. Aiming to go beyond these limitations, we innovatively propose two new and challenging tasks: class-incremental 3D object classification ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOC</i> ) and class-incremental 3D object retrieval ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CI-3DOR</i> ), the key to which is class-incremental 3D representation learning. It expects the network to update continually to learn new 3D class representations without forgetting the previously learned ones. To this end, we design a novel balanced distillation network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(BDNet)</i> that uses a dual supervision mechanism to balance between consolidating old knowledge (stability) and adapting to new 3D object classes (plasticity) carefully. On the one hand, we employ stability-based supervision to retain the stable and discriminative information of old classes that greatly benefit both classification and retrieval tasks. On the other hand, we use plasticity-based supervision to improve the network's generalization for learning new class 3D representations by transferring knowledge from a temporary teacher network to the current model. By properly handling the relationship between the two modules, we achieve a surprising performance improvement. Furthermore, considering there is no available dataset for evaluation, we build two 3D datasets, INOR-1 and INOR-2, to evaluate these two new tasks. Extensive experimental results demonstrate that our method can significantly outperform other state-of-the-art class-incremental learning methods. Even if we store 500-1000 fewer 3D objects than SOTA methods, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BDNet</i> still achieves comparable performance.

3D object detection and classification combined with point cloud depth information

3D object detection and classification combined with point cloud depth information

Point Cloud-Based 3D Object Classification With Non Local Attention and Lightweight Convolution Neural Networks

Point Cloud-Based 3D Object Classification With Non Local Attention and Lightweight Convolution Neural Networks

A system for finding identical geometries when building 3-D models

In today's fast-paced world of technology, where innovation plays a crucial role in our daily lives, the importance of search engines for identical models cannot be overstated. Every day, technological progress is becoming an integral part of our existence, penetrating various aspects of our lives. In this context, the search and identification of identical or similar 3D models become particularly relevant and become an integral part of innovative technologies. This article is an in-depth study in the field of creating and describing a mathematical model to search for similar 3D models. Developed using advanced comparison algorithms, this model has enormous potential and can be used in a variety of fields. It is important to emphasize that it is not limited to just one area of application, but finds its application in many areas of human activity. The application of such algorithms is important in various fields, including automated manufacturing, where accuracy and speed of component identification and classification are critical. They also find use in archaeology, where they help in the analysis and study of artifacts and structures. Architecture is also seeing great benefit from these innovations, allowing for more complex and detailed 3D models to be created for design and visualization. The new approach presented in the paper is based on an optimized matching algorithm that enables accurate recognition of geometric structures associated with 3D models without complex data preprocessing or image enhancement. This is a significant step forward in improving the efficiency and accuracy of identifying similar 3D models, even in low-resolution images. However, the application of this model is not limited to these areas. Its integration into real-time systems allows for the identification and classification of both individual 3D objects and complex 3D scenes created from image or video sequences. This is of great importance for the development of virtual and augmented reality technologies, where a high degree of detail and identification is necessary to create captivating and realistic visual effects. Thus, the presented mathematical model and the corresponding software open up enormous prospects for us in various fields. This is an important step forward in the development of technologies related to computer vision and image processing, as well as in the field of virtual and augmented reality, promoting progress and innovation

Vision-Based Object Localization and Classification for Electric Vehicle Driving Assistance

The continuous advances in intelligent systems and cutting-edge technology have greatly influenced the development of intelligent vehicles. Recently, integrating multiple sensors in cars has improved and spread the advanced drive-assistance systems (ADAS) solutions for achieving the goal of total autonomy. Despite current self-driving approaches and systems, autonomous driving is still an open research issue that must guarantee the safety and reliability of drivers. This work employs images from two cameras and Global Positioning System (GPS) data to propose a 3D vision-based object localization and classification method for assisting a car during driving. The experimental platform is a prototype of a two-sitter electric vehicle designed and assembled for navigating the campus under controlled mobility conditions. Simultaneously, color and depth images from the primary camera are combined to extract 2D features, which are reprojected into 3D space. Road detection and depth features isolate point clouds representing the objects to construct the occupancy map of the environment. A convolutional neural network was trained to classify typical urban objects in the color images. Experimental tests validate car and object pose in the occupancy map for different scenarios, reinforcing the car position visually estimated with GPS measurements.

Attentional Keypoint Detection on Point Clouds for 3D Object Part Segmentation

In the field of computer vision, segmenting a 3D object into its component parts is crucial to understanding its structure and characteristics. Much work has focused on 3D object part segmentation directly from point clouds, and significant progress has been made in this area. This paper proposes a novel 3D object part segmentation method that focuses on integrating three key modules: a keypoint-aware module, a feature extension module, and an attention-aware module. Our approach starts by detecting keypoints, which provide the global feature of the inner shape that serves as the basis for segmentation. Subsequently, we utilize the feature extension module to expand the dimensions, obtain the local representation of the obtained features, provide richer object representation, and improve segmentation accuracy. Furthermore, we introduce an attention-aware module that effectively combines the features of the global and local parts of objects to enhance the segmentation process. To validate the proposed model, we also conduct experiments on the point cloud classification task. The experimental results demonstrate the effectiveness of our method, thus outperforming several state-of-the-art methods in 3D object part segmentation and classification.

Self-supervised Multi-view Learning via Auto-encoding 3D Transformations

3D object representation learning is a fundamental challenge in computer vision to infer about the 3D world. Recent advances in deep learning have shown their efficiency in 3D object recognition, among which view-based methods have performed best so far. However, feature learning of multiple views in existing methods is mostly performed in a supervised fashion, which often requires a large amount of data labels with high costs. In contrast, self-supervised learning aims to learn multi-view feature representations without involving labeled data. To this end, we propose a novel self-supervised framework to learn Multi-View Transformation Equivariant Representations (MV-TER), exploring the equivariant transformations of a 3D object and its projected multiple views that we derive. Specifically, we perform a 3D transformation on a 3D object and obtain multiple views before and after the transformation via projection. Then, we train a representation encoding module to capture the intrinsic 3D object representation by decoding 3D transformation parameters from the fused feature representations of multiple views before and after the transformation. Experimental results demonstrate that the proposed MV-TER significantly outperforms the state-of-the-art view-based approaches in 3D object classification and retrieval tasks and show the generalization to real-world datasets. The code is available at https://github.com/gyshgx868/mvter .

Monitoring method for detecting and localizing overheat, smoke and fire faults in wind turbine nacelle

This study presents a monitoring method that utilizes 3D object classification to accurately detect mechanical and electrical components of a wind turbine by combining a geometric and statistic feature extractor (GSFE) with a multiview approach. The proposed monitoring method also detect outlier after executing object detection to localize overheat faults in these components with fused Optical or Infrared/LiDAR measurements. The proposed method hasthree key characteristics. First, the proposed outlier detection allocates two extremes of normal and faulty clusters by using 2D object classification/detection model or measuring thestandard deviation of temperature with sensor fusing measurements. Specifically, the outlier detection with sensor fusing measurements extracts the position coordinates andtemperature data to localize overheat faults, effectively detecting an overheat component. Second, the GSFE utilizes a group sampling approach to extract the local geometric feature information from neighboring point clouds, aggregating normal vectors and standard deviation. This method ensures the high accuracy of object classification. Third, a multi-view approach focuses on updating local geometric and statistic features through a graph convolution network, improving the accuracy and robustness of object classification. The proposed outlier detection is verified through overheat/fire field tests. The effectiveness of the proposed 3D object classification method is also validated by using a virtual wind turbine nacelle CAD dataset and a public CAD dataset named ModelNet40. Consequently, the proposed method is practical and effective for monitoring a fire and overheat component because it can accurately detect critical components with only a few virtual datasets because gathering bigdata for training a neural network is extremely difficult.

Recent Advances and Perspectives in Deep Learning Techniques for 3D Point Cloud Data Processing

In recent years, deep learning techniques for processing 3D point cloud data have seen significant advancements, given their unique ability to extract relevant features and handle unstructured data. These techniques find wide-ranging applications in fields like robotics, autonomous vehicles, and various other computer-vision applications. This paper reviews the recent literature on key tasks, including 3D object classification, tracking, pose estimation, segmentation, and point cloud completion. The review discusses the historical development of these methods, explores different model architectures, learning algorithms, and training datasets, and provides a comprehensive summary of the state-of-the-art in this domain. The paper presents a critical evaluation of the current limitations and challenges in the field, and identifies potential areas for future research. Furthermore, the emergence of transformative methodologies like PoinTr and SnowflakeNet is examined, highlighting their contributions and potential impact on the field. The potential cross-disciplinary applications of these techniques are also discussed, underscoring the broad scope and impact of these developments. This review fills a knowledge gap by offering a focused and comprehensive synthesis of recent research on deep learning techniques for 3D point cloud data processing, thereby serving as a useful resource for both novice and experienced researchers in the field.