3D Objects Research Articles

Material Attribute Estimation as Part of Telecommunication Augmented Reality, Virtual Reality, and Mixed Reality System: Systematic Review

The integration of material attribute estimation (MAE) within augmented reality, virtual reality, and mixed reality telecommunication systems stands as a pivotal domain, evolving rapidly with the advent of the Tactile Internet. This unifying implementation process has the potential for improvements in the realism and interactivity of immersive environments. The interaction between MAE and the haptic Internet could lead to significant advances in haptic feedback systems, enabling more accurate and responsive user experiences. This systematic review is focused on the intersection of MAE and the Tactile Internet, aiming to find an implementation path between these technologies. Motivated by the potential of the haptic Internet to advance telecommunications, we explore its potential to advance the analysis of material attributes within AR, VR, and MR applications. Through an extensive analysis of current research approaches, including machine learning methods, we explore the possibilities of integrating the TI into MAE. By exploiting haptic and visual properties stored in the materials of 3D objects and using them directly during rendering in remote access scenarios, we propose a conceptual framework that combines data capture, visual representation, processing, and communication in virtual environments.

Beyond local patches: Preserving global–local interactions by enhancing self-attention via 3D point cloud tokenization

Transformer-based architectures have recently shown impressive performance on various point cloud understanding tasks such as 3D object shape classification and semantic segmentation. Particularly, this can be attributed to their self-attention mechanism, which has the ability to capture long-range dependencies. However, current methods have constrained it to operate in local patches due to its quadratic memory constraints. This hinders their generalization ability and scaling capacity due to the loss of non-locality in early layers. To tackle this issue, we propose a window-based transformer architecture that captures long-range dependencies while aggregating information in the local patches. We do this by interacting each window with a set of global point cloud tokens — a representative subset of the entire scene — and augmenting the local geometry through a 3D Histogram of Oriented Gradients (HOG) descriptor. Through a series of experiments on segmentation and classification tasks, we show that our model exceeds the state-of-the-art on S3DIS semantic segmentation (+1.67% mIoU), ShapeNetPart part segmentation (+1.03% instance mIoU) and performs competitively on ScanObjectNN 3D object classification.11The code and trained models shall be made publicly available.

CoFormerNet: A Transformer-Based Fusion Approach for Enhanced Vehicle-Infrastructure Cooperative Perception.

Vehicle-infrastructure cooperative perception is becoming increasingly crucial for autonomous driving systems and involves leveraging infrastructure's broader spatial perspective and computational resources. This paper introduces CoFormerNet, which is a novel framework for improving cooperative perception. CoFormerNet employs a consistent structure for both vehicle and infrastructure branches, integrating the temporal aggregation module and spatial-modulated cross-attention to fuse intermediate features at two distinct stages. This design effectively handles communication delays and spatial misalignment. Experimental results using the DAIR-V2X and V2XSet datasets demonstrated that CoFormerNet significantly outperformed the existing methods, achieving state-of-the-art performance in 3D object detection.

Intracranial aneurysm detection based on 3D point cloud object detection method

Intracranial aneurysm detection based on 3D point cloud object detection method

LiDAR Point Cloud Augmentation for Adverse Conditions Using Conditional Generative Model

The perception systems of autonomous vehicles face significant challenges under adverse conditions, with issues such as obscured objects and false detections due to environmental noise. Traditional approaches, which typically focus on noise removal, often fall short in such scenarios. Addressing the lack of diverse adverse weather data in existing automotive datasets, we propose a novel data augmentation method that integrates realistically simulated adverse weather effects into clear condition datasets. This method not only addresses the scarcity of data but also effectively bridges domain gaps between different driving environments. Our approach centers on a conditional generative model that uses segmentation maps as a guiding mechanism to ensure the authentic generation of adverse effects, which greatly enhances the robustness of perception and object detection systems in autonomous vehicles, operating under varied and challenging conditions. Besides the capability of accurately and naturally recreating over 90% of the adverse effects, we demonstrate that this model significantly improves the performance and accuracy of deep learning algorithms for autonomous driving, particularly in adverse weather scenarios. In the experiments employing our augmented approach, we achieved a 2.46% raise in the 3D average precision, a marked enhancement in detection accuracy and system reliability, substantiating the model’s efficacy with quantifiable improvements in 3D object detection compared to models without augmentation. This work not only serves as an enhancement of autonomous vehicle perception systems under adverse conditions but also marked an advancement in deep learning models in adverse condition research.

Enhancing 3D object detection through multi-modal fusion for cooperative perception

Fueled by substantial advancements in deep learning, the domain of autonomous driving is swiftly advancing towards more robust and effective intelligent systems. One of the critical challenges in this field is achieving accurate 3D object detection, which is often hindered by data sparsity and occlusion. To address these issues, we propose a method centered around a multi-modal fusion strategy that leverages vehicle-road cooperation to enhance perception capabilities. Our approach integrates label information from roadside perception point clouds to harmonize and enrich the representation of image and LiDAR data. This comprehensive integration significantly improves detection accuracy by providing a fuller understanding of the surrounding environment. Rigorous evaluations of our proposed method on two benchmark datasets, KITTI and Waymo Open, demonstrate its superior performance, with our model achieving 87.52% 3D Average Precision (3D AP) and 93.71% Bird’s Eye View Average Precision (BEV AP) on the KITTI val set. These results highlight the effectiveness of our method in detecting sparse and distant objects, contributing to the development of safer and more efficient autonomous driving solutions.

Properties of molecular clumps and cores in colliding magnetized flows

ABSTRACT We simulate the formation of molecular clouds in colliding flows of warm neutral medium with the adaptive mesh refinement code flash in eight simulations with varying initial magnetic field strength, between 0.01–5 μG. We include a chemical network to treat heating and cooling and to follow the formation of molecular gas. The initial magnetic field strength influences the fragmentation of the forming cloud because it prohibits motions perpendicular to the field direction and hence impacts the formation of large-scale filamentary structures. Molecular clump and core formation occurs anyhow. We identify 3D clumps and 3D cores, which are defined as connected, CO-rich regions. Additionally, 3D cores are heavily shielded. While we do not claim those 3D objects to be directly comparable to observations, this enables us to analyse their full virial state. With increasing field strength, we find more fragments with a smaller average mass; yet the dynamics of the forming clumps and cores only weakly depends on the initial magnetic field strength. The molecular clumps are mostly unbound, probably transient objects, which are weakly confined by ram pressure or thermal pressure, indicating that they are swept up by the turbulent flow. They experience significant fluctuations in the mass flux through their surface, such that the Eulerian reference frame shows a dominant time-dependent term due to their indistinct nature. We define the cores to encompass highly shielded molecular gas. Most cores are in gravitational-kinetic equipartition and are well described by the common virial parameter $\alpha _\mathrm{vir}$, while some undergo minor dispersion by kinetic surface effects.

A Digital Preservation for the Indonesian First Terminus

Abstract The presence of modernization in Indonesia is supported by several factors, one of which is the railway network. Unfortunately, despite its importance in Indonesian history, the first terminus in Indonesia – namely, Semarang Station – has almost been forgotten. Conservation efforts are needed to describe the architectural evolution of Semarang Station, which can provide a lens to see the adaptation efforts in the development process and the global impact of the establishment of a railway network in the colony. Moreover, digital reconstruction as architectural preservation is necessary to convey the station’s historical value. Given the limited access and scarcity of architectural drawings for Semarang Station, textual sources were used to validate the information from the visual sources. The digitalization of visual data produced 2D objects that were used as the basis for making 3D models; these 2D objects were identified using textual data to see the relationship between their architectural transformation and the events related to the building. The findings of the architectural morphology analysis were then used as data to produce the 3D models, which were the main target of the digital reconstruction. The result of this process is a reconstruction of the morphology of the Semarang Station that demonstrates its historical value.

Automatic Removal of Non-Architectural Elements in 3D Models of Historic Buildings with Language Embedded Radiance Fields

Neural radiance fields have emerged as a dominant paradigm for creating complex 3D environments incorporating synthetic novel views. However, 3D object removal applications utilizing neural radiance fields have lagged behind in effectiveness, particularly when open set queries are necessary for determining the relevant objects. One such application area is in architectural heritage preservation, where the automatic removal of non-architectural objects from 3D environments is necessary for many downstream tasks. Furthermore, when modeling occupied buildings, it is crucial for modeling techniques to be privacy preserving by default; this also motivates the removal of non-architectural elements. In this paper, we propose a pipeline for the automatic creation of cleaned, architectural structure only point clouds utilizing a language embedded radiance field (LERF) with a specific application toward generating suitable point clouds for the structural integrity assessment of occupied buildings. We then validated the efficacy of our approach on the rooms of the historic Sion hospital, a national historic monument in Valais, Switzerland. By using our automatic removal pipeline on the point clouds of rooms filled with furniture, we decreased the average earth mover’s distance (EMD) to the ground truth point clouds of the physically emptied rooms by 31 percent. The success of our research points the way toward new paradigms in architectural modeling and cultural preservation.

Joint stereo 3D object detection and implicit surface reconstruction

We present a new learning-based framework S-3D-RCNN that can recover accurate object orientation in SO(3) and simultaneously predict implicit rigid shapes from stereo RGB images. For orientation estimation, in contrast to previous studies that map local appearance to observation angles, we propose a progressive approach by extracting meaningful Intermediate Geometrical Representations (IGRs). This approach features a deep model that transforms perceived intensities from one or two views to object part coordinates to achieve direct egocentric object orientation estimation in the camera coordinate system. To further achieve finer description inside 3D bounding boxes, we investigate the implicit shape estimation problem from stereo images. We model visible object surfaces by designing a point-based representation, augmenting IGRs to explicitly address the unseen surface hallucination problem. Extensive experiments validate the effectiveness of the proposed IGRs, and S-3D-RCNN achieves superior 3D scene understanding performance. We also designed new metrics on the KITTI benchmark for our evaluation of implicit shape estimation.

Impact of LiDAR point cloud compression on 3D object detection evaluated on the KITTI dataset

The rapid growth on the amount of generated 3D data, particularly in the form of Light Detection And Ranging (LiDAR) point clouds (PCs), poses very significant challenges in terms of data storage, transmission, and processing. Point cloud (PC) representation of 3D visual information has shown to be a very flexible format with many applications ranging from multimedia immersive communication to machine vision tasks in the robotics and autonomous driving domains. In this paper, we investigate the performance of four reference 3D object detection techniques, when the input PCs are compressed with varying levels of degradation. Compression is performed using two MPEG standard coders based on 2D projections and octree decomposition, as well as two coding methods based on Deep Learning (DL). For the DL coding methods, we used a Joint Photographic Experts Group (JPEG) reference PC coder, that we adapted to accept LiDAR PCs in both Cartesian and cylindrical coordinate systems. The detection performance of the four reference 3D object detection methods was evaluated using both pre-trained models and models specifically trained using degraded PCs reconstructed from compressed representations. It is shown that LiDAR PCs can be compressed down to 6 bits per point with no significant degradation on the object detection precision. Furthermore, employing specifically trained detection models improves the detection capabilities even at compression rates as low as 2 bits per point. These results show that LiDAR PCs can be coded to enable efficient storage and transmission, without significant object detection performance loss.

Bridging Formal Shape Models and Deep Learning: A Novel Fusion for Understanding 3D Objects

This article describes a novel fusion of a generative formal model for three-dimensional (3D) shapes with deep learning (DL) methods to understand the geometric structure of 3D objects and the relationships between their components, given a collection of unorganized point cloud measurements. Formal 3D shape models are implemented as shape grammar programs written in Procedural Shape Modeling Language (PSML). Users write PSML programs to describe complex objects, and DL networks estimate the configured free parameters of the program to generate 3D shapes. Users write PSML programs to enforce fundamental rules that define an object class and encode object attributes, including shapes, components, size, position, etc., into a parametric representation of objects. This fusion of the generative model with DL offers artificial intelligence (AI) models an opportunity to better understand the geometric organization of objects in terms of their components and their relationships to other objects. This approach allows human-in-the-loop control over DL estimates by specifying lists of candidate objects, the shape variations that each object can exhibit, and the level of detail or, equivalently, dimension of the latent representation of the shape. The results demonstrate the advantages of the proposed method over competing approaches.

DIFFERENCE IN SURFACE FITTING WITH STANDARD AND MATRIX TOPOLOGY

This paper presents the impact of the surface topology of the scanned 3D object on parametric fitting. Whether it is a simple NURBS (Non-uniform rational B-spline) or a more complex hierarchical spline version, it is important to apply the fitting procedure. Here we describe the differences between fitting a surface with a given topology as a result of a 3D scanning system and a matrix topology of the surface, where the original surface is replaced by the result of a preset number of sections of the original geometry. We use the matrix and the free-form distribution. The former is more stable with respect to the distribution of the control point, the latter is numerically more suitable. In the future, we plan to adopt the free-form distribution to utilize the advantages of both distributions.

FF3D: A Rapid and Accurate 3D Fruit Detector for Robotic Harvesting.

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments.

Surface Erosion Equations for Degradation Analysis of Several Common Three-Dimensional Shapes of Plastic Materials

Equations were derived for the biodegradative erosion of the surfaces of arbitrarily-sized circular and square objects, i.e., the decrease in volume caused by the catalytic activity of enzymes secreted by microorganisms attached to the object surfaces. Surface erosion is the primary mode of biodegradation for polyhydroxyalkanoate (PHA) objects resting on the ocean floor. Although the derivations were motivated by a need to assess the time-varying biodegradation and ultimate disintegration of PHA tubes and straws in the benthic environment, generality was maintained during the mathematical development such that the resulting equations are also applicable to other circular and square objects, including cylindrical rings, discs, and solid rods, and square plates, cubes, and prisms. Moreover, the equations are applicable to abiotic degradation via physical erosion, not just to biotic degradation caused by microbes. Surface erosion is expressed in terms of the ratio of the instantaneous mass to the initial mass, which is generally nonlinear with respect to time. The value of this ratio and the relative dimensions of the object establish a surface erosion function and its evolution over the lifetime of the object. Hence, the rate of degradation is not constant in the theory, but rather is prescribed by the irregular mass loss and the original geometry of the 3D object.

Rapid calculation of computer-generated holograms for line-drawn 3D objects with varying thicknesses

Accelerating the calculation of computer-generated holograms (CGHs) is an important requirement for three-dimensional (3D) electro-holography display systems. Among the several sophisticated algorithms for CGH acceleration, the CG-line method for projecting 3D wire-frame objects is highly promising due to its high acceleration capability. However, conventional algorithms cannot account for line thickness, and only lines with a constant thickness corresponding to the pixel pitch of the spatial light modulator can be drawn. This restricts the expressiveness and visibility of 3D images, particularly for high-resolution holograms. Therefore, we propose a method for widening the line thickness of the CG-line method. We successfully generated a 3D object comprising bold lines of different thicknesses without sacrificing speed.

AGAR - Attention Graph-RNN for Adaptative Motion Prediction of Point Clouds of Deformable Objects

This article focuses on motion prediction for point cloud sequences in the challenging case of deformable 3D objects, such as human body motion. First, we investigate the challenges caused by deformable shapes and complex motions present in this type of representation, with the ultimate goal of understanding the technical limitations of state-of-the-art models. From this understanding, we propose an improved architecture for point cloud prediction of deformable 3D objects. Specifically, to handle deformable shapes, we propose a graph-based approach that learns and exploits the spatial structure of point clouds to extract more representative features. Then, we propose a module able to combine the learned features in a adaptative manner according to the point cloud movements. The proposed adaptative module controls the composition of local and global motions for each point, enabling the network to model complex motions in deformable 3D objects more effectively. We tested the proposed method on the following datasets: MNIST moving digits, the Mixamo human bodies motions [ 15 ], JPEG [ 5 ] and CWIPC-SXR [ 32 ] real-world dynamic bodies. Simulation results demonstrate that our method outperforms the current baseline methods given its improved ability to model complex movements as well as preserve point cloud shape. Furthermore, we demonstrate the generalizability of the proposed framework for dynamic feature learning by testing the framework for action recognition on the MSRAction3D dataset [ 19 ] and achieving results on par with state-of-the-art methods.

Depth-resolved speckle correlation imaging using the axial memory effect.

Speckle correlation imaging (SCI) recovers the angular size of a 2D object hidden behind a scatterer, but it fails to recover its depth. Here, we extend SCI to obtain object depth by capturing speckled images at two detector distances and exploiting the scaling properties of the axial memory effect. This noninvasive method does not require access to the object side of the scatterer, e.g. for pre-calibration or addition of a guide star, and will be useful in any scenario where SCI is applicable, such as biological imaging or seeing through fog.

A RGB-D feature fusion network for occluded object 6D pose estimation

A RGB-D feature fusion network for occluded object 6D pose estimation

Efficient flexible voxel-based two-stage network for 3D object detection in autonomous driving

3D object detection from the LiDAR point cloud plays an important role in autonomous driving. It is difficult to balance inference speed and detection accuracy when performing 3D point cloud object detection due to the large size of point cloud data and its unstructured storage, which makes it difficult to represent its features. To address the challenge, we propose a two-stage point cloud object detector, AFV-RCNN. In stage-1, the attention flexible voxel feature encoding layer is introduced, which utilizes flexible voxels to enhance feature encoding speed and focuses on foreground points to be detected through voxel attention. In stage-2, the multi-level and grid-based multi-scale RoI (Region of Interest) feature fusion module is designed. It directly extracts complete 3D structures from 3D region proposals and focuses on both local and global features through multi-scale partitioning. In the training stage, GHM-C Loss is applied to address the challenges associated with imbalanced target categories and the imbalance between difficult and easy samples in the classification task. We evaluate the model on the public KITTI Dataset and Waymo Open Dataset. The mAP in KITTI for 3D detection is 73.41% and inference on a single GPU reaches 30.0 fps. Compared with other state-of-the-art methods, AFV-RCNN achieves both the inference speed of a single-stage detector and the detection accuracy of a two-stage detector. It ensures higher detection accuracy while efficiently processing the point cloud.