3D Objects Research Articles

Efficient and Accurate Semi-Automatic Neuron Tracing with Extended Reality.

Neuron tracing, alternately referred to as neuron reconstruction, is the procedure for extracting the digital representation of the three-dimensional neuronal morphology from stacks of microscopic images. Achieving accurate neuron tracing is critical for profiling the neuroanatomical structure at single-cell level and analyzing the neuronal circuits and projections at whole-brain scale. However, the process often demands substantial human involvement and represents a nontrivial task. Conventional solutions towards neuron tracing often contend with challenges such as non-intuitive user interactions, suboptimal data generation throughput, and ambiguous visualization. In this paper, we introduce a novel method that leverages the power of extended reality (XR) for intuitive and progressive semi-automatic neuron tracing in real time. In our method, we have defined a set of interactors for controllable and efficient interactions for neuron tracing in an immersive environment. We have also developed a GPU-accelerated automatic tracing algorithm that can generate updated neuron reconstruction in real time. In addition, we have built a visualizer for fast and improved visual experience, particularly when working with both volumetric images and 3D objects. Our method has been successfully implemented with one virtual reality (VR) headset and one augmented reality (AR) headset with satisfying results achieved. We also conducted two user studies and proved the effectiveness of the interactors and the efficiency of our method in comparison with other approaches for neuron tracing.

HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes.

We present HOIMotion - a novel approach for human motion forecasting during human-object interactions that integrates information about past body poses and egocentric 3D object bounding boxes. Human motion forecasting is important in many augmented reality applications but most existing methods have only used past body poses to predict future motion. HOIMotion first uses an encoder-residual graph convolutional network (GCN) and multi-layer perceptrons to extract features from body poses and egocentric 3D object bounding boxes, respectively. Our method then fuses pose and object features into a novel pose-object graph and uses a residual-decoder GCN to forecast future body motion. We extensively evaluate our method on the Aria digital twin (ADT) and MoGaze datasets and show that HOIMotion consistently outperforms state-of-the-art methods by a large margin of up to 8.7% on ADT and 7.2% on MoGaze in terms of mean per joint position error. Complementing these evaluations, we report a human study (N=20) that shows that the improvements achieved by our method result in forecasted poses being perceived as both more precise and more realistic than those of existing methods. Taken together, these results reveal the significant information content available in egocentric 3D object bounding boxes for human motion forecasting and the effectiveness of our method in exploiting this information.

Voxel Interface Control in Multimaterial Extrusion 3D Printing.

Interfaces are crucial in natural and engineered systems, dictating essential biological, ecological, and technological properties that augment performance, functionality, and user experience. Yet, achieving precise interfacial control poses significant challenges in both conventional and additive manufacturing, where scalability constraints impede the controlled deposition of quasi-2D layers within 3D objects. This paper introduces Voxel-Interface 3D Printing (VI3DP), which enables comprehensive control over extruded voxel interfaces irrespective of the printhead diameter that conventionally dictates feature size. Various optical, mechanical, and electrical functionalizations, attaining interface thicknesses up to three orders of magnitude smaller than the voxel size are reported. Notable applications include encoding data in soft matter through fluorescent interfaces, creating tight fits and movable mechanisms through non-adhesive interfaces, fabricating bio-inspired composites with tailored failure modes, and developing a single filament capacitive touch sensor. VI3DP opens new avenues for enhanced functionality and efficiency across multiple fields, including biomedical technology, electronics, optics, andnanotechnology.

MonoCAPE: Monocular 3D object detection with coordinate-aware position embeddings

3D monocular detection remains to be a focal point of research, particularly due to its capacity to deliver available precision under conditions of low cost and simplified configurations, making it especially valuable in fields like autonomous driving. Current 3D object detection methods often overlook the spatial information missing from images, which is critical to spatial perception, and optimize bounding box attributes separately, failing to meet the requirements of autonomous driving. We introduce MonoCAPE, a novel 3D detection framework addressing these issues by encoding spatial information and co-optimizing attributes through a Coordinate-Aware Position Encoding (CAPE) Generator and a Task Co-optimization Strategy (TCS). The CAPE Generator produces sparse positional embeddings, enabling spatial awareness with low computational cost, while the TCS utilizes Gaussian modeling to prevent suboptimal outputs. In this way, our framework comprehensively takes into account what existing approaches ignore. Extensive experiments on the KITTI dataset demonstrate MonoCAPE significantly improves AP3D and APBEV metrics compared to existing advanced methods.

Research on convolutional neural network to realize high-quality dynamic holographic display

There are still some challenges in the computation speed and reproduction quality of holographic display. In this paper, a method is proposed to realize a high-quality 3D holographic display by using a complex value convolutional neural network. Firstly, the layer-based method is used to cut and layer the object. The amplitude and phase of each layer of the object are calculated by a complex value network at the same time. Then, the output of the complex value type of the network is converted into a phase-only hologram, and each layer of the hologram is superimposed to reproduce the complete 3D object. In addition, a novel color convolutional neural network is used to generate R, G and B three-color holograms simultaneously, reducing the time required for three-channel recalculation of holograms. The proposed method only takes 16 ms to generate a hologram from single-layer object, the average peak signal-to-noise ratio of the reconstructed image is 28 dB, and the structural similarity exceeds 0.99, which realizes a high-quality dynamic holographic display. The experimental results confirm the feasibility of the proposed method and provide a new method for real-time holographic display of 3D object.

Semi-Supervised Online Continual Learning for 3D Object Detection in Mobile Robotics

Continual learning addresses the challenge of acquiring and retaining knowledge over time across multiple tasks and environments. Previous research primarily focuses on offline settings where models learn through increasing tasks from samples paired with ground truth annotations. In this work, we focus on an unsolved, challenging, yet practical scenario, specifically, the semi-supervised online continual learning in autonomous driving and mobile robotics. In our settings, models are tasked with learning new distributions from streaming unlabeled samples and performing 3D object detection as soon as the LiDAR point cloud arrives. Additionally, we conducted experiments on both the KITTI dataset, our newly built IUSL dataset and Canadian Adverse Driving Conditions (CADC) Dataset. The results indicate that our method achieves a balance between rapid adaptation and knowledge retention, showcasing its effectiveness in the dynamic and complex environment of autonomous driving and mobile robotics. The developed ROS packages and IUSL dataset will be publicly available at: https://github.com/npu-ius-lab/OCL3D.

Processing Parameter Setting Procedure for a Commercial Bowden Tube FDM Printer

Additive manufacturing (AM), especially fused deposition modeling (FDM), has experienced great development and diffusion during recent years. However, it still faces some limitations, such as poor dimensional accuracy or surface defects, the improvement of which motivates the elaboration of the present work. Contrary to an approach based on the optimization of parameters to obtain a single invariant value, the main objective of this study is the design of a procedure that anyone can follow to generate a printing profile for their specific FDM printer, environment, and imposed constraints through the adjustment of some selected parameters in the popular slicing software UltiMaker Cura. The resulting procedure consists of four ad hoc designed specimens and their analysis algorithms, all connected by a general workflow that ensures the correct execution of the procedure. Its applicability and effectiveness have been proved in a case study where a printing profile was developed for the real manufacturing project of a custom 3D object in polylactic acid (PLA), obtaining an improvement of 50% in tolerances and proving that the proposed parameter setting procedure represents a reduction in the setting time and material consumption versus conventional trial and error methodologies.

Point cloud segmentation method based on an image mask and its application verification

Abstract Accurately perceiving three-dimensional (3D) environments or objects is crucial for the advancement of artificial intelligence (AI) interaction technologies. Currently, various types of sensors are employed to obtain point cloud data for 3D object detection or segmentation tasks. While this multi-sensor approach provides more precise 3D data than monocular or stereo cameras, it is also more expensive. The advent of RGB-D cameras, which provide both RGB images and depth information, addresses this issue.
In this study, we propose a point cloud segmentation method based on image masks. By using an RGB-D camera to capture color and depth images, we generate image masks through object recognition and segmentation. Given the mapping relationship between RGB image pixels and point clouds, these image masks can be further used to extract the point cloud data of the target objects. The experimental results revealed that the average accuracy of target segmentation was 84.78%, which was close to that of PointNet++. Compared with three traditional segmentation algorithms, the accuracy was improved by nearly 23.97%. The running time of our algorithm is reduced by 95.76% compared to the PointNet++ algorithm, which has the longest running time; and by 15.65% compared to the LCCP algorithm, which has the shortest running time among traditional methods. Compared with PointNet++, the segmentation accuracy was improved. This method addressed the issues of low robustness and excessive reliance on manual feature extraction in traditional point cloud segmentation methods, providing valuable support and reference for the accurate segmentation of 3D point clouds.

A larger augmented-reality field of view improves interaction with nearby holographic objects.

Augmented-reality (AR) applications have shown potential for assisting and modulating gait in health-related fields, like AR cueing of foot-placement locations in people with Parkinson's disease. However, the size of the AR field of view (AR-FOV), which is smaller than one's own FOV, might affect interaction with nearby floor-based holographic objects. The study's primary objective was to evaluate the effect of AR-FOV size on the required head orientations for viewing and interacting with real-world and holographic floor-based objects during standstill and walking conditions. Secondary, we evaluated the effect of AR-FOV size on gait speed when interacting with real-world and holographic objects. Sixteen healthy middle-aged adults participated in two experiments wearing HoloLens 1 and 2 AR headsets that differ in AR-FOV size. To confirm participants' perceived differences in AR-FOV size, we examined the head orientations required for viewing nearby and far objects from a standstill position (Experiment 1). In Experiment 2, we examined the effect of AR-FOV size on head orientations and gait speeds for negotiating 2D and 3D objects during walking. Less downward head orientation was required for looking at nearby holographic objects with HoloLens 2 than with HoloLens 1, as expected given differences in perceived AR-FOV size (Experiment 1). In Experiment 2, a greater downward head orientation was observed for interacting with holographic objects compared to real-world objects, but again less so for HoloLens 2 than HoloLens 1 along the line of progression. Participants walked slightly but significantly slower when interacting with holographic objects compared to real-world objects, without any differences between the HoloLenses. To conclude, the increased size of the AR-FOV did not affect gait speed, but resulted in more real-world-like head orientations for seeing and picking up task-relevant information when interacting with floor-based holographic objects, improving the potential efficacy of AR cueing applications.

Multi-feature enhancement based on sparse networks for single-stage 3D object detection

In the field of autonomous driving, the accuracy and real-time requirements for 3D object detection technology continue to improve, which is directly related to the commercialization process and market popularity of autonomous vehicles. Despite the efficiency of pillar-based coding for onboard systems, it falls short in terms of accuracy and the reduction of incorrect positives. In this paper, we will examine how to solve the problem of high incorrect rate and low accuracy of existing methods. Firstly, a MAP coding module is introduced to optimize previous point cloud feature coding modules, allowing for the efficient extraction of fine-grained features from point cloud data. Then, we introduce an innovative sparse dual attention (SDA) to efficiently filter out irrelevant details in feature extraction, thereby improving the pertinence and efficiency of information extraction. Finally, to address the potential loss of information from single local feature extraction, a local and global fusion module (CTGC) is introduced. Our method has proactively demonstrated its efficiency and accuracy through rigorous experimentation across diverse datasets. Analysis of the results leads to the conclusion that our solutions provide accurate and robust detection results. Code will be available at https://github.com/lcy199905/MyOpenPCDet.git.

Three-Dimensional Outdoor Object Detection in Quadrupedal Robots for Surveillance Navigations

Quadrupedal robots are confronted with the intricate challenge of navigating dynamic environments fraught with diverse and unpredictable scenarios. Effectively identifying and responding to obstacles is paramount for ensuring safe and reliable navigation. This paper introduces a pioneering method for 3D object detection, termed viewpoint feature histograms, which leverages the established paradigm of 2D detection in projection. By translating 2D bounding boxes into 3D object proposals, this approach not only enables the reuse of existing 2D detectors but also significantly increases the performance with less computation required, allowing for real-time detection. Our method is versatile, targeting both bird’s eye view objects (e.g., cars) and frontal view objects (e.g., pedestrians), accommodating various types of 2D object detectors. We showcase the efficacy of our approach through the integration of YOLO3D, utilizing LiDAR point clouds on the KITTI dataset, to achieve real-time efficiency aligned with the demands of autonomous vehicle navigation. Our model selection process, tailored to the specific needs of quadrupedal robots, emphasizes considerations such as model complexity, inference speed, and customization flexibility, achieving an accuracy of up to 99.93%. This research represents a significant advancement in enabling quadrupedal robots to navigate complex and dynamic environments with heightened precision and safety.

AMVFNet: Attentive Multi-View Fusion Network for 3D Object Detection

Pillar-based method is significant in the field of LiDAR-based 3D object detection which could directly make use of efficient 2D backbone and save computational resources during reference. Existing methods usually sequentially project the original point clouds into the cylindrical view or the bird-eye view for feature extraction. However, the former suffers from obscured problems and the scales of instances vary greatly with distance, while the latter leads to considerable confusion problems due to the loss of semantic information caused by the sparsity of the projected point cloud. In this paper, we present a novel and efficient two-stage point-pillar hybrid architecture named Attentive Multi-View Fusion Network (AMVFNet), in which we abstract features from all cylindrical view, bird-eye view, and raw point clouds. Rather than designing more complex modules to solve the problems inherent in the single-view approach, our multi-view fusion architecture effectively combines the strengths of multiple perspectives to improve performance at a more fundamental level. Besides, to compensate for quantization distortion caused by projection operations, we propose attentive feature enhancement layers to further improve the capability of contextual information capturing. Extensive experiments on the KITTI detection benchmark illustrate that our proposed AMVFNet achieves competitive performance compared with other SOTA 3D object detectors.

Ultrafast diffraction algorithms for light-in-flight holography

Digital light-in-flight (LIF) holography is an ultrafast imaging technique capable of single-shot simultaneous 3D and femtosecond time resolution acquisitions of light pulse propagation. However, the numerical diffraction algorithms used to model light on femtosecond timescales are currently limited in scope, accuracy, and efficiency. We derive an analytical model capable of modeling LIF hologram formation for various optical setup configurations, able to model 3D objects and precisely account for the limited temporal coherence of the signal. We design an efficient algorithmic implementation and validate the system in numerical simulations and with an experimental LIF holographic recording setup. We report ultrafast numerical diffraction over 10,000 times faster than the reference technique, with higher accuracy and capable of modeling 3D samples, thereby broadening its application domain.

Filling the Holes on 3D Heritage Object Surface Based on Automatic Segmentation Algorithm

ABSTRACTReconstructing and processing 3D objects are activities in the research field of computer graphics, image processing and computer vision. The 3D objects are processed based on geometric modelling (a branch of applied mathematics and computational geometry) or machine learning algorithms based on image processing. The computation of geometrical objects includes processing the curves and surfaces, subdivision, simplification, meshing, holes filling or reconstructing the 3D surface's objects on both point cloud data and triangular mesh. While the machine learning methods are developed using deep learning models. With the support of 3D laser scan devices and LiDAR techniques, the obtained dataset is close to the original shape of the real objects. Besides, photography and its application based on modern techniques in recent years help us collect data and process the 3D models more precisely. This article proposes a new method for filling holes on the 3D object's surface based on automatic segmentation. Instead of filling the hole directly as the existing methods, we now subdivide the hole before filling it. The hole is first determined and segmented automatically based on the computation of its local curvature. It is then filled on each part of the hole to match its local curvature shape. The method can work on both 3D point cloud surfaces and triangular mesh surfaces. Compared to the state‐of‐the‐art (SOTA) methods, our proposed method obtained higher accuracy of the reconstructed 3D objects.

Rethinking the Non-Maximum Suppression Step in 3D Object Detection from a Bird’s-Eye View

In camera-based bird’s-eye view (BEV) 3D object detection, non-maximum suppression (NMS) plays a crucial role. However, traditional NMS methods become ineffective in BEV scenarios where the predicted bounding boxes of small object instances often have no overlapping areas. To address this issue, this paper proposes a BEV intersection over union (IoU) computation method based on relative position and absolute spatial information, referred to as B-IoU. Additionally, a BEV circular search method, called B-Grouping, is introduced to handle prediction boxes of varying scales. Utilizing these two methods, a novel NMS strategy called BEV-NMS is developed to handle the complex prediction boxes in BEV perspectives. This BEV-NMS strategy is implemented in several existing algorithms. Based on the results from the nuScenes validation set, there was an average increase of 7.9% in mAP when compared to the strategy without NMS. The NDS also showed an average increase of 7.9% under the same comparison. Furthermore, compared to the Scale-NMS strategy, the mAP increased by an average of 3.4%, and the NDS saw an average improvement of 3.1%.

Precision-induced localized molten liquid metal stamps for damage-free transfer printing of ultrathin membranes and 3D objects

Transfer printing, a crucial technique for heterogeneous integration, has gained attention for enabling unconventional layouts and high-performance electronic systems. Elastomer stamps are typically used for transfer printing, where localized heating for elastomer stamp can effectively control the transfer process. A key challenge is the potential damage to ultrathin membranes from the contact force of elastic stamps, especially with fragile inorganic nanomembranes. Herein, we present a precision-induced localized molten technique that employs either laser-induced transient heating or hotplate-induced directional heating to precisely melt solid gallium (Ga). By leveraging the fluidity of localized molten Ga, which provides gentle contact force and exceptional conformal adaptability, this technique avoids damage to fragile thin films and improves operational reliability compared to fully liquefied Ga stamps. Furthermore, the phase transition of Ga provides a reversible adhesion with high adhesion switchability. Once solidified, the Ga stamp hardens and securely adheres to the micro/nano-membrane during the pick-up process. The solidified stamp also exhibits the capability to maneuver arbitrarily shaped objects by generating a substantial grip force through the interlocking effects. Such a robust, damage-free, simply operable protocol illustrates its promising capabilities in transfer printing diverse ultrathin membranes and objects on complex surfaces for developing high-performance unconventional electronics.

Deep models for multi-view 3D object recognition: a review

This review paper focuses on the progress of deep learning-based methods for multi-view 3D object recognition. It covers the state-of-the-art techniques in this field, specifically those that utilize 3D multi-view data as input representation. The paper provides a comprehensive analysis of the pipeline for deep learning-based multi-view 3D object recognition, including the various techniques employed at each stage. It also presents the latest developments in CNN-based and transformer-based models for multi-view 3D object recognition. The review discusses existing models in detail, including the datasets, camera configurations, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance. Additionally, it examines various computer vision applications that use multi-view classification. Finally, it highlights future directions, factors impacting recognition performance, and trends for the development of multi-view 3D object recognition method.

3D Polymeric Lattice Microstructure-Based Microneedle Array for Transdermal Electrochemical Biosensing.

Microneedles (MNs) or microneedle arrays (MNAs) are critical components of minimally invasive devices comprised of a single or a series of micro-scale projections. MNs can bypass the outermost layer of the skin and painlessly access microcirculation of the epidermis and dermis layers, attracting great interest in the development of personalized healthcare monitoring and diagnostic devices. However, MN technology has not yet reached its full potential since current micro- and nanofabrication methods do not address the need of fabricating MNs with complex surfaces to facilitate the development of clinically adequate devices. This work presents a new approach that combines 3D printing technology based on two-photon polymerization with soft lithography for cost-effective and time-saving fabrication of complex MNAs. Specifically, this method relies on printing complex 3D objects efficiently replicated into polymeric substrates via soft lithography, resulting in a free-standing polymeric lattice (PL) membrane that can be transferred onto gold-coated MNs and used for electrochemical biosensing. This platform shows excellent electrochemical performance in detecting metabolite (glucose) and protein (insulin) biomarkers with a dynamic linear range sufficient for detecting biomarkers in healthy individuals and patients. The approach holds great potential for fabricating next-generationMNs, including their transfer into clinically adequate devices.

Open-Pose 3D zero-shot learning: Benchmark and challenges

With the explosive 3D data growth, the urgency of utilizing zero-shot learning to facilitate data labeling becomes evident. Recently, methods transferring language or language-image pre-training models like Contrastive Language-Image Pre-training (CLIP) to 3D vision have made significant progress in the 3D zero-shot classification task. These methods primarily focus on 3D object classification with an aligned pose; such a setting is, however, rather restrictive, which overlooks the recognition of 3D objects with open poses typically encountered in real-world scenarios, such as an overturned chair or a lying teddy bear. To this end, we propose a more realistic and challenging scenario named open-pose 3D zero-shot classification, focusing on the recognition of 3D objects regardless of their orientation. First, we revisit the current research on 3D zero-shot classification and propose two benchmark datasets specifically designed for the open-pose setting. We empirically validate many of the most popular methods in the proposed open-pose benchmark. Our investigations reveal that most current 3D zero-shot classification models suffer from poor performance, indicating a substantial exploration room towards the new direction. Furthermore, we study a concise pipeline with an iterative angle refinement mechanism that automatically optimizes one ideal angle to classify these open-pose 3D objects. In particular, to make validation more compelling and not just limited to existing CLIP-based methods, we also pioneer the exploration of knowledge transfer based on Diffusion models. While the proposed solutions can serve as a new benchmark for open-pose 3D zero-shot classification, we discuss the complexities and challenges of this scenario that remain for further research development. The code is available publicly at https://github.com/weiguangzhao/Diff-OP3D

Dense projection fusion for 3D object detection

Fusing information from LiDAR and cameras can effectively enhance the overall perceptivity of autonomous vehicles in various scenarios. Despite the relatively good results achieved by point-wise fusion and Bird’s-Eye-View (BEV) fusion, they still cannot fully leverage the image information and lack of effective depth information. For any fusion methods, the multi-modal features first need to be concatenated along the channel, and then the fused features are extracted using convolutional layers. However, this type of fusion methods is effective, but too coarse which causes that the fused features cannot pay more attention to the regions with important features and suffer from severe noise. To tackle these issues, we propose in this paper a Dense Projection Fusion (DPFusion) approach. It consists of two new modules: dense depth map guided BEV transform (DGBT) module and multi-modal feature adaptive fusion (MFAF) module. The DGBT module first quickly estimates the depth of each pixel and then projects all image features to the BEV space, making full use of the image information. The MFAF module computes the image weights and point cloud weights for each channel in each BEV grid and then adaptively weights and fuses the image BEV features with the point cloud BEV features. It is worth pointing out that the MFAF module makes the fused features pay more attention to background outline and object outline. Our proposed DPFusion demonstrates competitive results in 3D object detection, achieving a mean Average Precision (mAP) of 70.4 and a nuScenes detection score (NDS) of 72.3 on the nuScenes validation set.