3D Task Research Articles

Designing Spiking Neural Network-Based Reinforcement Learning for 3D Robotic Arm Applications

This study investigates a novel approach to robotic arm control through integrating spiking neural networks with the twin delayed deep deterministic policy gradient reinforcement learning algorithm. Specifically, it presents the first application of spiking neural networks-based twin delayed deep deterministic policy gradient in 3D robotic manipulation, demonstrating its extension from traditional 2D tasks to complex 3D target-reaching scenarios with improved energy efficiency and stability. Additionally, with the inertial measurement unit data the system successfully mimics human arm movements, achieving a success rate of 0.95 among 50 trials and enabling an intuitive and accurate human–robot interaction system. This pioneering attempt highlights the feasibility of combining the biologically inspired spiking neural networks with the reinforcement learning algorithm to address the real-time challenges in high-dimensional robotic environments and advance the field of human–robot interaction systems.

Point cloud graph for LiDAR segmentation

Point clouds have benefited autonomous driving applications. Thanks to advancements in CNN-based and Transformer-based architectures, projection and voxel representations for point clouds have become mainstream for 3D point cloud tasks. However, current feature backbones still face challenges in capturing local neighbor communication and contextual information due to the sparse and discrete point clouds. In this work, we explore an order reconstruction for discrete point clouds based on the LiDAR measurement principles and discuss the point graph representation. The point cloud graph provides geodesic information through directed edges. We integrate a graph branch based on GNN into a vanilla 3D CNN-based segmentation backbone while addressing the interaction between Euclidean and geodesic features. We examine the improvements in segmentation accuracy, semantic consistency, and receptive fields powered by the point cloud graph on 3D point cloud benchmarks. Our experimental results demonstrate that the proposed methods achieve the 3D CNNs-based state-of-the-art for LiDAR segmentation.

LCASAFormer: Cross-attention enhanced backbone network for 3D point cloud tasks

LCASAFormer: Cross-attention enhanced backbone network for 3D point cloud tasks

Touching the Ground: Evaluating the Effectiveness of Data Physicalizations for Spatial Data Analysis Tasks.

Inspired by recent advances in digital fabrication, artists and scientists have demonstrated that physical data encodings (i.e., data physicalizations) can increase engagement with data, foster collaboration, and in some cases, improve data legibility and analysis relative to digital alternatives. However, prior empirical studies have only investigated abstract data encoded in physical form (e.g., laser cut bar charts) and not continuously sampled spatial data fields relevant to climate and medical science (e.g., heights, temperatures, densities, and velocities sampled on a spatial grid). This paper presents the design and results of the first study to characterize human performance in 3D spatial data analysis tasks across analogous physical and digital visualizations. Participants analyzed continuous spatial elevation data with three visualization modalities: (1) 2D digital visualization; (2) perspective-tracked, stereoscopic "fishtank" virtual reality; and (3) 3D printed data physicalization. Their tasks included tracing paths downhill, looking up spatial locations and comparing their relative heights, and identifying and reporting the minimum and maximum heights within certain spatial regions. As hypothesized, in most cases, participants performed the tasks just as well or better in the physical modality (based on time and error metrics). Additional results include an analysis of open-ended feedback from participants and discussion of implications for further research on the value of data physicalization. All data and supplemental materials are available at https://osf.io/7xdq4/.

Using 3D immersive virtual reality interactive tasks for upper limb rehabilitation in children with cerebral palsy: A randomized controlled trial

ABSTRACT The aim of this study was to investigate the effects of the virtual reality device on the evolution of motivation, and motor, functional and kinematic parameters of the upper limb in children with cerebral palsy. Twenty children were randomly assigned in VR and control groups. VR group scored higher than the control group in the Movement Assessment Battery for Children – Second Edition (MABC-2; standardized motor skills test), exhibited an increased range of motion, and showed improved results in various movement parameters in the interaction with the 3D virtual space. All participants presented high motivation scores in the iVR sessions. This new Immertrack tool may improve the motor, kinematic parameters, and motivation in children with CP.

Understanding user experience, task performance, and task interdependence in symmetric and asymmetric VR collaborations

Asymmetric collaboration is an important topic for the research of multiuser collaborative systems. Previous works have shown that by providing different abilities, devices or content to different users, users can take advantage of the unique features of each side and collaborate effectively with each other. However, there is limited work comparing the differences between asymmetric and symmetric Virtual Reality (VR) collaboration systems. How task complexity may affect symmetric and asymmetric VR collaboration is also unclear. In this paper, we present a comparative study that investigated how user experiences and task performance vary in symmetric and asymmetric VR collaboration. In addition, we also explored how task interdependence correlates with user experience and task performance. Participants were asked to collaboratively perform 3D object selection and manipulation tasks in pairs. A within-subjects study was conducted, where participants used PC and PC, VR and VR, and PC and VR, respectively in three conditions. Our results revealed that the asymmetric collaboration using both PC and VR showed the best results in closeness of relationship, social presence and task performance; the PC symmetric collaborative system showed the worst user experiences and task performance. Both user experience and task performance showed a positive correlation with task interdependence. We discussed the effects of the collaborativ mode and device on the user experience and task performance, and the implications for future symmetric and asymmetric VR collaboration systems.Graphical abstract

Dynamic 3D Point Cloud Sequences as 2D Videos.

Dynamic 3D point cloud sequences serve as one of the most common and practical representation modalities of dynamic real-world environments. However, their unstructured nature in both spatial and temporal domains poses significant challenges to effective and efficient processing. Existing deep point cloud sequence modeling approaches imitate the mature 2D video learning mechanisms by developing complex spatio-temporal point neighbor grouping and feature aggregation schemes, often resulting in methods lacking effectiveness, efficiency, and expressive power. In this paper, we propose a novel generic representation called Structured Point Cloud Videos (SPCVs). Intuitively, by leveraging the fact that 3D geometric shapes are essentially 2D manifolds, SPCV re-organizes a point cloud sequence as a 2D video with spatial smoothness and temporal consistency, where the pixel values correspond to the 3D coordinates of points. The structured nature of our SPCV representation allows for the seamless adaptation of well-established 2D image/video techniques, enabling efficient and effective processing and analysis of 3D point cloud sequences. To achieve such re-organization, we design a self-supervised learning pipeline that is geometrically regularized and driven by self-reconstructive and deformation field learning objectives. Additionally, we construct SPCV-based frameworks for both low-level and high-level 3D point cloud sequence processing and analysis tasks, including action recognition, temporal interpolation, and compression. Extensive experiments demonstrate the versatility and superiority of the proposed SPCV, which has the potential to offer new possibilities for deep learning on unstructured 3D point cloud sequences.

Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation.

Visual Question Answering on 3D Point Cloud (VQA-3D) is an emerging yet challenging field that aims at answering various types of textual questions given an entire point cloud scene. To tackle this problem, we propose the CLEVR3D, a large-scale VQA-3D dataset consisting of 171K questions from 8,771 3D scenes. Specifically, we develop a question engine leveraging 3D scene graph structures to generate diverse reasoning questions, covering the questions of objects' attributes (i.e., size, color, and material) and their spatial relationships. Through such a manner, we initially generated 44K questions from 1,333 real-world scenes. Moreover, a more challenging setup is proposed to remove the confounding bias and adjust the context from a common-sense layout. Such a setup requires the network to achieve comprehensive visual understanding when the 3D scene is different from the general co-occurrence context (e.g., chairs always exist with tables). To this end, we further introduce the compositional scene manipulation strategy and generate 127K questions from 7,438 augmented 3D scenes, which can improve VQA-3D models for real-world comprehension. Built upon the proposed dataset, we baseline several VQA-3D models, where experimental results verify that the CLEVR3D can significantly boost other 3D scene understanding tasks.

Carbon Based Polymeric Nanocomposite Hydrogel Bioink: A Review.

Carbon-based polymeric nanocomposite hydrogels (NCHs) represent a groundbreaking advancement in biomedical materials by integrating nanoparticles such as graphene, carbon nanotubes (CNTs), carbon dots (CDs), and activated charcoal (AC) into polymeric matrices. These nanocomposites significantly enhance the mechanical strength, electrical conductivity, and bioactivity of hydrogels, making them highly effective for drug delivery, tissue engineering (TE), bioinks for 3D Bioprinting, and wound healing applications. Graphene improves the mechanical and electrical properties of hydrogels, facilitating advanced tissue scaffolding and drug delivery systems. CNTs, with their exceptional mechanical strength and conductivity, enhance rheological properties, facilitating their use as bioinks in supporting complex 3D bioprinting tasks for neural, bone, and cardiac tissues by mimicking the natural structure of tissues. CDs offer fluorescence capabilities for theranostic applications, integrating imaging and therapeutic functions. AC enhances mechanical strength, biocompatibility, and antibacterial effectiveness, making it suitable for wound healing and electroactive scaffolds. Despite these promising features, challenges remain, such as optimizing nanoparticle concentrations, ensuring biocompatibility, achieving uniform dispersion, scaling up production, and integrating multiple functionalities. Addressing these challenges through continued research and development is crucial for advancing the clinical and industrial applications of these innovative hydrogels.

Towards 3D Reconstruction of Multi-Shaped Tunnels Utilizing Mobile Laser Scanning Data

Using digital twin models of tunnels has become critical to their efficient maintenance and management. A high-precision 3D tunnel model is the prerequisite for a successful digital twin model of tunnel applications. However, constructing high-precision 3D tunnel models with high-quality textures and structural integrity based on mobile laser scanning data remains a challenge, particularly for tunnels of different shapes. This study addresses this problem by developing a novel method for the 3D reconstruction of multi-shaped tunnels based on mobile laser scanning data. This method does not require any predefined mathematical models or projection parameters to convert point clouds into 2D intensity images that conform to the geometric features of tunnel linings. This method also improves the accuracy of 3D tunnel mesh models by applying an adaptive threshold approach that reduces the number of pseudo-surfaces generated during the Poisson surface reconstruction of tunnels. This method was experimentally verified by conducting 3D reconstruction tasks involving tunnel point clouds of four different shapes. The superiority of this method was further confirmed through qualitative and quantitative comparisons with related approaches. By automatically and efficiently constructing a high-precision 3D tunnel model, the proposed method offers an important model foundation for digital twin engineering and a valuable reference for future tunnel model construction projects.

Enhancing Solar Convection Analysis With Multi‐Core Processors and GPUs

ABSTRACTIn the realm of astrophysical numerical calculations, the demand for enhanced computing power is imperative. The time‐consuming nature of calculations, particularly in the domain of solar convection, poses a significant challenge for Astrophysicists seeking to analyze new data efficiently. Because they let different kinds of data be worked on separately, parallel algorithms are a good way to speed up this kind of work. A lot of this study is about how to use both multi‐core computers and GPUs to do math work about solar energy at the same time. Cutting down on the time it takes to work with data is the main goal. This way, new data can be looked at more quickly and without having to practice for a long time. It works well when you do things in parallel, especially when you use GPUs for 3D tasks, which speeds up the work a lot. This is proof of how important it is to adjust the parallelization methods based on the size of the numbers. But for 2D math, computers with more than one core work better. The results not only fix bugs in models of solar convection, but they also show that speed changes a little based on the gear and how it is processed.

Fast‐MedNeXt: Accelerating the MedNeXt Architecture to Improve Brain Tumour Segmentation Efficiency

ABSTRACTWith the rapid development of medical imaging technology, 3D image segmentation technology has gradually become a mainstream method, especially in brain tumour detection and diagnosis showing its unique advantages. The technique makes full use of 3D spatial information to locate and analyze tumours more accurately, thus playing an important role in improving diagnostic accuracy, optimising treatment planning and promoting research. However, it also suffers from significant computational expenditure and delayed processing pace. In this paper, we propose an innovative optimisation scheme to address this problem. We thoroughly investigate the MedNeXt network and propose Fast‐MedNeXt, which aims to increase the processing speed while maintaining accuracy. First, we introduce the partial convolution (PConv) technique, which replaces the deep convolutional layers in the network. This improvement effectively reduces computation and memory requirements while maintaining efficient feature extraction. Second, based on PConv, we propose PConv‐Down and PConv‐Up modules, which are applied to the up‐sampling and down‐sampling modules to further optimise the network structure and improve efficiency. To confirm the efficacy of the approach, we carried out a sequence of tests in the multimodal brain tumour segmentation challenge 2021 (BraTS2021). By comparing with the MedNeXt series network, the Fast‐MedNeXt reduced the latency by 22.1%, 20.5%, 15.8%, and 11.4% respectively, while the average accuracy also increased by 0.475% and 0.2% respectively. These significant performance improvements demonstrate the effectiveness of Fast‐MedNeXt in 3D medical image segmentation tasks and provide a new and more efficient solution for the field.

Adversarial Unsupervised Domain Adaptation for 3D Semantic Segmentation with 2D Image Fusion of Dense Depth

AbstractUnsupervised domain adaptation (UDA) is increasingly used for 3D point cloud semantic segmentation tasks due to its ability to address the issue of missing labels for new domains. However, most existing unsupervised domain adaptation methods focus only on uni‐modal data and are rarely applied to multi‐modal data. Therefore, we propose a cross‐modal UDA on multi‐modal datasets that contain 3D point clouds and 2D images for 3D Semantic Segmentation. Specifically, we first propose a Dual discriminator‐based Domain Adaptation (Dd‐bDA) module to enhance the adaptability of different domains. Second, given that the robustness of depth information to domain shifts can provide more details for semantic segmentation, we further employ a Dense depth Feature Fusion (DdFF) module to extract image features with rich depth cues. We evaluate our model in four unsupervised domain adaptation scenarios, i.e., dataset‐to‐dataset (A2D2 → SemanticKITTI), Day‐to‐Night, country‐to‐country (USA → Singapore), and synthetic‐to‐real (VirtualKITTI → SemanticKITTI). In all settings, the experimental results achieve significant improvements and surpass state‐of‐the‐art models.

Gaze, Wall, and Racket: Combining Gaze and Hand-Controlled Plane for 3D Selection in Virtual Reality

Raypointing, the status-quo pointing technique for virtual reality, is challenging with many occluded and overlapping objects. In this work, we investigate how eye-tracking input can assist the gestural ray pointing in the disambiguation of targets in densely populated scenes. We explore the concept of Gaze + Plane, where the intersection between the user's gaze and a hand-controlled plane facilitates 3D position specification. In particular, two techniques are investigated: Gaze&Wall, which employs an indirect plane positioned in depth using a hand ray, and Gaze&Racket, featuring a hand-held and rotatable plane. In a first experiment, we reveal the speed-error trade-offs between Gaze + Plane techniques. In a second study, we compared the best techniques to newly designed gesture-only techniques, finding that Gaze&Wall is less error-prone and significantly faster. Our research has relevance for spatial interaction, specifically on advanced techniques for complex 3D tasks.

An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion from the MICCAI2022 challenge

Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from Computed Tomography (CT) or Magnetic Resonance (MR) imaging data are registered to the intraoperative laparoscopic images during this process. Regarding 3D–2D fusion, most algorithms use anatomical landmarks to guide registration, such as the liver’s inferior ridge, the falciform ligament, and the occluding contours. These are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and prone to error. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion challenge (P2ILF), held during the Medical Image Computing and Computer Assisted Intervention (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: (1) A 2D and 3D landmark segmentation task and (2) a 3D–2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated in the challenge, whose results were assessed for each task independently. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. The proposed methods were evaluated on 16 test images and 2 preoperative 3D models from 2 patients. In Task 1, the teams were able to segment most of the 2D landmarks, while the 3D landmarks showed to be more challenging to segment. In Task 2, only one team obtained acceptable qualitative and quantitative registration results. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.

Prediction of Attention Groups and Big Five Personality Traits from Gaze Features Collected from an Outlier Search Game.

This study explores the intersection of personality, attention and task performance in traditional 2D and immersive virtual reality (VR) environments. A visual search task was developed that required participants to find anomalous images embedded in normal background images in 3D space. Experiments were conducted with 30 subjects who performed the task in 2D and VR environments while their eye movements were tracked. Following an exploratory correlation analysis, we applied machine learning techniques to investigate the predictive power of gaze features on human data derived from different data collection methods. Our proposed methodology consists of a pipeline of steps for extracting fixation and saccade features from raw gaze data and training machine learning models to classify the Big Five personality traits and attention-related processing speed/accuracy levels computed from the Group Bourdon test. The models achieved above-chance predictive performance in both 2D and VR settings despite visually complex 3D stimuli. We also explored further relationships between task performance, personality traits and attention characteristics.

Bi-Interfusion: A bidirectional cross-fusion framework with semantic-guided transformers in LiDAR-camera fusion

Multi-sensor modal fusion has shown significant advantages in 3D object detection tasks. However, existing methods for fusing multi-modal features into the bird’s eye view (BEV) space often encounter challenges such as feature misalignment, underutilization of semantic information, and inaccurate depth estimation on the Z-axis, resulting in suboptimal performance. To address these issues, we propose Bi-Interfusion, a novel multi-modal fusion framework based on transformers. Bi-Interfusion incorporates a bidirectional fusion architecture, including components such as Pixel-wise Semantic Painting, Gaussian Depth Prior Distribution module, and Semantic Guidance Align module, to overcome the limitations of traditional fusion methods. Specifically, Bi-Interfusion employs a bidirectional cross-fusion strategy to merge image and LiDAR features, enabling the generation of multi-sensor BEV features. This approach leverages a refined Gaussian Depth Prior Distribution generated from LiDAR points, thereby improving the precision of view transformation. Additionally, we apply a pixel-wise semantic painting technique to embed image semantic information into LiDAR point clouds, facilitating a more comprehensive scene understanding. Furthermore, a transformer-based model is utilized to establish soft correspondences among multi-sensor BEV features, capturing positional dependencies and fully exploiting semantic information for alignment. Through experiments on nuScenes benchmark dataset, Bi-Interfusion demonstrates notable performance improvements, achieving a competitive performance of 72.6% mAP and 75.4% NDS in the 3D object detection task.

Exploring the categorical nature of colour perception: Insights from artificial networks

The electromagnetic spectrum of light from a rainbow is a continuous signal, yet we perceive it vividly in several distinct colour categories. The origins and underlying mechanisms of this phenomenon remain partly unexplained. We investigate categorical colour perception in artificial neural networks (ANNs) using the odd-one-out paradigm. In the first experiment, we compared unimodal vision networks (e.g., ImageNet object recognition) to multimodal vision-language models (e.g., CLIP text-image matching). Our results show that vision networks predict a significant portion of human data (approximately 80%), while vision-language models account for the remaining unexplained data, even in non-linguistic experiments. These findings suggest that categorical colour perception is a language-independent representation, though it is partly shaped by linguistic colour terms during its development. In the second experiment, we explored how the visual task influences the colour categories of an ANN by examining twenty-four Taskonomy networks. Our results indicate that human-like colour categories are task-dependent, predominantly emerging in semantic and 3D tasks, with a notable absence in low-level tasks. To explain this difference, we analysed kernel responses before the winner-takes-all stage, observing that networks with mismatching colour categories may still align in underlying continuous representations. Our findings quantify the dual influence of visual signals and linguistic factors in categorical colour perception and demonstrate the task-dependent nature of this phenomenon, suggesting that categorical colour perception emerges to facilitate certain visual tasks.

Enhancing MeshNet for 3D shape classification with focal and regularization losses

With the development of deep learning and computer vision, an increasing amount of research has focused on applying deep learning models to the recognition and classification of three-dimensional shapes. In classification tasks, differences in sample quantity, feature amount, model complexity, and other aspects among different categories of 3D model data cause significant variations in classification difficulty. However, simple cross-entropy loss is generally used as the loss function, but it is insufficient to address these differences. In this paper, we used MeshNet as the base model and introduced focal loss as a metric for the loss function. Additionally, to prevent deep learning models from developing a preference for specific categories, we incorporated regularization loss. The combined use of focal loss and regularization loss in optimizing the MeshNet model’s loss function resulted in a classification accuracy of up to 92.46%, representing a 0.20% improvement over the original model’s highest accuracy of 92.26%. Furthermore, the average accuracy over the final 50 epochs remained stable at a higher level of 92.01%, reflecting a 0.71% improvement compared to the original MeshNet model’s 91.30%. These results indicate that our method performs better in 3D shape classification task.

Point-Based Fusion for Multimodal 3D Detection in Autonomous Driving

In the broader field of mechanical technology, and specifically within the domain of self-driving vehicles, cameras and LIDAR are crucial sensor modalities that provide complementary information, offering significant potential for sensor fusion. However, directly merging multi-sensor data through point projection can lead to information loss due to quantization, and managing the differences between data formats from multiple sensors remains a challenge. To address these issues, we propose an new fusion method that leverages continuous convolution, point-pooling, and a learned MLP to achieve superior detection performance. Our approach integrates the segmentation mask with raw LIDAR points instead of using projected points, thereby avoiding quantization loss. We conduct neighbor searches on the points and retrieve corresponding semantic features from images to concatenate image and LIDAR data. Subsequently, we apply continuous convolution, point-pooling, and a learned MLP to obtain the fused output. The pooling and aggregation operations, as extensions of convolution, are specifically designed to handle the disparities in data formats. Our detection network is divided into two stages: in the first stage, preliminary proposals and segmentation features are generated; in the second stage, the fusion result with the segmentation mask is refined to produce the final prediction. Our method aims to achieve precise object detection in 3D environments by enhancing LIDAR point data with semantic features from images, allowing for the flexibility to alternate segmentation sub-algorithms as needed. Extensive experiments on the KITTI dataset demonstrate the effectiveness of our approach, which achieves high precision and robust performance in 3D object detection tasks.