3D Task Research Articles

TransConver: transformer and convolution parallel network for developing automatic brain tumor segmentation in MRI images.

Medical image segmentation plays a vital role in computer-aided diagnosis (CAD) systems. Both convolutional neural networks (CNNs) with strong local information extraction capacities and transformers with excellent global representation capacities have achieved remarkable performance in medical image segmentation. However, because of the semantic differences between local and global features, how to combine convolution and transformers effectively is an important challenge in medical image segmentation. In this paper, we proposed TransConver, a U-shaped segmentation network based on convolution and transformer for automatic and accurate brain tumor segmentation in MRI images. Unlike the recently proposed transformer and convolution based models, we proposed a parallel module named transformer-convolution inception (TC-inception), which extracts local and global information via convolution blocks and transformer blocks, respectively, and integrates them by a cross-attention fusion with global and local feature (CAFGL) mechanism. Meanwhile, the improved skip connection structure named skip connection with cross-attention fusion (SCCAF) mechanism can alleviate the semantic differences between encoder features and decoder features for better feature fusion. In addition, we designed 2D-TransConver and 3D-TransConver for 2D and 3D brain tumor segmentation tasks, respectively, and verified the performance and advantage of our model through brain tumor datasets. We trained our model on 335 cases from the training dataset of MICCAI BraTS2019 and evaluated the model's performance based on 66 cases from MICCAI BraTS2018 and 125 cases from MICCAI BraTS2019. Our TransConver achieved the best average Dice score of 83.72% and 86.32% on BraTS2019 and BraTS2018, respectively. We proposed a transformer and convolution parallel network named TransConver for brain tumor segmentation. The TC-Inception module effectively extracts global information while retaining local details. The experimental results demonstrated that good segmentation requires the model to extract local fine-grained details and global semantic information simultaneously, and our TransConver effectively improves the accuracy of brain tumor segmentation.

Multi-Level View Associative Convolution Network for View-Based 3D Model Retrieval

With the continuous improvement of image processing capabilities, a three-dimensional (3D) model that can contain rich information is becoming the fourth type of multimedia data (in addition to sound, image, and video). Moreover, since there is a wide range of applications of 3D models, how to quickly and effectively obtain the correct target model from the massive data has become a key issue. To date, 3D model retrieval approaches have been proposed, and in these approaches, view-based 3D model retrieval methods can achieve satisfactory performance. In the 3D model retrieval task, the latent relationship mining of all images in a 3D model, the adaptive fusion of different images, and the discriminative feature extraction are the main challenges, but in most existing solutions, these issues are separately performed and they are not explored in an end-to-end network architecture. To solve these issues, in this work, we propose a novel and effective multi-level view associative convolution network ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MLVACN</b> ) to realize view-based 3D model retrieval, where the relationship exploration of multiple-view images, the fusion of different images, and the feature discrimination learning are realized in a unified end-to-end framework. Specifically, we design the group association layer and the block association layer to study the latent relationships among different views from the view-level and the block-level, respectively. Moreover, the weight fusion layer is further designed to adaptively fuse different views in a 3D model. In addition, these three layers are embedded into the <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MLVACN</b> . Finally, the pairwise discrimination loss function is proposed to learn the discriminative features of the 3D model. Extensive experimental results on three 3D model retrieval datasets including ModelNet40, ModelNet10, and ShapeNetCore55 demonstrate that <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MLVACN</b> can outperform state-of-the-art methods in term of mAP. When the ModelNet40 dataset is used, the mAP of <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MLVACN</b> is improved by 13.25%, 7.75%, 3.95%, and 0.61% as compared to those of the MVCNN, GVCNN, PVNet, and MLVCNN methods, respectively.

RIConv++: Effective Rotation Invariant Convolutions for 3D Point Clouds Deep Learning

3D point clouds deep learning is a promising field of research that allows a neural network to learn features of point clouds directly, making it a robust tool for solving 3D scene understanding tasks. While recent works show that point cloud convolutions can be invariant to translation and point permutation, investigations of the rotation invariance property for point cloud convolution has been so far scarce. Some existing methods perform point cloud convolutions with rotation-invariant features, existing methods generally do not perform as well as translation-invariant only counterpart. In this work, we argue that a key reason is that compared to point coordinates, rotation-invariant features consumed by point cloud convolution are not as distinctive. To address this problem, we propose a simple yet effective convolution operator that enhances feature distinction by designing powerful rotation invariant features from the local regions. We consider the relationship between the point of interest and its neighbors as well as the internal relationship of the neighbors to largely improve the feature descriptiveness. Our network architecture can capture both local and global context by simply tuning the neighborhood size in each convolution layer. We conduct several experiments on synthetic and real-world point cloud classifications, part segmentation, and shape retrieval to evaluate our method, which achieves the state-of-the-art accuracy under challenging rotations.

Codesigning a ballistic body armour with an industry partner and local police officers: focused on improving ergonomic fit and comfort

ABSTRACT We aimed to identify fit and comfort issues associated with the case company’s body armour and suggest recommendations to mitigate the issues, guided by the integrated three-stage process model for codesign. Thirteen collaborators participated in the codesign practice, including eight company employees (ICs) and five local police officers (ECs). This study adopted one-on-one interviews, focus groups, 3D body scanning and occupational task assessment. In Stage 1, we identified the problems with the existing body armour and areas of improvement based on the inputs from the ICs and ECs. The design solutions for the defined needs were sought in Stage 2. In Stage 3, the final outcomes were validated and approved by the collaborators. The results from this study provided the case company practical guidance to improve the body armour design, and further offer a new means of involving diverse stakeholders’ voices and perspectives in the product design and development process.

Revisiting segmentation of lung tumors from CT images

Lung cancer is a leading cause of death throughout the world. Because the prompt diagnosis of tumors allows oncologists to discern their nature, type, and mode of treatment, tumor detection and segmentation from CT scan images is a crucial field of study. This paper investigates lung tumor segmentation via a two-dimensional Discrete Wavelet Transform (DWT) on the LOTUS dataset (31,247 training, and 4458 testing samples) and a Deeply Supervised MultiResUNet model. Coupling the DWT, which is used to achieve a more meticulous textural analysis while integrating information from neighboring CT slices, with the deep supervision of the model architecture results in an improved dice coefficient of 0.8472. A key characteristic of our approach is its avoidance of 3D kernels (despite being used for a 3D segmentation task), thereby making it quite lightweight.

Extract Descriptors for Point Cloud Registration by Graph Clustering Attention Network

Extracting geometric descriptors in 3D vision is the first step. It plays an important role in 3D registration, 3D reconstruction, and other applications. The success of many 3D tasks is closely related to whether the geometric descriptor has accurate characteristics. Today, the main methods are divided into manual production and neural network learning. The applicability of descriptors is limited to a low-level point, corner, edge, and fixed neighborhood features. For this, we use the class attention of the point cloud. In order to extract class attention, the graph clustering approach is utilized. It can collect points with similar structures and divide regions dynamically. While maintaining rotation invariance, features can enhance their fit to the original data. Point attention and edge attention are used to describe the structural characteristics of point clouds. We combine the three attentions indicated before to improve the features obtained by the PointNet decoder. This feature can dynamically reflect the structure of the point cloud, which includes both soft shape information and rich detail information. Finally, the 3D descriptors are extracted with the FoldingNet decoder. Our method is validated on both indoor and outdoor datasets. The accuracy of the final result is improved by two percentage points.

Time-space duality in 2D quantum gravity

An important task faced by all approaches of quantum gravity is to incorporate superpositions and quantify quantum uncertainties of spacetime causal relations. We address this task in 2D. By identifying a global Z 2 symmetry of 1 + 1D quantum gravity, we show that gravitational path integral configurations come in equal amplitude pairs with timelike and spacelike relations exchanged. As a consequence, any two points are equally probable to be timelike and spacelike separated in a Universe without boundary conditions. In the context of simplicial quantum gravity we identify a local symmetry of the action which shows that even with boundary conditions causal uncertainties are generically present. Depending on the boundary conditions, causal uncertainties can still be large and even maximal.

The relationship between visual impairments and activity of the neck/shoulder muscles among surgeons during simulated surgical tasks.

Minimally Invasive Surgery (MIS) has an impact on surgeons' musculoskeletal and visual systems. However, the relationship between visual symptoms and musculoskeletal problems is not well understood. This study used surface electromyography (sEMG) to examine changes in fatigue of the neck/shoulder muscles among surgeons with visual impairments when performing simulated surgical tasks in 2D and 3D viewing modes. Cross-sectional laboratory study. Changes in median frequency (MDF, an indicator of muscle fatigue) were examined in 17 experienced gynaecologists. Four simulated surgical laparoscopy tasks were performed in 2D and 3D viewing modes. The MDF of three neck/shoulder muscles (cervical erector spinae, [CES], upper trapezii [UT], and anterior deltoids [AD]) were examined bilaterally. Visual parameters (accommodation, convergence, and stereoacuity) were measured prior to commencement. There was a downward shift of MDF from simple to more complex tasks for the right and left CES and AD muscles but not the UT, which was consistent for surgeons with mild accommodation/convergence impairment and/or good stereoacuity. There were significant differences in the level of muscle fatigue of the neck/shoulder muscles according to the severity of visual impairment, muscle side, task and surgical performance level. The results show a relationship between the degree of visual impairments and muscle fatigue of the neck/shoulder muscles among MIS surgeons. These findings have important implications in understanding the concurrence of musculoskeletal problems and visual symptoms.

The number sense: a comparison between normal and adapted curriculum pupils in terms of subitization ability, the distance and magnitude effects

Number sense refers to the ability to easily understand, approximate and manipulate numerical quantities. In solving numerical problems, children use exact, respectively approximate calculation strategies. Unfortunately, some persons encounter difficulties in acquiring basic mathematical skills, and this condition represent a specific learning disability called developmental discalculia. The aim of the present study was to compare normal and adapted curriculum pupils in terms of subitization ability and efectiveness of number sense. The presence of magnitude and distance effects was tested in four distinct formats: 3D objects, 2D figures, verbal and visual Arabic numerical representation. The sample consisted of 111 normal and adapted curriculum pupils. Both number of errors and reaction times were registered. Adapted curriculum pupils registered more erroneous answers in the 2D, verbal, and arabic tasks and bigger reaction times in the 3D, tasks with Arabic or verbal numerals. Both normal and adapted curriculum pupils registered the best performances in the visual Arabic numerical representation task. Contrary to the expectations, the magnitude effect was not outlined in terms of reaction times, perhaps due to the calculation strategies which pupils adopted in different tasks.

MD Loss: Efficient Training of 3-D Seismic Fault Segmentation Network Under Sparse Labels by Weakening Anomaly Annotation

Data-driven fault detection has been regarded as a 3D image segmentation task. The models trained from synthetic data are difficult to generalize in some surveys. Recently, training 3D fault segmentation using sparse manual 2D slices is thought to yield promising results, but manual labeling has many false negative labels (abnormal annotations), which is detrimental to training and consequently to detection performance. Motivated to train 3D fault segmentation networks under sparse 2D labels while suppressing false negative labels, we analyze the training process gradient and propose the Mask Dice (MD) loss. Moreover, the fault is an edge feature, and current encoder-decoder architectures widely used for fault detection (e.g., U-shape network) are not conducive to edge representation. Consequently, Fault-Net is proposed, which is designed for the characteristics of faults, employs high-resolution propagation features, and embeds MultiScale Compression Fusion block to fuse multi-scale information, which allows the edge information to be fully preserved during propagation and fusion, thus enabling advanced performance via few computational resources. Experimental demonstrates that MD loss supports the inclusion of human experience in training and suppresses false negative labels therein, enabling baseline models to improve performance and generalize to more surveys. Fault-Net is capable to provide a more stable and reliable interpretation of faults, it uses extremely low computational resources and inference is significantly faster than other models. Our method indicates optimal performance in comparison with several mainstream methods.

Modeling and Calibration of a Galvanometer-Camera Imaging System

The galvanometer-camera imaging system (GCIS), which mainly consists of a galvanometer and a camera, can drastically enlarge the field of view without decreasing imaging resolution by quickly switching viewing directions. However, the lack of adequate imaging model has hindered GCIS from being used to 3D vision tasks with accuracy requirements, such as 3D reconstruction and 6D pose estimation. A nonphysical parameter imaging model is proposed, which quantitatively depicts the relation among the input parameters of the galvanometer, the imaging position of a 3D point on the camera image plane, and the equation of the corresponding incoming ray. An ingenious method is put forward to calibrate the proposed imaging model. The GCIS with calibrated imaging model has much enlarged applicability. As an example, the method for object pose estimation by using the calibrated GCIS is presented and verified. Experimental results demonstrate the effectiveness and high performance of the proposed whole set of methods.

LFT-Net: Local Feature Transformer Network for Point Clouds Analysis

6G network enables the rapid connection of autonomous vehicles, the generated internet of vehicles establishes a large-scale point cloud, which requires automatic point cloud analysis to build an intelligent transportation system in terms of the 3D object detection and segmentation. Recently, a great variety of deep convolution networks have been proposed for 3D data analysis, making significant progress in the application of deep learning in 3D computer vision. Inspired by the application of transformer network in 2D computer visual tasks, and in order to increase the expression ability of local fine-grained features, we propose an effective local feature transformer network to learn local feature information and correlations between point clouds. Our network is adaptive to the arrangement of set elements through transformer module, so it is suitable for the feature extraction of local point clouds. In addition, experimental results demonstrate that our LFT-network outperforms the state-of-the-art in 3D model classification tasks on ModelNet40 dataset and segmentation tasks on S3DIS dataset.

Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation.

Automatic segmentation methods are an important advancement in medical image analysis. Machine learning techniques, and deep neural networks in particular, are the state-of-the-art for most medical image segmentation tasks. Issues with class imbalance pose a significant challenge in medical datasets, with lesions often occupying a considerably smaller volume relative to the background. Loss functions used in the training of deep learning algorithms differ in their robustness to class imbalance, with direct consequences for model convergence. The most commonly used loss functions for segmentation are based on either the cross entropy loss, Dice loss or a combination of the two. We propose the Unified Focal loss, a new hierarchical framework that generalises Dice and cross entropy-based losses for handling class imbalance. We evaluate our proposed loss function on five publicly available, class imbalanced medical imaging datasets: CVC-ClinicDB, Digital Retinal Images for Vessel Extraction (DRIVE), Breast Ultrasound 2017 (BUS2017), Brain Tumour Segmentation 2020 (BraTS20) and Kidney Tumour Segmentation 2019 (KiTS19). We compare our loss function performance against six Dice or cross entropy-based loss functions, across 2D binary, 3D binary and 3D multiclass segmentation tasks, demonstrating that our proposed loss function is robust to class imbalance and consistently outperforms the other loss functions. Source code is available at: https://github.com/mlyg/unified-focal-loss.

EEG Peak Detection in Cognitive Conflict Processing Using Summit Navigator and Clustering-Based Ranking.

Correct detection of peaks in electroencephalogram (EEG) signals is of essence due to the significant correlation of those potentials with cognitive performance and disorders. This paper proposes a novel and non-parametric approach to detect prediction error negativity (PEN) in cognitive conflict processing. The PEN candidates are first located from the input signal via an adaptation of a recent effective method for local maxima extraction, processed in a multi-scale manner. The found candidates are then fused and ranked based on their shape and location-based features. False positives caused by candidates' magnitude are eliminated by rotating the sorted candidate list where the one with the second-best ranking score will be identified as PEN. The EEG data collected from a 3D object selection task have been used to verify the efficacy of the proposed approach. Compared with the state-of-the-art peak detection techniques, the proposed method shows an improvement of at least 2.67% in accuracy and 6.27% in sensitivity while requires only about 4 ms to process an epoch. The accuracy and computational efficiency of the proposed technique in the detection of PEN in cognitive conflict processing would lead to promising applications in performance improvement of brain-computer interfaces (BCIs).

Fast Quaternion Product Units for Learning Disentangled Representations in [Formula: see text

Real-world 3D structured data like point clouds and skeletons often can be represented as data in a 3D rotation group (denoted as [Formula: see text]). However, most existing neural networks are tailored for the data in the euclidean space, which makes the 3D rotation data not closed under their algebraic operations and leads to sub-optimal performance in 3D-related learning tasks. To resolve the issues caused by the above mismatching between data and model, we propose a novel non-real neuron model called quaternion product unit (QPU) to represent data on 3D rotation groups. The proposed QPU leverages quaternion algebra and the law of the 3D rotation group, representing 3D rotation data as quaternions and merging them via a weighted chain of Hamilton products. We demonstrate that the QPU mathematically maintains the [Formula: see text] structure of the 3D rotation data during the inference process and disentangles the 3D representations into "rotation-invariant" features and "rotation-equivariant" features, respectively. Moreover, we design a fast QPU to accelerate the computation of QPU. The fast QPU applies a tree-structured data indexing process, and accordingly, leverages the power of parallel computing, which reduces the computational complexity of QPU in a single thread from O(N) to O(logN). Taking the fast QPU as a basic module, we develop a series of quaternion neural networks (QNNs), including quaternion multi-layer perceptron (QMLP), quaternion message passing (QMP), and so on. In addition, we make the QNNs compatible with conventional real-valued neural networks and applicable for both skeletons and point clouds. Experiments on synthetic and real-world 3D tasks show that the QNNs based on our fast QPUs are superior to state-of-the-art real-valued models, especially in the scenarios requiring the robustness to random rotations. The code of this work is available at https://github.com/SuferQin/Fast-QPU.

Voxelisation Algorithms and Data Structures: A Review.

Voxel-based data structures, algorithms, frameworks, and interfaces have been used in computer graphics and many other applications for decades. There is a general necessity to seek adequate digital representations, such as voxels, that would secure unified data structures, multi-resolution options, robust validation procedures and flexible algorithms for different 3D tasks. In this review, we evaluate the most common properties and algorithms for voxelisation of 2D and 3D objects. Thus, many voxelisation algorithms and their characteristics are presented targeting points, lines, triangles, surfaces and solids as geometric primitives. For lines, we identify three groups of algorithms, where the first two achieve different voxelisation connectivity, while the third one presents voxelisation of curves. We can say that surface voxelisation is a more desired voxelisation type compared to solid voxelisation, as it can be achieved faster and requires less memory if voxels are stored in a sparse way. At the same time, we evaluate in the paper the available voxel data structures. We split all data structures into static and dynamic grids considering the frequency to update a data structure. Static grids are dominated by SVO-based data structures focusing on memory footprint reduction and attributes preservation, where SVDAG and SSVDAG are the most advanced methods. The state-of-the-art dynamic voxel data structure is NanoVDB which is superior to the rest in terms of speed as well as support for out-of-core processing and data management, which is the key to handling large dynamically changing scenes. Overall, we can say that this is the first review evaluating the available voxelisation algorithms for different geometric primitives as well as voxel data structures.

The structure of the superior and inferior parietal lobes predicts inter-individual suitability for virtual reality

The global virtual reality (VR) market is significantly expanding and being challenged with an increased demand owing to COVID-19. Unfortunately, VR is not useful for everyone due to large interindividual variability existing in VR suitability. To understand the neurobiological basis of this variability, we obtained neural structural and functional data from the participants using 3T magnetic resonance imaging. The participants completed one of two tasks (sports training or cognitive task) using VR, which differed in the time scale (months/minutes) and domain (motor learning/attention task). Behavioral results showed that some participants improved their motor skills in the real world after 1-month training in the virtual space or obtained high scores in the 3D attention task (high suitability for VR), whereas others did not (low suitability for VR). Brain structure analysis revealed that the structural properties of the superior and inferior parietal lobes contain information that can predict an individual’s suitability for VR.

ALA-Net: Adaptive Lesion-Aware Attention Network for 3D Colorectal Tumor Segmentation.

Accurate and reliable segmentation of colorectal tumors and surrounding colorectal tissues on 3D magnetic resonance images has critical importance in preoperative prediction, staging, and radiotherapy. Previous works simply combine multilevel features without aggregating representative semantic information and without compensating for the loss of spatial information caused by down-sampling. Therefore, they are vulnerable to noise from complex backgrounds and suffer from misclassification and target incompleteness-related failures. In this paper, we address these limitations with a novel adaptive lesion-aware attention network (ALA-Net) which explicitly integrates useful contextual information with spatial details and captures richer feature dependencies based on 3D attention mechanisms. The model comprises two parallel encoding paths. One of these is designed to explore global contextual features and enlarge the receptive field using a recurrent strategy. The other captures sharper object boundaries and the details of small objects that are lost in repeated down-sampling layers. Our lesion-aware attention module adaptively captures long-range semantic dependencies and highlights the most discriminative features, improving semantic consistency and completeness. Furthermore, we introduce a prediction aggregation module to combine multiscale feature maps and to further filter out irrelevant information for precise voxel-wise prediction. Experimental results show that ALA-Net outperforms state-of-the-art methods and inherently generalizes well to other 3D medical images segmentation tasks, providing multiple benefits in terms of target completeness, reduction of false positives, and accurate detection of ambiguous lesion regions.

Adjacent slices feature transformer network for single anisotropic 3D brain MRI image super-resolution

Magnetic resonance imaging (MRI) is widely used in clinical applications. However, due to the limitations in signal-to-noise ratio, physical properties of the scanner and scanning time, MRI images are usually acquired in low resolution, which restrains the accuracy of segmentation and recognition tasks. Recently, convolutional neural network (CNN) super-resolution methods have shown great potential in improving the resolution of MRI. Unfortunately, current methods neglect the data continuity and prior information of MRI images. In this paper, we handle the anisotropic 3D brain MRI images SR task as the problem of inserting new slices between adjacent in-plane slices. Then, we propose a novel adjacent slices feature transformer (ASFT) network to utilize the similarity of adjacent slices. Specifically, the backbone of the ASFT network consists of a series of stacked multi-branch features transformation and extraction (MFTE) blocks. In each MFTE block, we construct new spatial attention to focus on features from specific areas in reference branches and use channel attention to enhance the most valuable information. In addition, we propose a hybrid loss function with content and gradient information to refine the pixels and structures of the reconstructed image. We also introduce global residual learning to reduce the difficulty of network training. Experimental results show that the ASFT network gains the PSNR of 43.68 dB, 40.96 dB, and 41.22 dB with the scale factor of ×2 on the public Kirby21, ANVIL-adult, and MSSEG datasets, respectively. When compared with state-of-the-art MRI SR methods, the ASFT network achieves superior quantitative and qualitative performance.

Zoom‐fwd: Efficient technique for 3D gestual interaction with distant and occluded objects in virtual reality

AbstractIn the last decade, there has been an extraordinary acceleration in immersive technologies, including virtual reality (VR) and augmented reality (AR). The VR interfaces have widely explored various 3D interaction‐based approaches. 3D interaction is one of the main features of any VR system. However, the problems while selecting and manipulating distant and occluded 3D virtual objects in VR are still unresolved and suffer from a lack of precision and accuracy. To interact with VR/AR systems, various devices such as VR headsets or gloves are used. These devices are not very reliable yet; in fact they are inconvenient and invasive to a person's comfort. Currently, the natural user interfaces (NUIs) are increasingly introduced in human computer interaction (HCI) systems as gestures recognition, which prove to be very useful in the improvement of the user engagement and presence sensing. More particularly, they provide more user‐friendly and nonintrusive 3D interaction methods and techniques. In this article, the Zoom‐fwd, which is an efficient 3D interaction technique is presented. The proposed technique uses gesture recognition for different 3D interaction tasks like selection and manipulation. This new approach allows an efficient interaction with distant and occluded objects, while providing a precise selection, even when the environment is crowded. The Zoom‐fwd technique is a software solution for a number of hardware and software problems. A user study is conducted to determine whether the proposed technique is more suitable when performing interaction tasks. The results show that the Zoom‐fwd technique provides effective interaction with distant and occluded objects, by improving the user task completion performance. In addition, this indicates that selection precision has been enhanced significantly with the Zoom‐fwd technique.