3D Perception Research Articles

The Art of Defense: Letting Networks Fool the Attacker

3D perception of objects is critical for many real-world applications, such as autonomous cars and robots. Among them, most state-of-the-art (SOTA) 3D perception systems are based on deep learning models. Recently, the research community found that 3D object classifiers on point cloud based on deep learning are easily fooled by adversarial point cloud craft by attackers. To overcome this, adversarial defenses are considered the most effective ways to improve the robustness of deep learning models, and most adversarial defenses on point cloud are focused on input transformation. However, all previous defense methods decrease the natural accuracy, and the nature of the point cloud classifiers itself has been overlooked. To this end, in this paper, we propose a novel adversarial defense for 3D point cloud classifiers that makes full use of the nature of the point cloud classifiers. Due to the disorder of point cloud, all point cloud classifiers have the property of permutation invariant to the input point cloud. Based on this nature, we design invariant transformations defense (IT-Defense). We show that, even after accounting for obfuscated gradients, our IT-Defense is a resilient defense against SOTA 3D attacks. Moreover, IT-Defense does not hurt clean accuracy compared to previous SOTA 3D defenses. Our code will be available at: https://github.com/cuge1995/IT-Defense.

A Category-Contrastive Guided-Graph Convolutional Network Approach for the Semantic Segmentation of Point Clouds

The semantic segmentation of light detection and ranging (LiDAR) point clouds plays an important role in 3D scene intelligent perception and semantic modeling. The unstructured, sparse and uneven characteristics of point clouds pose great challenges to the representation of the local geometric shapes, which degrades semantic segmentation performance. To address the challenges of describing local geometric shapes due to unstructured and sparse 3D point clouds, this paper proposes a category-contrastive-guided graph convolutional network (CGGC-Net) for the semantic segmentation of LiDAR point clouds. First, a detailed geometric structure of the raw point clouds is encoded to represent the inherent geometric pattern within the local neighborhood. At the same time, the geometric structures information is transmitted across multiple layers, so that the geometric structure encoding information containing different receptive fields and richer neighborhood spatial structure can be aggregated. Following this, the graph convolution neural network uses the edge convolution layer to adaptively describe the semantic correlation between the query point and its neighboring points, and combines the attention mechanism to gather the surrounding feature information to the query point. As a result, the graph convolution neural network and attention mechanism are iteratively stacked for the aggregation and fusion of spatial context semantic information, to generate highly discriminative semantic feature representation. Finally, the super parameters of the model are learned through a multitask optimization strategy guided by category-aware contrastive loss and cross-entropy loss. Experiments are conducted on the public SemanticKITTI dataset and the Stanford Large-Scale 3D Indoor Spaces dataset to demonstrate the effectiveness and reliability of the proposed CGGC-Net from both quantitative and qualitative perspectives. The results indicate its capability of automatically classifying LiDAR point clouds, with a mean intersection-over-union (mIoU) of 58.4%. Moreover, multiple comparative experiments also demonstrate the superior performance of the proposed method, exceeding state-of-the-art methods.

Binocular Structured Light 3-D Reconstruction System for Low-Light Underwater Environments: Design, Modeling, and Laser-Based Calibration

The three dimensional (3D) perception of the low-light underwater environment has always been a major challenge which greatly limits the underwater operations. In this paper, an underwater active vision measurement system based on binocular structured light is designed to achieve high precision 3D reconstruction. Firstly, the fusion technology of binocular camera and laser addresses underwater optical attenuation and feature sparsity. Then, in order to avoid the huge inertia caused by the overall movement of the system, a laser scanner based on the mirror-galvanometer is used to accomplish static scanning of the scene. Subsequently, considering the influence of multiple media, underwater refraction models including monocular imaging model, binocular ranging model, and binocular polar curve constraint model are systematically proposed. What’s more, the conventional checkerboard-based passive visual calibration method is ineffective for low-light waters. Therefore, a simple calibration block is designed, and a new multi-objective laser-based calibration algorithm based on laser geometric constraints is proposed. Finally, the effectiveness of the system is verified by analyzing the 3D reconstruction results of underwater objects.

The Flipping-Free Full-Parallax Tabletop Integral Imaging with Enhanced Viewing Angle Based on Space-Multiplexed Voxel Screen and Compound Lens Array

Tabletop integral imaging display with a more realistic and immersive experience has always been a hot spot in three-dimensional imaging technology, widely used in biomedical imaging and visualization to enhance medical diagnosis. However, the traditional structural characteristics of integral imaging display inevitably introduce the flipping effect outside the effective viewing angle. Here, a full-parallax tabletop integral imaging display without the flipping effect based on space-multiplexed voxel screen and compound lens array is demonstrated, and two holographic functional screens with different parameters are optically designed and fabricated. To eliminate the flipping effect in the reconstruction process, the space-multiplexed voxel screen consisting of a projector array and the holographic functional screen is presented to constrain light beams passing through the corresponding lens. To greatly promote imaging quality within the viewing area, the aspherical structure of the compound lens is optimized to balance the aberrations. It cooperates with the holographic functional screen to modulate the light field spatial distribution. Compared with the simulation results, the distortion rate of the imaging display is reduced to less than 9% from more than 30%. In the experiment, the floating high-quality reconstructed three-dimensional image without the flipping effect can be observed with the correct 3D perception at 96° × 96° viewing angle, where 44,100 viewpoints are employed.

Node Density Adjustment Method using Volume Estimation in Growing Neural Gas based 3D Perception System

Node Density Adjustment Method using Volume Estimation in Growing Neural Gas based 3D Perception System

Neural substrates that maintain perceiving 3d information: an ale meta-analysis study

3D perception is a crucial ability for human existence in the environment. Numerous studies have been focused on the neural mechanisms that are at the core of perceiving 3D information. However, there is no clear consensus on the reported results due to the wide variety of utilized tasks, stimuli, and visual cues. This fMRI meta-analysis study aims to a) define which specific brain areas are more active in processing of different depth cues during perceiving 3D information across the neuroimaging studies, b) explore a map of the functional brain activation associated with perceiving 3D within the brain areas that have received little attention, and c) identify selective areas that are more sensitive to types of stimuli and task paradigms. Data from 26 experiments was included in an Activation Likelihood Estimation analysis (ALE). The findings revealed six clusters of activation including the bilateral occipital, bilateral temporal, right parietal, and left frontal areas associated with the processing of visual depth cues. The analyses of task types showed higher activation in the right precuneus, and the left middle, and inferior occipital gyri for the active judgment paradigm and the left fusiform gyrus for passive viewing. The results showed that the left fusiform gyrus is sensitive to static image stimuli. This study for the first time provides a concordant map of activation for the perception process of 3D (rather than 2D) and suggests that perceiving 3D requires increased brain resources.

From pictures to reality: modelling the phenomenology and psychophysics of 3D perception.

The dominant inferential approach to human 3D perception assumes a model of spatial encoding based on a physical description of objects and space. Prevailing models based on this physicalist approach assume that the visual system infers an objective, unitary and mostly veridical representation ofthe external world. However, careful consideration of the phenomenology of 3Dperception challenges these assumptions. I review important aspects of phenomenology, psychophysics and neurophysiology which suggest that human visual perception of 3D objects and space is underwritten by distinct and dissociated spatial encodings that are optimized for specific regions of space. Specifically, I argue that 3D perception is underwritten by at least three distinct encodings for (1) egocentric distance perception at the ambulatory scale, (2) exocentric distance (scaled depth) perception optimized for near space, and (3) perception of object shape and layout (unscaled depth). This tripartite division can more satisfactorily account for the phenomenology, psychophysics and adaptive logic of human 3D perception. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Minimal theory of 3D vision: new approach to visual scale and visual shape.

Since Kepler and Descartes in the early-1600s, vision science has been committed to a triangulation model of stereo vision. But in the early-1800s, we realized that disparities are responsible for stereo vision. And we have spent the past 200 years trying to shoe-horn disparities back into the triangulation account. The first part of this article argues that this is a mistake, and that stereo vision is a solution to a different problem: the eradication of rivalry between the two retinal images, rather than the triangulation of objects in space. This leads to a 'minimal theory of 3D vision', where 3D vision is no longer tied to estimating the scale, shape, and direction of objects in the world. The second part of this article then asks whether the other aspects of 3D vision, which go beyond stereo vision, really operate at the same level of visual experience as stereo vision? I argue they do not. Whilst we want a theory of real-world 3D vision, the literature risks giving us a theory of picture perception instead. And I argue for a two-stage theory, where our purely internal 'minimal' 3D percept (from stereo vision) is linked to the world through cognition. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

A Survey on Deep-Learning-Based LiDAR 3D Object Detection for Autonomous Driving.

LiDAR is a commonly used sensor for autonomous driving to make accurate, robust, and fast decision-making when driving. The sensor is used in the perception system, especially object detection, to understand the driving environment. Although 2D object detection has succeeded during the deep-learning era, the lack of depth information limits understanding of the driving environment and object location. Three-dimensional sensors, such as LiDAR, give 3D information about the surrounding environment, which is essential for a 3D perception system. Despite the attention of the computer vision community to 3D object detection due to multiple applications in robotics and autonomous driving, there are challenges, such as scale change, sparsity, uneven distribution of LiDAR data, and occlusions. Different representations of LiDAR data and methods to minimize the effect of the sparsity of LiDAR data have been proposed. This survey presents the LiDAR-based 3D object detection and feature-extraction techniques for LiDAR data. The 3D coordinate systems differ in camera and LiDAR-based datasets and methods. Therefore, the commonly used 3D coordinate systems are summarized. Then, state-of-the-art LiDAR-based 3D object-detection methods are reviewed with a selected comparison among methods.

Binocular viewing geometry shapes neural processing of slanted planes: Results from theoretical V1 modeling and human psychophysics

An important component of real-world scene perception is the estimation of 3D surface orientation (slant and tilt). However, neither the neural circuits that compute 3D orientation nor the supporting computations have been fully uncovered. To help address that gap, we apply a projective geometry framework to the study of both the neural processing and perception of 3D orientation, building on recent work that suggests the mapping of the 3D environment onto the two retinae shapes both the tuning of neurons in area MT to motion-in-depth and corresponding patterns of perceptual errors in human psychophysics (Bonnen et al., 2020). The projection of slanted planes results in interocular disparities in retinal orientation, and interestingly, binocular V1 neurons can have different monocular orientation preferences (Bridge & Cumming, 2001), which could make them sensitive to orientation disparities. Therefore, we have constructed an encoding model that predicts responses of binocular V1 neurons to the orientation disparities produced by a stimulus plane with varying slant, given each theoretical neuron’s monocular orientation preferences and ocular dominance level. The resulting slant tuning curves are noncanonical in shape, do not tile slant evenly, and are strongly affected by viewing distance. Notably, at greater viewing distances, extreme slants (beyond approximately +/- 70 degrees) are encoded by steep portions of the curves, but moderate and low slants are encoded by relatively flat portions. As expected based on those observations, our model population decoder performs best at extreme slants, and performance degrades as slant approaches zero and as viewing distance increases. We additionally observed these patterns in human perceptual performance on a slant discrimination task. These results support the theory that the neural processing of 3D scene components is shaped by projective geometry and represent a step toward advancing understanding of the computations supporting 3D orientation perception.

Recovery of vivid 3D percepts from 125 years of historical motion film and animation clips, part 2 - 3D movies

Recovery of vivid 3D percepts from 125 years of historical motion film and animation clips, part 2 - 3D movies

Perception of value and effectiveness of cadaveric dissection in human anatomy teaching among undergraduate and surgical post-graduate students in the university of Nairobi

Background: Over the years medical educational approach to the teaching of anatomy has been centered on the use of cadaveric dissection or prosected specimen. However, there is currently a worldwide debate on the value and duration of dissection as a teaching tool in Human Anatomy. Therefore, it was important to determine what the students felt about cadaveric dissection and evaluate its value as a teaching method in learning anatomy. Objective: To investigate the perception of undergraduate and postgraduate medical and dental students towards the value and effectiveness of cadaveric dissection in the teaching of Human Anatomy at the University of Nairobi. Methods: A cross sectional descriptive study was conducted. Questionnaires were distributed among undergraduate and postgraduate medical and dental practitioners in the University of Nairobi. The data collected was analyzed using SPSS (Statistics Package for Social Sciences) and presented in tables and graphs Results: Out of the 107 students sampled, 73.33% were against a reduction in cadaveric dissection because it provides 3D perception of the human body (76.36%) and it assists in relating pathology to the relevant anatomy (70.91%). The students who preferred a reduction in cadaveric reported that it was too time consuming (90%) and felt that the time should be allocated to the clinical sciences (45%). Majority of the post-graduate students sampled thought that learning anatomy was relevant to their current profession (70.67). Conclusion: The results demonstrate that students perceive cadaveric dissection to be a valuable tool that should remain at the core of learning anatomy.

Bi-projection for 360°image object detection bridged by RoI Searcher

Object detection on 360°images is a vital component of 3D environment perception. The existing methods either treat panoramic images (usually represented in equirectangular projection—ERP) as normal FoV images and endure the distortions or project them into the less-distortion format and narrow the FoV, leading to unsatisfactory performance in practical applications. To solve this problem, we propose a dual-projection 360°object detection network named Bip R-CNN, consisting of three modules: a bi-projection feature extractor, a cross-projection region-of-interest (RoI) searcher, and a classification and regression predictor. Specifically, we extract the equirectangular and corresponding dual-cubemap features simultaneously from the input images. Besides, Projection-Inter Feature Fusion and Projection-Intra Feature Fusion are designed to allow the mutual interaction between the bi-projective features and promote the integration of features at different scales, respectively. In the proposed cross-projection RoI Searcher, we search for the bounding box (BBox) locations on cubemap from the corresponding ERP spherical proposals, bridging the RoIs of two different projection formats at feature level. Finally, the cube proposals are used to detect objects in the last predictor module. Considering the scarceness of the existing panoramic dataset (only indoor scenes), we propose an efficient approach to convert conventional datasets into annotated panoramic datasets without manual intervention, increasing the diversity of panoramic datasets. Extensive experiments are conducted on the synthetic and real-world datasets with spherical criteria, demonstrating our superiority to other state-of-the-art solutions.

Research on Mapping Error Control of Underground Space Mobile LiDAR Constrained by Cooperative Targets

In the underground environment, it is difficult to obtain spatial three-dimensional data because of occlusion and its complexity. Mobile light detection and ranging (LiDAR) measurement technology has the ability to obtain three-dimensional spatial information quickly and accurately, but in the underground environment, because of the lack of global navigation satellite system (GNSS) signal in the integrated navigation system, the measurement accuracy decreases with the increase of time. In this paper, an extended Kalman filter-based loose mode is constructed using the real coordinates of the center of gravity of the target and the measurement information of the laser foothold. It provides an additional space-time reference for the integrated navigation system of the mobile LiDAR system and restricts the errors. The simulation results show that the proposed method can effectively improve the measurement accuracy of mobile LiDAR system and enhance the 3D spatial perception ability of underground space.

Convolution Neural Network Based Human and Robot Collaboration for 3D Scene Perception

The approach of algorithmic operation of the Convolutional Neural Network (ConvNet/CNN) is like that of Deep Learning. The algorithm receives input instruction as an image by assigning characteristics of learnable weights and preferences on all inputs (aspects/objects) to the image. These features enable scalable input processing and effective differentiation and assignment of input and output images. For pre-processing requirements, ConvNet uses significantly low attributes as compared to other algorithms. However, in a primitive approach, filters are hand-engineered having sufficient algorithmic training. The ConvNets possess other abilities to learn image filtering and its characteristics. In this paper, an analysis of robot visual reasoning (for pick and place) is conducted. This refers to reasoning the latent meaning of visual signals or indications for future robot actions from visual observations of an HRC scene. In this paper, projection matrix estimation is computed using 2D points to represent the figure plane and 3D points to represent scene detection for initiating pick and place operation. During simulation, equations are represented using a matrix format based on figure coordinates. The design also implements automatic calibration and accuracy for the robot workspace. In addition, vision-based management for the robot end effector is also conducted based on horizontal targets, and upright targets. However, the pick and the place are articulated in the experiment without visual control. In our results, analysis of joint tangential forces is computed for the three joints, Alink1, Blink2, and Clink3. The outcome of axial force variation of the three joints is conducted in their state of mobility. Finally, the result of torque acting on the three joints increases when the simulation time is increased to achieve optimum performance

Time-to-Label: Temporal Consistency for Self-Supervised Monocular 3D Object Detection

Monocular 3D object detection continues to attract attention due to the cost benefits and wider availability of RGB cameras. Despite the recent advances and the ability to acquire data at scale, annotation cost and complexity still limit the size of 3D object detection datasets in the supervised settings. Self-supervised methods, on the other hand, aim at training deep networks relying on pretext tasks or various consistency constraints. Moreover, other 3D perception tasks (such as depth estimation) have shown the benefits of temporal priors as a self-supervision signal. In this work, we argue that the temporal consistency on the level of object poses, provides an important supervision signal given the strong prior on physical motion. Specifically, we propose a self-supervised loss which uses this consistency, in addition to render-and-compare losses defined on lidar point clouds, to refine noisy pose predictions and derive high-quality pseudo labels. To assess the effectiveness of the proposed method, we finetune a synthetically trained monocular 3D object detection model using the pseudo-labels that we generated on real data. Evaluation on the standard KITTI3D benchmark demonstrates that our method reaches competitive performance compared to other monocular self-supervised and supervised methods.

SGM3D: Stereo Guided Monocular 3D Object Detection

Monocular 3D object detection aims to predict the object location, dimension and orientation in 3D space alongside the object category given only a monocular image. It poses a great challenge due to its ill-posed property, which is a critical lack of depth information in the 2D image plane. While existing approaches leverage off-the-shelf depth estimation or rely on LiDAR sensors to mitigate this problem, dependence on the additional depth model or expensive equipment severely limits their scalability to generic 3D perception. In this paper, we propose a stereo-guided monocular 3D object detection framework, dubbed SGM3D, which aligns the monocular feature to the stereo feature and exploits the network's ability of generating the cross-modal feature for accurate 3D object detection. We present a multi-granularity feature alignment (MG-FA) mechanism to exploit the network's ability to generate stereo-mimicking features given only monocular cues. Coarse feature-level, as well as fine anchor-level feature alignment, are both leveraged for monocular feature guidance. In addition, we introduce an IoU matching-based feature alignment (IoU-MA) method for object-level feature alignment between the stereo and monocular predictions to alleviate the mismatches while adopting the MG-FA. Extensive experiments on the KITTI datasets demonstrate the effectiveness of our method. Code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhouzheyuan/sgm3d</uri> .

Vehicle-ride sensation sharing system with stereoscopic 3D visual perception and vibro-vestibular feedback for immersive remote collaboration

In this study, using a personal vehicle (i.e. Segway) and a wheelchair-type motion display, we proposed a vehicle-ride sensation sharing system to enable a local rider to collaborate with a remote driver immersively. The local rider sitting in the motion display can receive both the 3D visual perception and the vibro-vestibular sensation. The remote driver side of the system was developed by attaching the Segway with two 360-degree cameras and a stabilizer to capture stereoscopic 3D images and send them to each eye of a head-mounted display worn by a local rider. By modifying a conventional wheelchair with a simple, lightweight mechanism for actuation and vibration by two DC motors, we developed the prototype of the vibro-vestibular display for local riders. Then, we investigated the effectiveness of a vibro-vestibular wheelchair. The result showed that the acceleration/deceleration of the wheelchair proportional to that of visual cue could significantly reduce virtual reality (VR) sickness by approximately 54% and increase the sense of riding a vehicle by approximately 2.25 times. Moreover, we conducted a demo experience in SIGGRAPH ASIA 2019 for 3 days and 89 participants filled the questionnaire related to our system validation. The results suggested that vibro-vestibular feedback by the wheelchair is important for remote collaboration that uses a mobile vehicle.

Quantifying perceptual resolution for autostereoscopic three‐dimensional displays

AbstractThe perceptual experience issue is particularly important for autostereoscopic three‐dimensional (3D) displays since the perceptual resolution has a great correlation between panel resolution and other 3D display performances, but normally used effective resolution only considers the amount of information being sent to a certain viewing point. This work describes that the visual factors should be taken into consideration associated with resolution and contrast simultaneously to evaluate perceptual 3D resolution. We propose a quantitative evaluation of perceptual resolution of 3D displays from modulated contrast based on the contrast sensitivity function (CSF). A switchable parallax‐barrier display and a directional backlight display providing glasses‐free autostereoscopic vision are experimentally studied to verify the validity of the characterization. Due to the nonlinear response of the CSF threshold curve, less than 50% of the panel resolution in parallax‐barrier 3D displays is perceived, while the display with directional backlight illumination is shown to retain the panel resolution for perceptual experience as expected. We foresee that this study provides a practically objective paradigm to characterize the subjective perceptual resolution and the presented perceptual resolution can be served as a design guideline to achieve an improved 3D viewing experience.

Holographic 3D display using Fraunhofer diffraction with occlusion effect

Computer-generated holography is a key technology for three-dimensional (3D) visualization for its capability of true-to-life recreation of the recorded 3D scenes. Although multiple holographic algorithms have recently made impressive progress, high-quality 3D holographic display using Fraunhofer diffraction, which is characterized by long-distance reconstruction and fast calculation, has not been adequately investigated. In this paper, a layer-based computer-generated holographic algorithm based on Fraunhofer diffraction with digital lens phase (DLP) and occlusion effect for improved 3D holographic display is proposed. In order to relocate the reconstructed image plane, a lens group composed of the DLP and a real converging lens is utilized. Both simulated and experimental results verify the feasibility of the method for reconstructing 3D scenes in high quality with long diffraction range and corrected 3D perception. This method is envisioned to find itself important applications in future holographic 3D schemes, such as holographic 3D display applying neural networks since Fraunhofer diffraction is the most computationally saving in all diffraction methods.