3D Perception Research Articles

A Real-Time 3D Perception and Reconstruction System Based on a 2D Laser Scanner

This paper presents a real-time and low-cost 3D perception and reconstruction system which is suitable for autonomous navigation and large-scale environment reconstruction. The 3D mapping system is based on a rotating 2D planar laser scanner driven by a step motor, which is suitable for continuous mapping. However, for such a continuous mapping system, the challenge is that the range measurements are received at different times when the 3D LiDAR is moving, which will result in big distortion of the local 3D point cloud. As a result, the errors in motion estimation can cause misregistration of the resulting point cloud. In order to continuously estimate the trajectory of the sensor, we first extract feature points from the local point cloud and then estimate the transformation between current frame to local map to get the LiDAR odometry. After that, we use the estimated motion to remove the distortion of the local point cloud and then register the undistorted local point cloud to the global point cloud to get accurate global map. Finally, we propose a coarse-to-fine graph optimization method to minimize the global drift. The proposed 3D sensor system is advantageous due to its mechanical simplicity, mobility, low weight, low cost, and real-time estimation. To validate the performance of the proposed system, we carried out several experiments to verify its accuracy, robustness, and efficiency. The experimental results show that our system can accurately estimate the trajectory of the sensor and build a quality 3D point cloud map simultaneously.

Building extraction for 3D city modelling using airborne laser scanning data and high-resolution aerial photo

Light detection and ranging (LiDAR) technology has become a standard tool for three-dimensional mapping because it offers fast rate of data acquisition with unprecedented level of accuracy. This study presents an approach to accurately extract and model building in three-dimensional space from airborne laser scanning data acquired over Universiti Putra Malaysia in 2015. First, the point cloud was classified into ground and non-ground xyz points. The ground points was used to generate digital terrain model (DTM) while digital surface model (DSM) was produced from the entire point cloud. From DSM and DTM, we obtained normalise DSM (nDSM) representing the height of features above the terrain surface. Thereafter, the DSM, DTM, nDSM, laser intensity image and orthophoto were combined as a single data file by layer stacking. After integrating the data, it was segmented into image objects using Object Based Image Analysis (OBIA) and subsequently, the resulting image object classified into four land cover classes: building, road, waterbody and pavement. Assessment of the classification accuracy produced overall accuracy and Kappa coefficient of 94.02% and 0.88 respectively. Then the extracted building footprints from the building class were further processed to generate 3D model. The model provides 3D visual perception of the spatial pattern of the buildings which is useful for simulating disaster scenario for emergency management.

Blind Stereo Quality Assessment Based on Learned Features From Binocular Combined Images

Quality assessment of stereo images confronts more challenges than its 2D counterparts. Direct use of 2D assessment methods is not sufficient to deal with the challenges of 3D perception. In this paper, an efficient general-purpose no-reference stereo image quality assessment, based on unsupervised feature learning, is presented. The proposed method extracts features without any prior knowledge about the types and levels of distortions. This property enables our method to be adaptable for different applications. The perceived contrast and phase of the binocular combination of original stereo images are utilized to learn individual dictionaries. For each distorted stereo image, two feature vectors are pooled, in a hierarchical manner, over all sparse representation vectors of phase and contrast blocks by their corresponding dictionaries. Performance results of learning a regression model by the features acknowledge the superiority of the proposed method to state-of-the-art algorithms.

Improving 3D Shape Recognition withElectrostatic Friction Display.

Electrovibration technology has the potential for seamless integration into ordinary smartphones and tablets to provide programmable haptic feedback. The aim of this work is to seek effective ways to improve 3D perception of visual objects rendered on an electrovibration display. Utilizing a gradient-based algorithm, we first investigated whether rendering only lateral frictional force on an electrovibration display improves 3D shape perception compared to doing the same using a force-feedback interface. We observed that although users do not naturally associate electrovibration patterns to geometrical shapes, they can map patterns to shapes with moderate accuracy if guidance or context is given. Motivated by this finding, we generalized the gradient-based rendering algorithm to estimate the surface gradient for any 3D mesh and added an edge detection algorithm to render sharp edges. Then, we evaluated the advantages of our algorithm in a user study and found that our algorithm can notably improve the performance of 3D shape recognition when visual information is limited.

Split‐Depth Image Generation and Optimization

AbstractSplit‐depth images use an optical illusion, which can enhance the 3D impression of a 2D animation. In split‐depth images (also often called split‐depth GIFs due to the commonly used file format), static virtual occluders inform of vertical or horizontal bars are added to a video clip, which leads to occlusions that are interpreted by the observer as a depth cue. In this paper, we study different factors that contribute to the illusion and propose a solution to generate split‐depth images for a given RGB + depth image sequence. The presented solution builds upon a motion summarization of the object of interest (OOI) through space and time. It allows us to formulate the bar positioning as an energy‐minimization problem, which we solve efficiently. We take a variety of important features into account, such as the changes of the 3D effect due to changes in the motion topology, occlusion, the proximity of bars or the OOI, and scene saliency. We conducted a number of psycho‐visual experiments to derive an appropriate energy formulation. Our method helps in finding optimal positions for the bars and, thus, improves the 3D perception of the original animation. We demonstrate the effectiveness of our approach on a variety of examples. Our study with novice users shows that our approach allows them to quickly create satisfying results even for complex animations.

Walking through Architectural Spaces: The Impact of Interior Forms on Human Brain Dynamics.

Neuroarchitecture uses neuroscientific tools to better understand architectural design and its impact on human perception and subjective experience. The form or shape of the built environment is fundamental to architectural design, but not many studies have shown the impact of different forms on the inhabitants’ emotions. This study investigated the neurophysiological correlates of different interior forms on the perceivers’ affective state and the accompanying brain activity. To understand the impact of naturalistic three-dimensional (3D) architectural forms, it is essential to perceive forms from different perspectives. We computed clusters of form features extracted from pictures of residential interiors and constructed exemplary 3D room models based on and representing different formal clusters. To investigate human brain activity during 3D perception of architectural spaces, we used a mobile brain/body imaging (MoBI) approach recording the electroencephalogram (EEG) of participants while they naturally walk through different interior forms in virtual reality (VR). The results revealed a strong impact of curvature geometries on activity in the anterior cingulate cortex (ACC). Theta band activity in ACC correlated with specific feature types (rs (14) = 0.525, p = 0.037) and geometry (rs (14) = −0.579, p = 0.019), providing evidence for a role of this structure in processing architectural features beyond their emotional impact. The posterior cingulate cortex and the occipital lobe were involved in the perception of different room perspectives during the stroll through the rooms. This study sheds new light on the use of mobile EEG and VR in architectural studies and provides the opportunity to study human brain dynamics in participants that actively explore and realistically experience architectural spaces.

3D visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection

Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction.

Occlusion Leak Compensation for Optical See-Through Displays Using a Single-Layer Transmissive Spatial Light Modulator.

We propose an occlusion compensation method for optical see-through head-mounted displays (OST-HMDs) equipped with a singlelayer transmissive spatial light modulator (SLM), in particular, a liquid crystal display (LCD). Occlusion is an important depth cue for 3D perception, yet realizing it on OST-HMDs is particularly difficult due to the displays' semitransparent nature. A key component for the occlusion support is the SLM-a device that can selectively interfere with light rays passing through it. For example, an LCD is a transmissive SLM that can block or pass incoming light rays by turning pixels black or transparent. A straightforward solution places an LCD in front of an OST-HMD and drives the LCD to block light rays that could pass through rendered virtual objects at the viewpoint. This simple approach is, however, defective due to the depth mismatch between the LCD panel and the virtual objects, leading to blurred occlusion. This led existing OST-HMDs to employ dedicated hardware such as focus optics and multi-stacked SLMs. Contrary to these viable, yet complex and/or computationally expensive solutions, we return to the single-layer LCD approach for the hardware simplicity while maintaining fine occlusion-we compensate for a degraded occlusion area by overlaying a compensation image. We compute the image based on the HMD parameters and the background scene captured by a scene camera. The evaluation demonstrates that the proposed method reduced the occlusion leak error by 61.4% and the occlusion error by 85.7%.

Interocular contrast difference drives illusory 3D percept

Any processing delay between the two eyes can result in illusory 3D percepts for moving objects because of either changes in the pure disparities over time for disparity sensors or by changes to sensors that encode motion/disparity conjointly. This is demonstrated by viewing a fronto-parallel pendulum through a neutral density (ND) filter placed over one eye, resulting in the illusory 3D percept of the pendulum following an elliptical orbit in depth, the so-called Pulfrich phenomenon. Here we use a paradigm where a cylinder rotating in depth, defined by moving Gabor patches is presented at different interocular phases, generating strong to ambiguous depth percepts. This paradigm allows one to manipulate independently the contrast and the luminance of the patches to determine their influence on perceived motion-in-depth. Thus we show psychophysically that an interocular contrast difference can itself result in a similar illusory 3D percept of motion-in-depth. We argue that contrast, like luminance (ND filter) can modify the dynamics of visual neurons resulting in an interocular processing delay or an interocular velocity difference.

3D Mapping for Autonomous Quadrotor Aircraft

Autonomous navigation in global positioning system (GPS)-denied environments is one of the challenging problems in robotics. For small flying robots, autonomous navigation is even more challenging. These robots have limitations such as fast dynamics and limited sensor payload. To develop an autonomous robot, many challenges including two-dimensional (2D) and three-dimensional (3D) perception, path planning, exploration, and obstacle avoidance should be addressed in real-time and with limited resources. In this paper, a complete solution for autonomous navigation of a quadrotor rotorcraft is presented. The proposed solution includes 2D and 3D mapping with several autonomous behaviors such as target localization and displaying maps on multiple remote tablets. Multiple tests were performed in simulated and indoor/outdoor environments to show the effectiveness of the proposed solution.

Wide field of view tabletop light field display based on piece-wise tracking and off-axis pickup

A wide field of view (FOV) tabletop light field display (LFD) based on piece-wise tracking and off-axis pickup is presented to display the floating three-dimensional (3D) scene, which is 360°surrounding viewable. The demonstrated LFD is specially designed with an integral imaging display (IID) with 83 × 83 viewpoints and a full-parallax holographic functional screen (HFS). To improve the FOV, a piece-wise tracking based FOV enhancement method is proposed. The relationship between the viewing zone and the elemental images (EIs) is formulated. A ray-tracing based method using off-axis pickup instead of parallel pickup directly is adopted to render the 3D scene to EIs. Then the piece-wise tracking method of varying the viewing zone by placing the EIs according to the position of viewer is analyzed. The floating 3D scene with a FOV of 70°× 70°is experimentally demonstrated with a good 3D perception.

Perceptual ambiguity maps for robot localizability with range perception

Abstract A mobile robot equipped with 2D or 3D range sensors can move without changing its range readings if the perceived environment is poor in features. This is an ambiguous situation because a single perception can be associated with several robot poses. In consequence, robot localization capability is reduced. The problem we address is the quantification of this perceptual ambiguity as a property inherent to the system composed of the sensor and the static environment. Perceptual ambiguity is different from uncertainty of robot localization, although it is a cause of it. We propose an ambiguity model independent of the robot navigation system. It includes a probabilistic model of the indistinguishability of range readings, a generic range sensor model that supports laser and sonar sensors, and a generic range scanner model that supports any 2D or 3D range perception platform. Ambiguity is expressed in colour floor maps, which may be available before navigation starts, and where “bad localization” zones are easily detected. Experiments with virtual and real environments perceived from 2D laser and sonar scanners are presented, including the validation of the model with a real scans dataset. Results show how ambiguous zones are precisely determined, how to determine the optimum scanner orientation and aperture, and how to reduce the number of readings per scan for improving the robot's computational load and navigation speed.

Louay Eldada: LiDAR for Autonomous Vehicles: The future of 3D Sensing and Perception

Louay Eldada: LiDAR for Autonomous Vehicles: The future of 3D Sensing and Perception

Parkinson’s patients can rely on perspective cues to perceive 3D space

3D perception, which is necessary for an optimal navigation in our environment, relies on 2 complementary kinds of cues; binocular cues allowing precise depth localization near the point of visual interest and monocular ones that are necessary for correct global perception of visual space. Recent studies described deficient binocular 3D vision in PD patients; here we tested 3D vision in PD patients when based on monocular cues (m3D).Sixteen PD patients and 16 controls had to categorize visual stimuli as perceived in 2D (flat) or 3D (with depth). Both performance and response times were measured. EEGs were recorded to extract Visual Evoked Potentials. Effects of PD were tested by comparing psychometric and electrophysiological data obtained in controls and PD patients evaluated without dopaminergic treatment. Effects of Levodopa were tested by comparing data in PD patients with and without dopaminergic treatment.We didn’t find statistical differences between PD patients and controls’ performance. Severity of PD (UPDRS III) in OFF condition is positively correlated with P1 amplitudes and latencies for both 2D and m3D stimuli. Levodopa administration didn’t modify either PD patients’ performances although it increases principal visual components latencies for both 2D and m3D stimuli.Unlike binocular 3D vision, monocular 3D vision does not seem to get affected by PD. However given the correlation between severity of PD and VEPs’ modifications, alteration of visual cortical processing might have nonetheless begun. PD patients reporting trouble in perceiving space must rely more on m3D cues present in the environment.

Aging changes 3D perception: Evidence for hemispheric rebalancing of lateralized processes

When judging the 3D shape of a shaded image, young observers assume that the light source is placed above and to the left. This leftward bias has been attributed to hemispheric lateralization or experiential factors. Since aging is associated with loss of hemispheric lateralization, in the current study we measured the effect of aging on the assumed light source direction. Older participants exhibited, on average, a decreased left bias compared to young participants, as well as greater within-group variability in the distribution of assumed light source directions. In a separate sample of young and old participants, we replicated the age related effect in the assumed light source direction. Furthermore, in both young and old participants the assumed light source direction and the lateralized bias in a line bisection task were correlated. These findings suggest that diminished hemispheric lateralization, which accompanies aging, may affect the perception of the 3D structure of shaded surfaces. Shape from shading may thus provide a simple behavioral tool to track age related changes in hemispheric organization.

Space-time representation of people based on 3D skeletal data: A review

Spatiotemporal human representation based on 3D visual perception data is a rapidly growing research area. Representations can be broadly categorized into two groups, depending on whether they use RGB-D information or 3D skeleton data. Recently, skeleton-based human representations have been intensively studied and kept attracting an increasing attention, due to their robustness to variations of viewpoint, human body scale and motion speed as well as the realtime, online performance. This paper presents a comprehensive survey of existing space-time representations of people based on 3D skeletal data, and provides an informative categorization and analysis of these methods from the perspectives, including information modality, representation encoding, structure and transition, and feature engineering. We also provide a brief overview of skeleton acquisition devices and construction methods, enlist a number of benchmark datasets with skeleton data, and discuss potential future research directions.

A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems

Stereo vision is an important feature that enables machine vision systems to perceive their environment in 3D. While machine vision has spawned a variety of software algorithms to solve the stereo-correspondence problem, their implementation and integration in small, fast, and efficient hardware vision systems remains a difficult challenge. Recent advances made in neuromorphic engineering offer a possible solution to this problem, with the use of a new class of event-based vision sensors and neural processing devices inspired by the organizing principles of the brain. Here we propose a radically novel model that solves the stereo-correspondence problem with a spiking neural network that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. We validate the model with experimental results, highlighting features that are in agreement with both computational neuroscience stereo vision theories and experimental findings. We demonstrate its features with a prototype neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological stereo vision processing systems.

GazeDx: Interactive Visual Analytics Framework for Comparative Gaze Analysis with Volumetric Medical Images.

We present an interactive visual analytics framework, GazeDx (abbr. of GazeDiagnosis), for the comparative analysis of gaze data from multiple readers examining volumetric images while integrating important contextual information with the gaze data. Gaze pattern comparison is essential to understanding how radiologists examine medical images, and to identifying factors influencing the examination. Most prior work depended upon comparisons with manually juxtaposed static images of gaze tracking results. Comparative gaze analysis with volumetric images is more challenging due to the additional cognitive load on 3D perception. A recent study proposed a visualization design based on direct volume rendering (DVR) for visualizing gaze patterns in volumetric images; however, effective and comprehensive gaze pattern comparison is still challenging due to a lack of interactive visualization tools for comparative gaze analysis. We take the challenge with GazeDx while integrating crucial contextual information such as pupil size and windowing into the analysis process for more in-depth and ecologically valid findings. Among the interactive visualization components in GazeDx, a context-embedded interactive scatterplot is especially designed to help users examine abstract gaze data in diverse contexts by embedding medical imaging representations well known to radiologists in it. We present the results from two case studies with two experienced radiologists, where they compared the gaze patterns of 14 radiologists reading two patients' volumetric CT images.

Perceptual Metrics Quality

A 3D mesh can be subjected to different types of operations, such as compression, watermarking etc. Such processes lead to geometric distortions compared to the original version. In this context, quantifying the resultant modifications to the original mesh and evaluating the perceptual quality of degraded meshes become a critical issue. The perceptual 3D meshes quality is central in several applications to preserve the visual appearance of these treatments. The used metrics results have to be well correlated to the visual perception of humans. Although there are objective metrics, they do not allow the prediction of the perceptual quality, and do not include the human visual system properties. In the current work, a comparative study between the perceptual quality assessment metrics for 3D meshes was conducted. The experimental study on subjective database published by LIRIS / EPFL was used to test and to validate the results of six metrics. The results established that the Mesh Structural Distortion Measure metric achieved superior results compared to the other metrics.

Interhemispheric Connections between the Primary Visual Cortical Areas via the Anterior Commissure in Human Callosal Agenesis.

Aim: In humans, images in the median plane of the head either fall on both nasal hemi-retinas or on both temporal hemi-retinas. Interhemispheric connections allow cortical cells to have receptive fields on opposite sides. The major interhemispheric connection, the corpus callosum, is implicated in central stereopsis and disparity detection in front of the fixation plane. Yet individuals with agenesis of the corpus callosum may show normal stereopsis and disparity vergence. We set out to study a possible interhemispheric connection between primary visual cortical areas via the anterior commissure to explain this inconsistency because of the major role of these cortical areas in elaborating 3D visual perception.Methods: MRI, DTI and tractography of the brain of a 53-year old man with complete callosal agenesis and normal binocular single vision was undertaken. Tractography seed points were placed in both the right and the left V1 and V2. Nine individuals with both an intact corpus callosum and normal binocularity served as controls.Results: Interhemispheric tracts through the anterior commissure linking both V1 and V2 visual cortical areas bilaterally were indeed shown in the subject with callosal agenesis. All other individuals showed interhemispheric visual connections through the corpus callosum only.Conclusion: Callosal agenesis may result in anomalous interhemispheric connections of the primary visual areas via the anterior commissure. It is proposed here that these connections form as alternative to the normal callosal pathway and may participate in binocularity.