Depth Cues Research Articles

Neural substrates that maintain perceiving 3d information: an ale meta-analysis study

3D perception is a crucial ability for human existence in the environment. Numerous studies have been focused on the neural mechanisms that are at the core of perceiving 3D information. However, there is no clear consensus on the reported results due to the wide variety of utilized tasks, stimuli, and visual cues. This fMRI meta-analysis study aims to a) define which specific brain areas are more active in processing of different depth cues during perceiving 3D information across the neuroimaging studies, b) explore a map of the functional brain activation associated with perceiving 3D within the brain areas that have received little attention, and c) identify selective areas that are more sensitive to types of stimuli and task paradigms. Data from 26 experiments was included in an Activation Likelihood Estimation analysis (ALE). The findings revealed six clusters of activation including the bilateral occipital, bilateral temporal, right parietal, and left frontal areas associated with the processing of visual depth cues. The analyses of task types showed higher activation in the right precuneus, and the left middle, and inferior occipital gyri for the active judgment paradigm and the left fusiform gyrus for passive viewing. The results showed that the left fusiform gyrus is sensitive to static image stimuli. This study for the first time provides a concordant map of activation for the perception process of 3D (rather than 2D) and suggests that perceiving 3D requires increased brain resources.

Peripheral target detection can be modulated by target distance but not attended distance in 3D space simulated by monocular depth cues

Most studies of visuo-spatial attention present stimuli on a 2D plane, and less is known about how attention varies in 3D space. Previous studies found better peripheral detection performance for targets at a near compared to a far depth, simulated by pictorial cues and optical flow. The current study examined whether target detectability is monotonically related to distance along the depth axis, and whether the attended distance modulates the effect of target distance. We investigated these questions in two experiments that measured how apparent distance and target eccentricity affects peripheral target detection when performed alone during passive simulated self-motion, or during a simultaneous, active central car-following task. Experiment 1 found that targets at an apparent distance of 18.5 virtual meters were detected faster and more accurately than targets at 9.25 and 37 virtual meters, and detectability declined with eccentricity. Experiment 2 examined the effect of the attended location by varying the distance between the viewer and the lead car on which participants were instructed to fixate (i.e. the headway) while equating target distances across headway conditions. Experiment 2 replicated the effects found in Experiment 1, and headway did not modulate the effect of target distance. These results are consistent with the hypothesis that target detection depends non-monotonically on the distance between the viewer and the target, and is not affected by the distance between the target and attended location. However, target detection may also have been affected by stimulus characteristics that co-varied with apparent depth, rather than depth per se.

Naturalistic visualization of reaching movements using head-mounted displays improves movement quality compared to conventional computer screens and proves high usability.

The relearning of movements after brain injury can be optimized by providing intensive, meaningful, and motivating training using virtual reality (VR). However, most current solutions use two-dimensional (2D) screens, where patients interact via symbolic representations of their limbs (e.g., a cursor). These 2D screens lack depth cues, potentially deteriorating movement quality and increasing cognitive load. Head-mounted displays (HMDs) have great potential to provide naturalistic movement visualization by incorporating improved depth cues, reduce visuospatial transformations by rendering movements in the space where they are performed, and preserve eye-hand coordination by showing an avatar-with immersive VR (IVR)-or the user's real body-with augmented reality (AR). However, elderly populations might not find these novel technologies usable, hampering potential motor and cognitive benefits. We compared movement quality, cognitive load, motivation, and system usability in twenty elderly participants (>59 years old) while performing a dual motor-cognitive task with different visualization technologies: IVR HMD, AR HMD, and a 2D screen. We evaluated participants' self-reported cognitive load, motivation, and usability using questionnaires. We also conducted a pilot study with five brain-injured patients comparing the visualization technologies while using an assistive device. Elderly participants performed straighter, shorter duration, and smoother movements when the task was visualized with the HMDs than screen. The IVR HMD led to shorter duration movements than AR. Movement onsets were shorter with IVR than AR, and shorter for both HMDs than the screen, potentially indicating facilitated reaction times due to reduced cognitive load. No differences were found in the questionnaires regarding cognitive load, motivation, or usability between technologies in elderly participants. Both HMDs proved high usability in our small sample of patients. HMDs are a promising technology to be incorporated into neurorehabilitation, as their more naturalistic movement visualization improves movement quality compared to conventional screens. HMDs demonstrate high usability, without decreasing participants' motivation, and might potentially lower cognitive load. Our preliminary clinical results suggest that brain-injured patients may especially benefit from more immersive technologies. However, larger patient samples are needed to draw stronger conclusions.*.

Optimized computer-generated hologram for enhancing depth cue based on complex amplitude modulation.

In this Letter, we introduce a computer-generated hologram (CGH) optimization method to enhance the depth cue based on complex amplitude modulation (CAM). An iterative algorithm is designed to generate the optimized random phase (ORAP) according to the size of the target image and the bandwidth limitation condition. The ORAP with limited bandwidth is used as the initial phase of the target image and the hologram is encoded based on the analytical formula. Our proposal can maintain the advantages of CAM and achieve holographic three-dimensional (3D) display with an enhanced depth cue. It is expected that the proposed method could be widely used in holographic field in the future.

Super multi-view near-eye display with a lightguide combiner.

We propose a lightguide-type super multi-view near-eye display that uses a digital micromirror device and a LED array. The proposed method presents three-dimensional images with a natural monocular depth cue using a compact combiner optics which consists of a thin lightguide and holographic optical elements (HOEs). Feasibility of the proposed method is verified by optical experiments which demonstrate monocular three-dimensional image presentation over a wide depth range. We also analyze the degradation of the image quality stemming from the spectral spread of the HOEs and show its reduction by a pre-compensation exploiting an adaptive moment estimation (Adam) optimizer.

Sensorimotor learning of depth cue combination for reach-to-grasp actions

An observer’s planned grip size during reach-to-grasp actions is a combination of the estimates derived from visual depth cues like binocular disparity and texture gradients. Here, we investigate how sensorimotor learning mechanisms dynamically adjust cue combination based on experience with haptically-derived estimates of object shape, termed 3D cue remapping. Classic models of depth perception predict that 3D cue remapping occurs when the visual system detects a mismatch between visual and haptic information (i.e., ‘cue reweighting’). According to these models, a cue conflict between visual cue estimates is required to determine which estimator should be adjusted so that the combined-cue estimate aligns with haptic feedback. A recently developed alternative model named Intrinsic Constraint proposes that cue combination is approximated as a vector sum, resulting in larger depth estimates for stimuli that have more cues (e.g., disparity only vs. disparity and texture). Accordingly, we reasoned that if an observer grasps an unchanging shape but is shown virtual cue consistent renderings with alternately one and more cues, the visuomotor system should detect sensorimotor errors for stimuli with additional cues and attenuate the influence of those cues in subsequent depth judgements. We tested this prediction psychophysically. Participants repeatedly grasped a physical object of unchanging size while viewing virtual objects with one, two or three cues, which were confirmed to produce different depth estimates in a pre-test. Importantly, the physical object and all cues were rendered to specify consistent depth. Across the experiment we observed a change in motor planning consistent with 3D cue remapping. Planned grip size was initially larger for stimuli with more cues, but these differences decreased with repeated grasps. This learning is predicted by a computational model that dynamically adjusts cue combination based on the pattern of sensorimotor error signals obtained across grasps.

Real-world object size inferred from pictorial depth cues contributes to object recognition.

Real-world object size inferred from pictorial depth cues contributes to object recognition.

Effect of Pictorial Depth Cues on Illusory Spatial Depth of the Lin Family Garden

The methods of creating three-dimensional depth in two-dimensional paintings in the West and the East are quite different. Chinese paintings use layers of pictorial cues of relative size to create a sense of depth, while Western paintings utilize a linear perspective to integrate size related pictorial cues. The Chinese Scholar's Garden is known for its sense of endless spatial experience and illusory scene depth. This study examined how the pictorial space composed in Chinese landscape paintings was realized in the Chinese Scholar's Garden. The Lin Family Garden was selected for its adoption of the architectural style of the Chinese Scholar's Garden, and perceptual studies were conducted. The results confirmed that the uncommon reductions of the architectural elements observed in the garden can be an intentional effort to implement the pictorial depth cue of relative size utilized in Chinese landscape paintings to enrich the spatial depth of the garden scenes.

Learning Selective Mutual Attention and Contrast for RGB-D Saliency Detection.

How to effectively fuse cross-modal information is a key problem for RGB-D salient object detection. Early fusion and result fusion schemes fuse RGB and depth information at the input and output stages, respectively, and hence incur distribution gaps or information loss. Many models instead employ a feature fusion strategy, but they are limited by their use of low-order point-to-point fusion methods. In this paper, we propose a novel mutual attention model by fusing attention and context from different modalities. We use the non-local attention of one modality to propagate long-range contextual dependencies for the other, thus leveraging complementary attention cues to achieve high-order and trilinear cross-modal interaction. We also propose to induce contrast inference from the mutual attention and obtain a unified model. Considering that low-quality depth data may be detrimental to model performance, we further propose a selective attention to reweight the added depth cues. We embed the proposed modules in a two-stream CNN for RGB-D SOD. Experimental results demonstrate the effectiveness of our proposed model. Moreover, we also construct a new and challenging large-scale RGB-D SOD dataset of high-quality, which can promote both the training and evaluation of deep models.

Shedding Light on Cast Shadows: An Investigation of Perceived Ground Contact in AR and VR.

Virtual objects in augmented reality (AR) often appear to float atop real world surfaces, which makes it difficult to determine where they are positioned in space. This is problematic as many applications for AR require accurate spatial perception. In the current study, we examine how the way we render cast shadows-which act as an important monocular depth cue for creating a sense of contact between an object and the surface beneath it-impacts spatial perception. Over two experiments, we evaluate people's sense of surface contact given both traditional and non-traditional shadow shading methods in optical see-through augmented reality (OST AR), video see-through augmented reality (VST AR), and virtual reality (VR) head-mounted displays. Our results provide evidence that nontraditional shading techniques for rendering shadows in AR displays may enhance the accuracy of one's perception of surface contact. This finding implies a possible tradeoff between photorealism and accuracy of depth perception, especially in OST AR displays. However, it also supports the use of more stylized graphics like non-traditional cast shadows to improve perception and interaction in AR applications.

Pupil-Aware Holography

Holographic displays promise to deliver unprecedented display capabilities in augmented reality applications, featuring a wide field of view, wide color gamut, spatial resolution, and depth cues all in a compact form factor. While emerging holographic display approaches have been successful in achieving large étendue and high image quality as seen by a camera, the large étendue also reveals a problem that makes existing displays impractical: the sampling of the holographic field by the eye pupil. Existing methods have not investigated this issue due to the lack of displays with large enough étendue, and, as such, they suffer from severe artifacts with varying eye pupil size and location. We show that the holographic field as sampled by the eye pupil is highly varying for existing display setups, and we propose pupil-aware holography that maximizes the perceptual image quality irrespective of the size, location, and orientation of the eye pupil in a near-eye holographic display. We validate the proposed approach both in simulations and on a prototype holographic display and show that our method eliminates severe artifacts and significantly outperforms existing approaches.

Transformers and CNNs fusion network for salient object detection

Recently, Salient Object Detection (SOD) has achieved promising results thanks to the rapid evolution of CNN architectures. However, those CNN-based methods show limited capacity in modeling long-range interaction among pixels. In this paper, we propose to combine the merits from CNN and Transformer and design a unified model for RGB and RGB-D SOD. First, we represent the image with a sequence-to-sequence perspective and use a Transformer-based branch to express long-distance relationships of image tokens to obtain global semantic information and predict a coarse saliency map. Second, we employ a CNN-based branch to extract multi-scale local detail features to predict contour prediction for auxiliary supervision at each level. Finally, we propose the Bi-enhancement Fusion Module to fuse multi-scale cues from two branches to predict a more accurate saliency map. In addition, for RGB-D SOD, to obtain effective cross-modality features, we propose a Cross-modality Multi-Scale Transformer Module and a Depth-induced Enhancement Module to fuse RGB and depth cues in the Transformers branch and the CNNs branch, respectively. Experiments on both RGB and RGB-D SOD datasets demonstrate that our proposed model achieves satisfactory performance compared with state-of-the-art methods.

Depth dynamic center difference convolutions for monocular 3D object detection

Accurate 3D information is essential in the fields of autonomous driving and mobile robotics. Monocular 3D object detection provides a more economical solution than traditional LiDAR-dependent methods. Due to the lack of depth cues, monocular 3D object detection is extremely challenging to efficiently detect objects in 3D space from a single image. To mitigate this issue, we first identify how the depth information on surrounding pixels provides additional support for depth map estimation in the 3D detection of driving scenes. Based on this observation, we design Depth Dynamic Center Difference Convolution (DDCDC), which introduces surrounding pixel cues in depth estimation and has different convolution kernels weights for each pixel of all examples. This module not only overcomes the limitations of conventional 2D convolution, but also highlights the differences in depth information between the target and the background, so more attention is paid to interesting objects. Finally, we design an end-to-end monocular 3D object detection network with proposed DDCDC convolution modules. As a demonstration of the effectiveness of our method, our module is validated on two datasets: KITTI and nuScenes. The DDCDC achieves the most significant improvement in a simple setup compared to existing methods. Our evaluation results for the KITTI split1/split2 set are 23.83/21.48,16.00/13.92,12.04/10.59 (based on easy, medium, and hard), while the results for the nuScenes test set are mAP = 0.364 and NDS = 0.434.

MoADNet: Mobile Asymmetric Dual-Stream Networks for Real-Time and Lightweight RGB-D Salient Object Detection

RGB-D Salient Object Detection (RGB-D SOD) aims at detecting remarkable objects by complementary information from RGB images and depth cues. Although many outstanding prior arts have been proposed for RGB-D SOD, most of them focus on performance enhancement, while lacking concern about practical deployment on mobile devices. In this paper, we propose mobile asymmetric dual-stream networks (MoADNet) for real-time and lightweight RGB-D SOD. First, inspired by the intrinsic discrepancy between RGB and depth modalities, we observe that depth maps can be represented by fewer channels than RGB images. Thus, we design asymmetric dual-stream encoders based on MobileNetV3. Second, we develop an inverted bottleneck cross-modality fusion (IBCMF) module to fuse multimodality features, which adopts an inverted bottleneck structure to compensate for the information loss in the lightweight backbones. Third, we present an adaptive atrous spatial pyramid (A2SP) module to speed up the inference, while maintaining the performance by appropriately selecting multiscale features in the decoder. Extensive experiments are conducted to compare our method with 15 state-of-the-art approaches. Our MoADNet obtains competitive results on five benchmark datasets under four evaluation metrics. For efficiency analysis, the proposed method significantly outperforms other baselines by a large margin. The MoADNet only contains 5.03 M parameters and runs 80 FPS when testing a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$256\times 256$ </tex-math></inline-formula> image on a single NVIDIA 2080Ti GPU.

Virtual stereo content rendering technology review for light-field display

Light field display (LFD) is considered as a promising technology to reconstruct the light rays’ distribution of the real 3D scene, which approximates the original light field of target displayed objects with all depth cues in human vision including binocular disparity, motion parallax, color hint and correct occlusion relationship. Currently, computer-generated content is widely used for the LFD system, therefore rich 3D content can be provided. This paper firstly introduces applications of light field technologies in display system. Additionally, virtual stereo content rendering techniques and their application scenes are thoroughly combed and pointed out its pros and cons. Moreover, according to the different characteristics of light field system, the coding and correction algorithms in virtual stereo content rendering techniques are reviewed. Through the above discussion, there are still many problems in the existing rendering techniques for LFD. New rendering algorithms should be introduced to solve the real-time light-field rendering problem for large-scale virtual scenes.

Real-time realistic computer-generated hologram with accurate depth precision and a large depth range.

Holographic display is an ideal technology for near-eye display to realize virtual and augmented reality applications, because it can provide all depth perception cues. However, depth performance is sacrificed by exiting computer-generated hologram (CGH) methods for real-time calculation. In this paper, volume representation and improved ray tracing algorithm are proposed for real-time CGH generation with enhanced depth performance. Using the single fast Fourier transform (S-FFT) method, the volume representation enables a low calculation burden and is efficient for Graphics Processing Unit (GPU) to implement diffraction calculation. The improved ray tracing algorithm accounts for accurate depth cues in complex 3D scenes with reflection and refraction, which is represented by adding extra shapes in the volume. Numerical evaluation is used to verify the depth precision. And experiments show that the proposed method can provide a real-time interactive holographic display with accurate depth precision and a large depth range. CGH of a 3D scene with 256 depth values is calculated at 30fps, and the depth range can be hundreds of millimeters. Depth cues of reflection and refraction images can also be reconstructed correctly. The proposed method significantly outperforms existing fast methods by achieving a more realistic 3D holographic display with ideal depth performance and real-time calculation at the same time.

Depth‐Aware Shadow Removal

AbstractShadow removal from a single image is an ill‐posed problem because shadow generation is affected by the complex interactions of geometry, albedo, and illumination. Most recent deep learning‐based methods try to directly estimate the mapping between the non‐shadow and shadow image pairs to predict the shadow‐free image. However, they are not very effective for shadow images with complex shadows or messy backgrounds. In this paper, we propose a novel end‐to‐end depth‐aware shadow removal method without using depth images, which estimates depth information from RGB images and leverages the depth feature as guidance to enhance shadow removal and refinement. The proposed framework consists of three components, including depth prediction, shadow removal, and boundary refinement. First, the depth prediction module is used to predict the corresponding depth map of the input shadow image. Then, we propose a new generative adversarial network (GAN) method integrated with depth information to remove shadows in the RGB image. Finally, we propose an effective boundary refinement framework to alleviate the artifact around boundaries after shadow removal by depth cues. We conduct experiments on several public datasets and real‐world shadow images. The experimental results demonstrate the efficiency of the proposed method and superior performance against state‐of‐the‐art methods.

Mixed reality-integrated 3D/2D vision mapping for intuitive teleoperation of mobile manipulator

• This work evaluates the impact of the integrated 3D/2D vision and motion mapping schemes on teleoperation efficiency and user experience in a mixed reality (MR) environment. • A significant enhancement in task performance is achieved with the point cloud within an MR environment (MR-3DPC) vision and motion mapping scheme, while minimizing training effort and cognitive workload. • The MR-integrated 3D/2D vision and motion mapping schemes of teleoperation reduced overall task completion times by 34% and 17%. • It is found using point cloud within an MR environment (MR-3DPC) outperforms displaying stereoscopic vision (MR-3DSV) in MR and can enhance users’ stereoscopic perception and task performance. Depth cues are crucial to increase user perception and spatial awareness of the remote environment when remotely guiding complex robotic systems. A mixed reality (MR) integrated 3D/2D vision and motion mapping framework for immersive and intuitive telemanipulation of a complex mobile manipulator is presented. The proposed 3D immersive telerobotic schemes provide the users with depth perception through the merging of multiple 3D/2D views of the remote environment via MR subspace. The mobile manipulator platform consists of a 6°-of-freedom (DOF) industrial manipulator, 3D-printed parallel gripper, and mobile base, which can be controlled by non-skilled operators who are physically separated from the robot working space through a velocity-based imitative motion mapping approach. This work evaluates the impact of depth perception and immersion provided by integrated 3D/2D vision and motion mapping schemes on teleoperation efficiency and user experience in an MR environment. In particular, the MR enhanced systems maintain spatial awareness and perceptual salience of the remote scene in 3D, facilitating intuitive mixed reality human-robot interaction (MR-HRI). This study compared two MR-integrated 3D/2D vision and motion mapping schemes against a typical 2D Baseline visual display method through pick-and-place, assembly, and dexterous manufacturing tasks. The MR-integrated 3D/2D vision and motion mapping schemes of teleoperation reduced overall task completion times by 34% and 17%, compared to the MR-2D Baseline, while minimizing training effort and cognitive workload.

Distance estimation from monocular cues in an ethological visuomotor task.

In natural contexts, sensory processing and motor output are closely coupled, which is reflected in the fact that many brain areas contain both sensory and movement signals. However, standard reductionist paradigms decouple sensory decisions from their natural motor consequences, and head-fixation prevents the natural sensory consequences of self-motion. In particular, movement through the environment provides a number of depth cues beyond stereo vision that are poorly understood. To study the integration of visual processing and motor output in a naturalistic task, we investigated distance estimation in freely moving mice. We found that mice use vision to accurately jump across a variable gap, thus directly coupling a visual computation to its corresponding ethological motor output. Monocular eyelid suture did not affect gap jumping success, thus mice can use cues that do not depend on binocular disparity and stereo vision. Under monocular conditions, mice altered their head positioning and performed more vertical head movements, consistent with a shift from using stereopsis to other monocular cues, such as motion or position parallax. Finally, optogenetic suppression of primary visual cortex impaired task performance under both binocular and monocular conditions when optical fiber placement was localized to binocular or monocular zone V1, respectively. Together, these results show that mice can use monocular cues, relying on visual cortex, to accurately judge distance. Furthermore, this behavioral paradigm provides a foundation for studying how neural circuits convert sensory information into ethological motor output.

Visual discomfort factor analysis and modeling for worldwide stereoscopic 3D maps

EarthView3D is the first worldwide orthographic stereoscopic 3D (S3D) map. It presents an accurate presentation of the Earth’s surface and can provide an immersive geospatial infrastructure for the upcoming metaverse ecosystems. This study conducted the first comprehensive assessment of the worldwide S3D maps, investigated the factors of visual discomfort, predicted discomfort level, and provided design recommendations for improving visual experience. Participants rated the S3D map images and reported the reasons for their visual discomfort. General Eye Symptom Questionnaires (GESQs) and Simplified Simulator Sickness Questionnaires (SSSQs) were used to record the development of visual fatigue with time. Six categories of measurable variables presumably related to visual discomfort were proposed based on participants’ reports, IEEE standards, and literature on visual experience. Factor analysis extracted four principal factors, disparity, terrain texture, luminance, and amplitude spectrum, whose contributions to variance were 38%, 23%, 16%, and 11%, respectively. The selected variables and subjective mean rating scores were used to construct a regression model for the prediction of visual discomfort. Performance evaluation Root Mean Square Error (RMSE) was lower than that reported in previous studies on different S3D databases. The results indicated that besides vergence-accommodation conflict and depth cue conflict, perception of Earth terrain’s textures and luminance played an important role in visual discomfort of viewing orthographic S3D maps.