Depth Cues Research Articles

AFLNet: Adversarial focal loss network for RGB-D salient object detection

Because salient objects usually have fewer data in a scene, the problem of class imbalance is often encountered in salient object detection (SOD). In order to address this issue and achieve the consistent salient objects, we propose an adversarial focal loss network with improving generative adversarial networks for RGB-D SOD (called AFLNet), in which color and depth branches constitute the generator to achieve the saliency map, and adversarial branch with high-order potentials, instead of pixel-wise loss function, refines the output of the generator to obtain contextual information of objects. We infer the adversarial focal loss function to solve the problem of foreground–background class imbalance. To sufficiently fuse the high-level features of color and depth cues, an inception model is adopted in deep layers. We conduct a large number of experiments using our proposed model and its variants, and compare them with state-of-the-art methods. Quantitative and qualitative experimental results exhibit that our proposed approach can improve the accuracy of salient object detection and achieve the consistent objects.

Multicamera 3D Viewpoint Adjustment for Robotic Surgery via Deep Reinforcement Learning

While robot-assisted minimally invasive surgery (RMIS) procedures afford a variety of benefits over open surgery and manual laparoscopic operations (including increased tool dexterity, reduced patient pain, incision size, trauma and recovery time, and lower infection rates [ 1 ], lack of spatial awareness remains an issue. Typical laparoscopic imaging can lack sufficient depth cues and haptic feedback, if provided, rarely reflects realistic tissue–tool interactions. This work is part of a larger ongoing research effort to reconstruct 3D surfaces using multiple viewpoints in RMIS to increase visual perception. The manual placement and adjustment of multicamera systems in RMIS are nonideal and prone to error [ 2 ], and other autonomous approaches focus on tool tracking and do not consider reconstruction of the surgical scene [ 3 , 4 , 5 ]. The group’s previous work investigated a novel, context-aware autonomous camera positioning method [ 6 ], which incorporated both tool location and scene coverage for multiple camera viewpoint adjustments. In this paper, the authors expand upon this prior work by implementing a streamlined deep reinforcement learning approach between optimal viewpoints calculated using the prior method [ 6 ] which encourages discovery of otherwise unobserved and additional camera viewpoints. Combining the framework and robustness of the previous work with the efficiency and additional viewpoints of the augmentations presented here results in improved performance and scene coverage promising towards real-time implementation.

Is accommodation a confounder in pupillometry research?

Much psychological research uses pupil diameter measurements to investigate the cognitive and emotional effects of visual stimuli. A potential problem is that accommodating at a nearby point causes the pupil to constrict. This study examined to what extent accommodation is a confounder in pupillometry research. Participants solved multiplication problems at different distances (Experiment 1) and looked at line drawings with different monocular depth cues (Experiment 2) while their pupil diameter, refraction, and vergence angle were recorded using a photorefractor. Experiment 1 showed that the pupils dilated while performing the multiplications, for all presentation distances. Pupillary constriction due to accommodation was not strong enough to override pupil dilation due to cognitive load. Experiment 2 showed that monocular depth cues caused a small shift in refraction in the expected direction. We conclude that, for the young student sample we used, pupil diameter measurements are not substantially affected by accommodation.

Motion parallax via head movements modulates visuo-motor control in pigeons.

Although it has been proposed that birds acquire visual depth cues through dynamic head movements, behavioral evidence on how birds use motion parallax depth cues caused by self-motion is lacking. This study investigated whether self-generated motion parallax modulates pecking motor control and visual size perception in pigeons (Columba livia). We trained pigeons to peck a target on a touch monitor and to classify it as small or large. To manipulate motion parallax of the target, we changed the target position on the monitor according to the bird's head position in real time using a custom-built head tracker with two cameras. Pecking motor control was affected by the manipulation of motion parallax: when the motion parallax signified the target position farther than the monitor surface, the head position just before pecking to target was near the monitor surface, and vice versa. By contrast, motion parallax did not affect how the pigeons classified target sizes, implying that motion parallax might not contribute to size constancy in pigeons. These results indicate that motion parallax via head movements modulates pecking motor control in pigeons, suggesting that head movements of pigeons have the visual function of accessing motion parallax depth cues.

Visual perspective-taking and image-like representations: We don't see it

The ability to represent another agent's visual perspective has recently been attributed to a process called “perceptual simulation”, whereby we generate an image-like or “quasi-perceptual” representation of another agent's vision. In an extensive series of experiments we tested this notion. Adult observers were presented with pictures of an agent looking at two horizontal lines, one of which was closer to the agent and hence appeared longer from his/her visual perspective. In each case approximately as many participants judged the closer line to appear shorter as longer (to the agent), i.e., failures to take the agent's perspective. This occurred when clear depth cues were added to emphasise the agent's location relative to the stimuli, when the agent was moved closer to the lines, when the lines were oriented vertically, when judgments could be made while viewing the image, and when participants imagined themselves in the agent's place. It also persisted when we asked participants to imagine what a photo taken from the same location as the agent would show, ruling out a misinterpretation of the instructions. Overall, our data suggest that adults attempt to solve visual perspective-taking problems by drawing upon naïve and often erroneous ideas about how vision works.

Deep learning for monocular depth estimation: A review

Depth estimation is a classic task in computer vision, which is of great significance for many applications such as augmented reality, target tracking and autonomous driving. Traditional monocular depth estimation methods are based on depth cues for depth prediction with strict requirements, e.g. shape-from-focus/ defocus methods require low depth of field on the scenes and images. Recently, a large body of deep learning methods have been proposed and has shown great promise in handling the traditional ill-posed problem. This paper aims to review the state-of-the-art development in deep learning-based monocular depth estimation. We give an overview of published papers between 2014 and 2020 in terms of training manners and task types. We firstly summarize the deep learning models for monocular depth estimation. Secondly, we categorize various deep learning-based methods in monocular depth estimation. Thirdly, we introduce the publicly available dataset and the evaluation metrics. And we also analysis the properties of these methods and compare their performance. Finally, we highlight the challenges in order to inform the future research directions.

Exploring Chromatic Aberration and Defocus Blur for Relative Depth Estimation From Monocular Hyperspectral Image.

This article investigates spectral chromatic and spatial defocus aberration in a monocular hyperspectral image (HSI) and proposes methods on how these cues can be utilized for relative depth estimation. The main aim of this work is to develop a framework by exploring intrinsic and extrinsic reflectance properties in HSI that can be useful for depth estimation. Depth estimation from a monocular image is a challenging task. An additional level of difficulty is added due to low resolution and noises in hyperspectral data. Our contribution to handling depth estimation in HSI is threefold. Firstly, we propose that change in focus across band images of HSI due to chromatic aberration and band-wise defocus blur can be integrated for depth estimation. Novel methods are developed to estimate sparse depth maps based on different integration models. Secondly, by adopting manifold learning, an effective objective function is developed to combine all sparse depth maps into a final optimized sparse depth map. Lastly, a new dense depth map generation approach is proposed, which extrapolate sparse depth cues by using material-based properties on graph Laplacian. Experimental results show that our methods successfully exploit HSI properties to generate depth cues. We also compare our method with state-of-the-art RGB image-based approaches, which shows that our methods produce better sparse and dense depth maps than those from the benchmark methods.

Two-Stage Cascaded Decoder for Semantic Segmentation of RGB-D Images

Exploiting RGB and depth information can boost the performance of semantic segmentation. However, owing to the differences between RGB images and the corresponding depth maps, such multimodal information should be effectively used and combined. Most existing methods use the same fusion strategy to explore multilevel complementary information at various levels, likely ignoring different feature contributions at various levels for segmentation. To address this problem, we propose a network using a two-stage cascaded decoder (TCD), embedding a detail polishing module, to effectively integrate high- and low-level features and suppress noise from low-level details. Additionally, we introduce a depth filter and fusion module to extract informative regions from depth cues with the guidance of RGB images. The proposed TCD network achieves comparable performance to state-of-the-art RGB-D semantic segmentation methods on the benchmark NYUDv2 and SUN RGB-D datasets.

Bifurcated Backbone Strategy for RGB-D Salient Object Detection.

Multi-level feature fusion is a fundamental topic in computer vision. It has been exploited to detect, segment and classify objects at various scales. When multi-level features meet multi-modal cues, the optimal feature aggregation and multi-modal learning strategy become a hot potato. In this paper, we leverage the inherent multi-modal and multi-level nature of RGB-D salient object detection to devise a novel Bifurcated Backbone Strategy Network (BBS-Net). Our architecture, is simple, efficient, and backbone-independent. In particular, first, we propose to regroup the multi-level features into teacher and student features using a bifurcated backbone strategy (BBS). Second, we introduce a depth-enhanced module (DEM) to excavate informative depth cues from the channel and spatial views. Then, RGB and depth modalities are fused in a complementary way. Extensive experiments show that BBS-Net significantly outperforms 18 state-of-the-art (SOTA) models on eight challenging datasets under five evaluation measures, demonstrating the superiority of our approach (~4% improvement in S-measure vs . the top-ranked model: DMRA). In addition, we provide a comprehensive analysis on the generalization ability of different RGB-D datasets and provide a powerful training set for future research. The complete algorithm, benchmark results, and post-processing toolbox are publicly available at https://github.com/zyjwuyan/BBS-Net.

Binocular light field acquisition using planar mirrors

Three-dimensional information in the real world is observed not only by using the parallax cue created by the different positions of the two eyes, but also by using the depth cue created by adjustment of the crystalline lens. Here we describe a catadioptric light-field camera using two planar mirrors to imitate the process by which humans observe the world. Using a microlens array and planar mirrors, both parallax and accommodation information can be obtained. The design of the proposed system and calibration procedure of its computational model are presented in detail. The proposed method was verified using a prototype with digital refocusing; results were obtained by applying the acquired calibrated light field from both view positions.

Comparative assessment of spatial perception in augmented reality depending on the consistency of depth cues

Comparative assessment of spatial perception in augmented reality depending on the consistency of depth cues

Hierarchical Alternate Interaction Network for RGB-D Salient Object Detection.

Existing RGB-D Salient Object Detection (SOD) methods take advantage of depth cues to improve the detection accuracy, while pay insufficient attention to the quality of depth information. In practice, a depth map is often with uneven quality and sometimes suffers from distractors, due to various factors in the acquisition procedure. In this article, to mitigate distractors in depth maps and highlight salient objects in RGB images, we propose a Hierarchical Alternate Interactions Network (HAINet) for RGB-D SOD. Specifically, HAINet consists of three key stages: feature encoding, cross-modal alternate interaction, and saliency reasoning. The main innovation in HAINet is the Hierarchical Alternate Interaction Module (HAIM), which plays a key role in the second stage for cross-modal feature interaction. HAIM first uses RGB features to filter distractors in depth features, and then the purified depth features are exploited to enhance RGB features in turn. The alternate RGB-depth-RGB interaction proceeds in a hierarchical manner, which progressively integrates local and global contexts within a single feature scale. In addition, we adopt a hybrid loss function to facilitate the training of HAINet. Extensive experiments on seven datasets demonstrate that our HAINet not only achieves competitive performance as compared with 19 relevant state-of-the-art methods, but also reaches a real-time processing speed of 43 fps on a single NVIDIA Titan X GPU. The code and results of our method are available at https://github.com/MathLee/HAINet.

Size-distance rescaling in the ensemble representation of range: Study with binocular and monocular cues

According to numerous studies observers can rapidly and precisely evaluate mean or range of the set. Recent studies have shown that the mean size estimated based on sizes of objects rescaled to their distances (Tiurina & Utochkin, 2019). In the current study, we directly tested this rescaling mechanism on the perception of range using binocular and monocular cues.In Experiment 1, a sample set of circles with different angular sizes and in different apparent distances were stereoscopically presented. Participants had to adjust the range of the test set to match the range of the sample set. The main manipulation was the size-distance correlation for sample and test sets: in negative size-distance correlation, the apparent range had to decrease, while in positive correlation - increase. We found the highest underestimation in the condition with the negative sample correlation and positive test correlation, which could be explained only if ensemble summary statistics were estimated after the item's rescaling.In Experiment 2, we used Ponzo-like illusion and spatial positions as a depth cue. Sets were presented with positive, negative or without size-distance correlation on a grey background or the background with Ponzo-like illusion. We found that the range was underestimated in negative correlation and overestimated in positive correlation.Thus, items of ensemble could be automatically rescaled according to their distance, based on both binocular and monocular cues, and ensemble summary statistics estimation is based on perceived sizes.

Retro-Reflective-Marker-Aided Target Pose Estimation in a Safety-Critical Environment

Visual technologies have an indispensable role in safety-critical applications, where tasks must often be performed through teleoperation. Due to the lack of stereoscopic and motion parallax depth cues in conventional images, alignment tasks pose a significant challenge to remote operation. In this context, machine vision can provide mission-critical information to augment the operator’s perception. In this paper, we propose a retro-reflector marker-based teleoperation aid to be used in hostile remote handling environments. The system computes the remote manipulator’s position with respect to the target using a set of one or two low-resolution cameras attached to its wrist. We develop an end-to-end pipeline of calibration, marker detection, and pose estimation, and extensively study the performance of the overall system. The results demonstrate that we have successfully engineered a retro-reflective marker from materials that can withstand the extreme temperature and radiation levels of the environment. Furthermore, we demonstrate that the proposed maker-based approach provides robust and reliable estimates and significantly outperforms a previous stereo-matching-based approach, even with a single camera.

Unimpaired perception of relative depth from perspective cues in strabismus.

Strabismus is a relatively common ophthalmological condition where the coordination of eye muscles to binocularly fixate a single point in space is impaired. This leads to deficits in vision and particularly in three-dimensional (3D) space perception. The exact nature of the deficits in 3D perception is poorly understood as much of understanding has relied on anecdotal reports or conjecture. Here, we investigated, for the first time, the perception of relative depth comparing strabismic and typically developed binocular observers. Specifically, we assessed the susceptibility to the depth cue of perspective convergence as well as the capacity to use this cue to make accurate judgements of relative depth. Susceptibility was measured by examining a 3D bias in making two-dimensional (2D) interval equidistance judgements and accuracy was measured by examining 3D interval equidistance judgements. We tested both monocular and binocular viewing of images of perspective scenes under two different psychophysical methods: two-alternative forced-choice (2AFC) and the method of adjustment. The biasing effect of perspective information on the 2D judgements (3D cue susceptibility) was highly significant and comparable for both subject groups in both the psychophysical tasks (all ps < 0.001) with no statistically significant difference found between the two groups. Both groups showed an underestimation in the 3D task with no significant difference between the group's judgements in the 2AFC task, but a small statistically significant difference (ratio difference of approx. 10%, p = 0.016) in the method of adjustment task. A small but significant effect of viewing condition (monocular versus binocular) was revealed only in the non-strabismic group (ratio difference of approx. 6%, p = 0.002). Our results show that both the automatic susceptibility to, and accuracy in the use of, the perspective convergence cue in strabismus is largely comparable to that found in typically developed binocular vision, and have implications on the nature of the encoding of depth in the human visual system.

Automatic RGBD Object Segmentation Based on MSRM Framework Integrating Depth Value

In this paper, an automatic RGBD object segmentation method is described. The method integrates depth feature with the cues from RGB images and then uses maximal similarity based region merging (MSRM) method to obtain the segmentation results. Firstly, the depth information is fused to the simple linear iterative clustering (SLIC) method so as to produce superpixels whose boundaries are well adhered to the edges of the natural image. Meanwhile, the depth prior is also incorporated into the saliency estimation, which helps a more accurate localization of representative object and background seeds. By introducing the depth cue into the region merging rule, the maximal geometry weighted similarity (MGWS) is considered, and the resulting segmentation framework has the ability to handle the complex image with similar colour appearance between object and background. Extensive experiments on public RGBD image datasets show that our proposed approach can reliably and automatically provide very promising segmentation results.

Infants distinguish and represent pattern as an object feature from externally generated patterns superimposed on real, 3‐dimensional objects' surfaces

AbstractAs infants form object representations, the patterns viewed on objects' surfaces may be challenging to decipher because these patterns may be created from the surface reflectance of an object (an object property) or from an external source, such as a cast shadow. We tested 7 ½‐month‐old infants' use of cues that specify the source of patterns seen on the surfaces of real, 3‐dimensional objects to individuate those objects. Results suggest that when forming object representations based on patterns, 7½‐month‐olds rely heavily on temporal and depth cues to distinguish patterns inherent to the object from other types of patterns.

Soundtrack loudness as a depth cue in stereoscopic 3D media

Assisted by the technological advances of the past decades, stereoscopic 3D cues are being integrated increasingly in both interactive and non-interactive media. Arguably, the main focus of this effort is placed on the creation of an increased sense of visual depth. Considering that human perception relies heavily on the audiovisual integration rather than on visual information alone, it is rather surprising that relatively little attention has been given so far to the potential effect of the soundtrack on 3D depth perception in contrast to the evident interest towards the study of realistic 3D audio spatialisation techniques and technologies. The multisensory nature of human perception suggests that the potential of sound design as a means to influence depth perception in the stereoscopic 3D visual environments may be worthy of further exploration. This study reports on our research into the possibilities of using alterations of the volume levels of the soundtrack in order to influence the perception of visual depth while viewing stereoscopic 3D animation clips. Based on previous findings indicating that the volume level of the soundtrack may be related to the perception of visual depth, a series of experiments further explored the effectiveness of this auditory cue. Results suggest that, under certain conditions, differences in volume levels of the soundtrack could influence the judgement of visual depth in a way that frequently contradicts real-life expectations. It is suggested that different, more metaphorical perceptual mechanisms may be in play when viewing stereoscopic 3D presentations than in real life. In this context, we conclude that stereoscopic 3D media can benefit from further exploration of the effectiveness of certain auditory cues as a means to influence the perception of depth within the stereoscopic 3D environment.

Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective.

Wearable near-eye displays for virtual and augmented reality (VR/AR) have seen enormous growth in recent years. While researchers are exploiting a plethora of techniques to create life-like three-dimensional (3D) objects, there is a lack of awareness of the role of human perception in guiding the hardware development. An ultimate VR/AR headset must integrate the display, sensors, and processors in a compact enclosure that people can comfortably wear for a long time while allowing a superior immersion experience and user-friendly human-computer interaction. Compared with other 3D displays, the holographic display has unique advantages in providing natural depth cues and correcting eye aberrations. Therefore, it holds great promise to be the enabling technology for next-generation VR/AR devices. In this review, we survey the recent progress in holographic near-eye displays from the human-centric perspective.

Multi-scale Feature Fusion for 3D Saliency Detection

3D saliency detection aims to take advantage of the disparity map, depth map and color information to automatically detect informative objects from natural scenes. Although studies have concentrated on this issue in recent years, there are challenges such as how to leverage disparity map or depth map effectively to compute depth-induced saliency, and how to fuse optimally multiple visual features and cues. A novel 3D saliency detection approach is proposed, which fuses local contrast, region contrast, texture feature, depth cue, and location cue into a unified saliency computation framework. Results show that the proposed approach achieves significant and consistent improvements on other advanced methods in the RGBD1000 datasets.